Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
6ced817
Add no-GIL interpreter support
clin1234 Jul 20, 2025
75cdd03
Exclude PYTHON_GIL=0 for now
clin1234 Jul 20, 2025
cafe4f0
Lint
clin1234 Jul 20, 2025
3cc2a38
Exclude PYTHON_GIL for now
clin1234 Jul 25, 2025
61aa23f
Update .github/workflows/wheel.yml
clin1234 Aug 6, 2025
7b7b1bf
Merge branch 'main' into nogil
clin1234 Aug 6, 2025
d965025
Fix extremely silly typo
clin1234 Aug 6, 2025
bf58057
Drop 3.8 support in test.yml
clin1234 Aug 7, 2025
42f4cba
Drop pytest-run-parallel as test requirement in requirements.txt
clin1234 Aug 7, 2025
d0797bd
Separate C extension and fallback tests for GIL and no-GIL interpreters
clin1234 Aug 7, 2025
79e5de1
Work around Windows wonkiness
clin1234 Aug 7, 2025
9a7659f
Fix minor YAML oops in .github/workflows/test.yml
clin1234 Sep 9, 2025
e0e9b12
Fix minor YAML oops in .github/workflows/test.yml
clin1234 Sep 9, 2025
ccfd254
Drop spurious multhreading test
clin1234 Sep 11, 2025
48c56cd
Drop -dev suffix in test.yml
clin1234 Sep 25, 2025
c2a9f1f
ci: add support for building windows on arm wheels (#643)
Greenie0701 Sep 26, 2025
716e08c
Merge branch 'main' into nogil
clin1234 Sep 27, 2025
0f3c4be
README: fix typos and grammar (#648)
ThomasWaldmann Oct 8, 2025
19b5d33
release v1.1.2 (#649)
methane Oct 8, 2025
31f8f8b
Merge branch 'main' into nogil
methane Oct 8, 2025
ef4f83d
relax setuptools version (#652)
methane Oct 9, 2025
c2546ea
update setuptools requirements to >=78.1.1 (#653)
methane Oct 9, 2025
af45640
cython: freethreading_compatible (#654)
methane Oct 9, 2025
c1ecd23
drop Python 3.9 (#656)
methane Oct 9, 2025
8019165
Merge branch 'main' into nogil
methane Oct 9, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 28 additions & 8 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,18 +9,21 @@ jobs:
test:
strategy:
matrix:
os: ["ubuntu-latest", "windows-latest", "macos-latest"]
py: ["3.14-dev", "3.13", "3.12", "3.11", "3.10", "3.9"]
os: ["ubuntu-latest", "windows-latest", "windows-11-arm", "macos-latest"]
py: ["3.14", "3.14t", "3.13", "3.12", "3.11", "3.10"]
exclude:
- os: windows-11-arm
py: "3.10"

runs-on: ${{ matrix.os }}
name: Run test with Python ${{ matrix.py }} on ${{ matrix.os }}

steps:
- name: Checkout
uses: actions/checkout@v4
uses: actions/checkout@v5

- name: Set up Python
uses: actions/setup-python@v5
uses: actions/setup-python@v6
with:
python-version: ${{ matrix.py }}
allow-prereleases: true
Expand All @@ -29,8 +32,12 @@ jobs:
- name: Prepare
shell: bash
run: |
pip install -U pip
pip install -r requirements.txt pytest
python -m pip install -r requirements.txt pytest

- name: Install pytest-run-parallel under free-threading
if: contains(matrix.py, 't')
run: |
pip install pytest-run-parallel

- name: Build
shell: bash
Expand All @@ -39,20 +46,33 @@ jobs:
pip install .

- name: Test (C extension)
if: ${{ ! contains(matrix.py, 't') }}
shell: bash
run: |
pytest -v test

- name: Test (pure Python fallback)
if: ${{ ! contains(matrix.py, 't') }}
shell: bash
run: |
MSGPACK_PUREPYTHON=1 pytest -v test

- name: Test (C extension) in parallel under free-threading
if: contains(matrix.py, 't')
shell: bash
run: |
pytest -v --parallel-threads=auto --iterations=20 test

- name: Test (pure Python fallback) in parallel under free-threading
if: contains(matrix.py, 't')
shell: bash
run: |
MSGPACK_PUREPYTHON=1 pytest -v --parallel-threads=auto --iterations=20 test

- name: build packages
shell: bash
run: |
pip install build
python -m build
python -m build -nv

- name: upload packages
uses: actions/upload-artifact@v4
Expand Down
10 changes: 5 additions & 5 deletions .github/workflows/wheel.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,13 @@ jobs:
strategy:
matrix:
# macos-13 is for intel
os: ["ubuntu-24.04", "ubuntu-24.04-arm", "windows-latest", "macos-13", "macos-latest"]
os: ["ubuntu-24.04", "ubuntu-24.04-arm", "windows-latest", "windows-11-arm", "macos-13", "macos-latest"]
runs-on: ${{ matrix.os }}
name: Build wheels on ${{ matrix.os }}

steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
- uses: actions/checkout@v5
- uses: actions/setup-python@v6
with:
python-version: "3.x"
cache: "pip"
Expand All @@ -29,11 +29,11 @@ jobs:
make cython

- name: Build
uses: pypa/cibuildwheel@v3.1.1
uses: pypa/cibuildwheel@v3.2.0
env:
CIBW_TEST_REQUIRES: "pytest"
CIBW_TEST_COMMAND: "pytest {package}/test"
CIBW_SKIP: "pp* cp38-*"
CIBW_SKIP: "pp* cp38-* cp39-* cp310-win_arm64"

- name: Build sdist
if: runner.os == 'Linux' && runner.arch == 'X64'
Expand Down
13 changes: 13 additions & 0 deletions ChangeLog.rst
Original file line number Diff line number Diff line change
@@ -1,3 +1,16 @@
1.1.2
=====

Release Date: 2025-10-08

This release does not change source code. It updates only building wheels:

* Update Cython to v3.1.4
* Update cibuildwheel to v3.2.0
* Drop Python 3.8
* Add Python 3.14
* Add windows-arm

1.1.1
=====

Expand Down
2 changes: 1 addition & 1 deletion MANIFEST.in
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
include setup.py
include COPYING
include README.md
recursive-include msgpack *.h *.c *.pyx *.cpp
recursive-include msgpack *.h *.c *.pyx
recursive-include test *.py
64 changes: 32 additions & 32 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
[![Build Status](https://github.com/msgpack/msgpack-python/actions/workflows/wheel.yml/badge.svg)](https://github.com/msgpack/msgpack-python/actions/workflows/wheel.yml)
[![Documentation Status](https://readthedocs.org/projects/msgpack-python/badge/?version=latest)](https://msgpack-python.readthedocs.io/en/latest/?badge=latest)

## What's this
## What is this?

[MessagePack](https://msgpack.org/) is an efficient binary serialization format.
It lets you exchange data among multiple languages like JSON.
Expand All @@ -25,21 +25,21 @@ But msgpack provides a pure Python implementation (`msgpack.fallback`) for PyPy.

### Windows

When you can't use a binary distribution, you need to install Visual Studio
or Windows SDK on Windows.
Without extension, using pure Python implementation on CPython runs slowly.
If you can't use a binary distribution, you need to install Visual Studio
or the Windows SDK on Windows.
Without the extension, the pure Python implementation on CPython runs slowly.


## How to use

### One-shot pack & unpack

Use `packb` for packing and `unpackb` for unpacking.
msgpack provides `dumps` and `loads` as an alias for compatibility with
msgpack provides `dumps` and `loads` as aliases for compatibility with
`json` and `pickle`.

`pack` and `dump` packs to a file-like object.
`unpack` and `load` unpacks from a file-like object.
`pack` and `dump` pack to a file-like object.
`unpack` and `load` unpack from a file-like object.

```pycon
>>> import msgpack
Expand Down Expand Up @@ -73,7 +73,7 @@ for unpacked in unpacker:
```


### Packing/unpacking of custom data type
### Packing/unpacking of custom data types

It is also possible to pack/unpack custom data types. Here is an example for
`datetime.datetime`.
Expand Down Expand Up @@ -140,16 +140,16 @@ True
### Advanced unpacking control

As an alternative to iteration, `Unpacker` objects provide `unpack`,
`skip`, `read_array_header` and `read_map_header` methods. The former two
read an entire message from the stream, respectively de-serialising and returning
`skip`, `read_array_header`, and `read_map_header` methods. The former two
read an entire message from the stream, respectively deserializing and returning
the result, or ignoring it. The latter two methods return the number of elements
in the upcoming container, so that each element in an array, or key-value pair
in a map, can be unpacked or skipped individually.


## Notes

### string and binary type in old msgpack spec
### String and binary types in the old MessagePack spec

Early versions of msgpack didn't distinguish string and binary types.
The type for representing both string and binary types was named **raw**.
Expand All @@ -167,7 +167,7 @@ and `raw=True` options.

### ext type

To use the **ext** type, pass `msgpack.ExtType` object to packer.
To use the **ext** type, pass a `msgpack.ExtType` object to the packer.

```pycon
>>> import msgpack
Expand All @@ -181,34 +181,34 @@ You can use it with `default` and `ext_hook`. See below.

### Security

To unpacking data received from unreliable source, msgpack provides
When unpacking data received from an unreliable source, msgpack provides
two security options.

`max_buffer_size` (default: `100*1024*1024`) limits the internal buffer size.
It is used to limit the preallocated list size too.
It is also used to limit preallocated list sizes.

`strict_map_key` (default: `True`) limits the type of map keys to bytes and str.
While msgpack spec doesn't limit the types of the map keys,
there is a risk of the hashdos.
While the MessagePack spec doesn't limit map key types,
there is a risk of a hash DoS.
If you need to support other types for map keys, use `strict_map_key=False`.


### Performance tips

CPython's GC starts when growing allocated object.
This means unpacking may cause useless GC.
You can use `gc.disable()` when unpacking large message.
CPython's GC starts when the number of allocated objects grows.
This means unpacking may trigger unnecessary GC.
You can use `gc.disable()` when unpacking a large message.

List is the default sequence type of Python.
But tuple is lighter than list.
A list is the default sequence type in Python.
However, a tuple is lighter than a list.
You can use `use_list=False` while unpacking when performance is important.


## Major breaking changes in the history

### msgpack 0.5

Package name on PyPI was changed from `msgpack-python` to `msgpack` from 0.5.
The package name on PyPI was changed from `msgpack-python` to `msgpack` in 0.5.

When upgrading from msgpack-0.4 or earlier, do `pip uninstall msgpack-python` before
`pip install -U msgpack`.
Expand All @@ -218,25 +218,25 @@ When upgrading from msgpack-0.4 or earlier, do `pip uninstall msgpack-python` be

* Python 2 support

* The extension module does not support Python 2 anymore.
* The extension module no longer supports Python 2.
The pure Python implementation (`msgpack.fallback`) is used for Python 2.

* msgpack 1.0.6 drops official support of Python 2.7, as pip and
GitHub Action (setup-python) no longer support Python 2.7.
GitHub Action "setup-python" no longer supports Python 2.7.

* Packer

* Packer uses `use_bin_type=True` by default.
Bytes are encoded in bin type in msgpack.
* The `encoding` option is removed. UTF-8 is used always.
Bytes are encoded in the bin type in MessagePack.
* The `encoding` option is removed. UTF-8 is always used.

* Unpacker

* Unpacker uses `raw=False` by default. It assumes str types are valid UTF-8 string
and decode them to Python str (unicode) object.
* Unpacker uses `raw=False` by default. It assumes str values are valid UTF-8 strings
and decodes them to Python str (Unicode) objects.
* `encoding` option is removed. You can use `raw=True` to support old format (e.g. unpack into bytes, not str).
* Default value of `max_buffer_size` is changed from 0 to 100 MiB to avoid DoS attack.
* The default value of `max_buffer_size` is changed from 0 to 100 MiB to avoid DoS attacks.
You need to pass `max_buffer_size=0` if you have large but safe data.
* Default value of `strict_map_key` is changed to True to avoid hashdos.
You need to pass `strict_map_key=False` if you have data which contain map keys
which type is not bytes or str.
* The default value of `strict_map_key` is changed to True to avoid hash DoS.
You need to pass `strict_map_key=False` if you have data that contain map keys
whose type is neither bytes nor str.
4 changes: 2 additions & 2 deletions msgpack/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@
from .exceptions import * # noqa: F403
from .ext import ExtType, Timestamp

version = (1, 1, 1)
__version__ = "1.1.1"
version = (1, 1, 2)
__version__ = "1.1.2"


if os.environ.get("MSGPACK_PUREPYTHON"):
Expand Down
3 changes: 2 additions & 1 deletion msgpack/_cmsgpack.pyx
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# coding: utf-8
#cython: embedsignature=True, c_string_encoding=ascii, language_level=3
#cython: freethreading_compatible = True
import cython
from cpython.datetime cimport import_datetime, datetime_new
import_datetime()

Expand Down
10 changes: 8 additions & 2 deletions msgpack/_packer.pyx
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
# coding: utf-8

from cpython cimport *
from cpython.bytearray cimport PyByteArray_Check, PyByteArray_CheckExact
from cpython.datetime cimport (
Expand Down Expand Up @@ -129,6 +127,7 @@ cdef class Packer:
if self.exports > 0:
raise BufferError("Existing exports of data: Packer cannot be changed")

@cython.critical_section
def __init__(self, *, default=None,
bint use_single_float=False, bint autoreset=True, bint use_bin_type=True,
bint strict_types=False, bint datetime=False, unicode_errors=None,
Expand Down Expand Up @@ -269,6 +268,7 @@ cdef class Packer:
return ret
return self._pack_inner(o, 0, nest_limit)

@cython.critical_section
def pack(self, object obj):
cdef int ret
self._check_exports()
Expand All @@ -284,13 +284,15 @@ cdef class Packer:
self.pk.length = 0
return buf

@cython.critical_section
def pack_ext_type(self, typecode, data):
self._check_exports()
if len(data) > ITEM_LIMIT:
raise ValueError("ext data too large")
msgpack_pack_ext(&self.pk, typecode, len(data))
msgpack_pack_raw_body(&self.pk, data, len(data))

@cython.critical_section
def pack_array_header(self, long long size):
self._check_exports()
if size > ITEM_LIMIT:
Expand All @@ -301,6 +303,7 @@ cdef class Packer:
self.pk.length = 0
return buf

@cython.critical_section
def pack_map_header(self, long long size):
self._check_exports()
if size > ITEM_LIMIT:
Expand All @@ -311,6 +314,7 @@ cdef class Packer:
self.pk.length = 0
return buf

@cython.critical_section
def pack_map_pairs(self, object pairs):
"""
Pack *pairs* as msgpack map type.
Expand All @@ -331,6 +335,7 @@ cdef class Packer:
self.pk.length = 0
return buf

@cython.critical_section
def reset(self):
"""Reset internal buffer.

Expand All @@ -339,6 +344,7 @@ cdef class Packer:
self._check_exports()
self.pk.length = 0

@cython.critical_section
def bytes(self):
"""Return internal buffer contents as bytes object"""
return PyBytes_FromStringAndSize(self.pk.buf, self.pk.length)
Expand Down
Loading