Schema Representations

MiniPB supports 3 schema representations natively: Message class, format string and key-value format list.

Message class

MiniPB provides a special Message class that can be inherited to create a class capable of serializing/deserializing the state of their instances.

Translating a proto into a Message class:

    message Person {
      required string name = 1; // *U
      required int32 id = 2; // *t
      optional string email = 3; // U

      // enums are not supported natively in minipb. Use enums.IntEnum with a int field instead.
      enum PhoneType {
        MOBILE = 0;
        HOME = 1;
        WORK = 2;
      }

      message PhoneNumber { // [
        required string number = 1; // *U
        optional PhoneType type = 2 [default = HOME]; // t
      } // ]

      repeated PhoneNumber phone = 4; // +[*Ut]
    }

from minipb import TYPE_STRING, TYPE_INT, process_message_fields, Message, Field

@process_message_fields
class Person(Message):
    name   = Field(1, TYPE_STRING, required=True)
    id     = Field(2, TYPE_INT, required=True)
    email  = Field(3, TYPE_STRING)

    @process_message_fields
    class PhoneNumber(Message):
        number = Field(1, TYPE_STRING, required=True)
        type   = Field(2, TYPE_INT)

    phone  = Field(4, PhoneNumber, repeated=True)

Supported Field Types

MiniPB Type	Protobuf type	Python type	Comments
TYPE_SFIXED32	sfixed32	int	32-bit signed integer, little endian
TYPE_FIXED32	fixed32	int	32-bit unsigned integer, little endian
TYPE_SFIXED64	sfixed64	int	64-bit signed integer, little endian
TYPE_FIXED64	fixed64	int	64-bit unsigned integer, little endian
TYPE_FLOAT	float	float	32-bit float
TYPE_DOUBLE	double	float	64-bit float
TYPE_BYTES	bytes	bytes	"ASCII" type. This was a reference to Ruby Array#pack directive rather than Python struct format string.
TYPE_BOOL	bool	bool
TYPE_SINT	sint32, sint64	int	Signed vint encoded with zigzag (2)
TYPE_INT	int32, int64	int	Signed vint encoded with two's complement (1)
TYPE_UINT	uint32, uint64	int	Unsigned vint encoded with two's complement (2)
TYPE_STRING	string	str	"Unicode" type (contrary to a) strings. UTF-8 encoding is assumed
TYPE_EMPTY	empty field

Supported Field Options (True/False)

MiniPB Option	Comments
required	Is field required? NOTE: proto3 no longer supports required fields
repeated	Is field repeated?
repeated_packed	Is field repeated and should be packed? Within MiniPB, recommended only for numerical types

Format String

In this mode the schema is represented in a similar format as struct.Struct would accept, with some key differences:

• There is no option to specify byte order and alignment since it is not necessary (Protobuf data fields are always little endian)
• The field copy count works the same as struct's repeat count, but it is a suffix (i3) instead of a prefix (3i)
• In addition to the suffixes, there are also prefixes that specify field properties (like required, repeated and repeated [packed=true]).
• Although some of the type characters are interchangable with the format characters in struct, one should NOT assume that every struct format character works the same way as the MiniPB type character.

Initialize the Wire object with a given format_string by doing

minipb.Wire(format_string)

Syntax

The format string generally follows the following syntax:

[prefix]<type-character>[suffix]...

Prefixes

Prefix	Protobuf equivalent	Comments
*	required ...	(1)
+	repeated ...	(2)
#	repeated ... [packed=true]	(3)
[	message <name> { ... }; <name> ...	(4)

(1): The field that has this prefix cannot be set as None.

(2): Accepts/returns a tuple of objects. Each repeated field can only have one type of data present (like an array).

(3): Although it is currently possible to use packed repeated field with strings/bytes in MiniPB, it is not recommended since Google's Protobuf implementation does not allow packed strings/bytes due to compatibility between packed and non-packed fields which MiniPB does not currently support.

(4): Embedded message. Must be in the format of [<format_string_for_embedded_message>]. Accepts/returns a tuple of objects that matches the enclosed format string/schema. Must end with ]. Can be used with the other prefixes (in the format of e.g. +[...])

Types

The type-character represents primitive Protobuf types and can be one of the following:

Type	Protobuf type	Python type	Comments
i	sfixed32	int	32-bit signed integer, little endian
I	fixed32	int	32-bit unsigned integer, little endian
q	sfixed64	int	64-bit signed integer, little endian
Q	fixed64	int	64-bit unsigned integer, little endian
l	sfixed32	int	Alias of i
L	sfixed32	int	Alias of I
f	float	float	32-bit float
d	double	float	64-bit float
a	bytes	bytes	"ASCII" type. This was a reference to Ruby Array#pack directive rather than Python struct format string.
b	bool	bool
z	sint32, sint64	int	Signed vint encoded with zigzag (2)
t	int32, int64	int	Signed vint encoded with two's complement (1)
T	uint32, uint64	int	Unsigned vint encoded with two's complement (2)
U	string	str	"Unicode" type (contrary to a) strings. UTF-8 encoding is assumed
u	string	str	Alias of U (since 2.0.0)
v	sint32, sint64	int	Alias of z
V	uint32, uint64	int	Alias of T
x	empty field

(1): Encoding negative integers with this type is generally not recommended since the serialized result will take the full 64 bit space (10 bytes vint) when the absolute value is small. However some protocols do use it when the negative value case is very occasional.

(2): MiniPB does not enforce 32-bit and 64-bit constraints on any unsigned vints (T) and signed vints that use zigzag encoding (z). If maximum compatibility with Protobuf is desired, manual bitmasking before serializing (e.g. via value &= 0xffffffff) might be a good idea.

Suffixes

Currently there are 2 types of suffixes: field copy count and field seek (since 2.0.0). They must be used in this order:

[field_copy_count][@<field_seek>]

Field copy count duplicates the type to several adjacent fields (e.g. i3 is the same as iii and both set the field 1, 2 and 3 to the type sfixed32).

Field seek specifies which ID is used for the current field. It also resets the starting point for subsequent fields (e.g. v@3U will define a sint32|sint64 at field 3 and a string at field 4). Overlapping fields are not allowed and will cause the parser to raise a BadFormatString exception. In format string mode, the fields are mapped to Python tuple in the order they show up (e.g. v@3v@1 on input (1, 2) will set field 1 to 2 and field 3 to 1). In key-value format list mode, the fields are mapped to the keys they belong to (if it's not obvious enough).

In addition to being a suffix, field seek can also be used after an embedded message in format string mode (e.g. [v]@2). Similarly, in key-value format list mode, appending field seek to embedded message type is also possible. Refer below for the exact format.

Key-value format list

Key-value format list uses the same type representation system as the format string, while adding extra information so that it is possible for Wire to accept and output dictionaries instead of tuples.

The format is similar to Structure._fields_ attribute in ctypes, in the format of

schema = (
    (key, type), # Variant 1: normal field, or
    (key, other_schema), # Variant 2: embedded message, or
    (key, '{prefix}[{@<field_seek>}', other_schema), # Variant 3: embedded message with required/repeated property and/or field seek
    ...
)

where the key matches the dictionary key in the input and output object.

In the first variant, type is a string that follows the syntax of

[prefix]<type-character>[@<field_seek>]

where type-character can be any valid type characters listed in the previous section except the x (skip) type. The prefix can be any valid prefixes except [ (start of embedded message) since embedded messages are handled with Variant 2 and 3. In other words, type is a format string that describes exactly one non-skipped and primitive-typed field. Field copying is also implicitly disallowed due to obvious reasons.

To represent consecutive empty fields, one can set the type to

x[count][@<field_seek>]

where the count is the number of consecutive fields that are marked empty. If no count specified it defaults to 1 (i.e. works exactly like field copy count). In this case key is also ignored since there will be no keys assigned to intentionally skipped fields.

To add an embedded message, simply replace type with another format list (Variant 2). In case of a required or repeated embedded message field or field seeking, set the type to <prefix>[ (or [ if only doing field seek), followed by @<field_seek> (if needed) and add the format list after type (Variant 3). In this case prefix can be any valid prefixes except for [. Also note that using # on embedded messages is possible but non-standard and discouraged, as per Differences between MiniPB and Official Protobuf.