Serde

The serde types encode and decode in Rust. serialize and deserialize return an Either, so a bad payload is an Err you handle, not an exception that escapes.

JSON

from pyrula import JsonSerde

serde = JsonSerde()

serde.serialize({"id": 1, "name": "alice"})   # Ok(b'{"id":1,...}')
serde.deserialize(b'{"id": 1}')               # Ok({"id": 1})
serde.deserialize(b'not json')                # Err(DeserializationError(...))

Decoding gives what json.loads gives: objects keep their keys in the order they were written, integers of any size come back exact, and a number too large for a float is inf. The difference is that input which is not JSON is refused where json.loads would accept it: the NaN and Infinity literals and unpaired surrogate escapes such as "\ud800". Arrays and objects nested more than about 200 deep are refused too. The error names the line and column where the input went wrong.

Avro

AvroSerde takes a schema as a JSON string and validates against it.

from pyrula import AvroSerde

serde = AvroSerde(schema_json=SCHEMA)
serde.serialize({"id": 1, "name": "alice"})   # Ok(bytes)

Records and maps are dicts, arrays are lists, string and enum are str, bytes and fixed are bytes, float and double are float, and a union holds whichever of its branches the value is (None for null). Map keys come back in the order they were written. A named type used more than once, or inside itself as in a linked list, works both ways.

Logical types read and write byte for byte the way fastavro does (and so the way confluent-kafka’s AvroSerializer and AvroDeserializer do):

Logical type	Decodes to	Encodes from
timestamp-millis, timestamp-micros	datetime in UTC	datetime (a naive one is local time, as fastavro takes it), or the int
local-timestamp-millis, local-timestamp-micros	naive datetime	datetime (its wall clock, whatever the zone), or the int
date	date	date, an ISO str, or the int
time-millis, time-micros	time	time, or the int
decimal (bytes or fixed)	Decimal	Decimal within the schema’s precision and scale, or the raw bytes
uuid	UUID	UUID, or a str

fastavro has no conversion for the nanosecond timestamps or duration, so neither has AvroSerde: they are the int and the twelve bytes on the wire. big-decimal is not supported: using one gives an Err, and a schema that puts uuid on a fixed is refused when the serde is built.

In a union, a string goes to a date branch when it parses as an ISO date and on to a later string branch otherwise, as fastavro chooses. Where this differs from fastavro: Decimal("-0") on a fixed decimal writes zero, where fastavro writes a value that reads back as the smallest negative step; and a registry serde given a reader schema converts by the reader’s logical types where fastavro goes by the writer’s, and cannot resolve a bytes decimal into a reader schema at all (an apache-avro limitation).

Two keyword options change this, on AvroSerde and AvroRegistrySerde alike:

AvroSerde(SCHEMA, strict=True)
AvroSerde(SCHEMA, logical_types="raw")

strict=True refuses the conversions that fastavro makes by guessing. A naive datetime for a timestamp-* field is an Err, because it names no instant (use an aware one, or a local-timestamp-* field for a wall-clock time). A datetime for a date field is an Err, because its time would be dropped. A decoded decimal with more digits than its schema’s precision is an Err where fastavro rounds it. Use it when both ends of the topic are yours.

logical_types="raw" decodes every logical type to its value on the wire: the int for dates, times and timestamps, the unscaled bytes for a decimal, and the str for a uuid. Encoding still takes either form. Use it for pass-through, or when the values go on to something that wants the numbers.

Decoding refuses bytes that end part way through a value rather than filling the missing part in. Values nested more than 500 deep are refused in both directions, which only a recursive schema can reach.

Batches

The _many methods take an IList and run in one Rust call. serialize_many returns an IList of per-item results. The packed variant returns a BytesList: serialize_many_raw on JsonSerde, serialize_many_bytes on the Avro serdes.

from pyrula import IList

rows = IList([{"id": 1}, {"id": 2}])
json_serde.serialize_many(rows)        # IList([Ok(b'{"id":1}'), Ok(b'{"id":2}')])
packed = json_serde.serialize_many_raw(rows)   # BytesList
json_serde.deserialize_many_raw(packed)        # IList([{'id': 1}, {'id': 2}])

avro_serde.serialize_many_bytes(rows)  # BytesList

Errors

Everything raises through one hierarchy when you do unwrap it:

from pyrula import SerdeError, SerializationError, DeserializationError

SerializationError and DeserializationError both subclass SerdeError, so you can catch one or all three.

Schema Registry

For Confluent Schema Registry framing (the [0x00][schema_id][payload] wire format), there are registry-aware serdes: AvroRegistrySerde, JsonSchemaRegistrySerde, and ProtobufRegistrySerde. They handle registration, schema-id lookup, and TLS/mTLS to the registry. Those live on the Kafka path. See Kafka.

AvroRegistrySerde decodes the way AvroSerde does, with one exception: when you pin a reader_schema that differs from the schema a record was written with, the record is resolved into the reader’s shape by apache-avro, the reference implementation of Avro’s schema-evolution rules, and apache-avro does not keep map keys in the order they were written.