Correctness
Gated on someone else's implementation
A format library that only passes its own tests has proved nothing, so I gated every format tin here against the reference implementation of the format — in both directions where the format has two.
Iceberg
Row-sets cell-exact against PyIceberg 0.11.1 and DuckDB 1.5.5 across 48 of 48 filter cases, in both directions, including deletes, deletion vectors and schema evolution. Tables magmalake writes are read back cell-exact by both — and PyIceberg can append to a table magmalake created.
Parquet
Value-exact against pyarrow on 33 fixtures — every value of every column — and
pyarrow reads back every file magmalake writes. The Arrow export imports through
pyarrow.Array._import_from_c.
Avro & Thrift
Avro round-trips against fastavro both ways across all four codecs. Thrift is byte-identical to Apache Thrift on generated wire vectors, in both the compact and the binary protocol.
S3 & SigV4
37 of 37 cases of the official AWS SigV4 test suite, every stage. S3 is verified end-to-end against MinIO in CI, not only on a developer's machine.
Roaring
Byte-exact against pyroaring in both directions, with pyroaring-produced payloads baked into the tests as constants so the oracle cannot drift.
Everything else
Codecs against Python zstandard, lz4,
brotli and python-snappy; hashes against zlib, mmh3 and
xxhash plus the Iceberg spec's own vectors; threads against contended-count and
memory-visibility proofs.
Building it found bugs upstream
Holding three implementations against each other surfaces disagreements, and not all of them turned out to be my fault. I cut each one below down to a standalone script or unit test with no magmalake code in it, then either reported it upstream or — where someone had got there first — matched it to the report that already existed.
- Apache Arrow apache/arrow#51097 reported The Parquet writer undercounts nulls for a fixed-width leaf under list<struct> when some lists are null or empty. The BYTE_ARRAY leaf beside it in the same struct is right, so two leaves disagree about the same level records.
- iceberg-rust apache/iceberg-rust#3118 reported An In predicate whose literals straddle a file's bounds is never pruned: the lower and upper bounds are each tested against the whole literal set instead of narrowing it, as Java and PyIceberg do.
- PyIceberg apache/iceberg-python#3690 reproduced v3 manifests and manifest lists are read with the v2 projection, so first_row_id, referenced_data_file, content_offset and content_size_in_bytes are dropped on the way in.
- PyIceberg apache/iceberg-python#3620 reproduced The mirror image on the write path: the manifest writer builds its record schema at DEFAULT_READ_VERSION, so those same v3 fields would serialise as null.
- PyIceberg apache/iceberg-python#3833 reproduced A null list<struct> or map<K, struct> is rebuilt as an empty one — on the write path too, so the null is gone at rest and no reader can recover it.