magmalake
GitHub

An exploration of running data workflows on Mojo

A data lake that lives
natively in Mojo

An experiment: can a modern data stack — Apache Iceberg and everything underneath it — exist in Mojo without reaching back through Python or the JVM? I built twelve small tins to find out. Each is independently usable, and each is checked against the reference implementation of the format it claims to speak. 

from iceberg import Catalog

var table = Catalog.rest("https://polaris.example/api/catalog").load("db.events")
var rows = table.scan().filter('["=", "region", "eu"]').select(["id", "amount"]).to_table()

Metadata, manifests, scan planning, Parquet decode, deletes and deletion vectors, S3 IO and the REST catalog — all of it Mojo.

Browse the org on GitHub See the twelve tins
12
tins on mojoshelf
v1–v3
Iceberg formats read
232M
Parquet rows/s, one core
0
JVM or Python in the path

What this is

An experiment, not a product

Now that Mojo has reached 1.0 I used Claude to see if one can implement a meaningfully large data library in Mojo. I set my sights on Apache Iceberg. The following formats are implemented rather than wrapped: Parquet, Avro, Thrift, Iceberg itself, Roaring bitmaps, Snappy and the hashes all decode bytes in Mojo instead of delegating. Some very mature C libraries are currently linked, not reimplemented — libzstd and liblz4 for those two codecs, libcurl for HTTP and TLS, and pthreads for the threads the language does not yet provide. What is gone is the Python and JVM layer that every other route from Mojo to Iceberg has to go through.

What came out is twelve tins. Some of the answers are encouraging — native Parquet decode is faster than pyarrow on a single core. Some are not: writes are slower, thread-pooled scans lose, and the language is still missing pieces a data stack assumes. I don't think the misses are limitations of the approach or the tech stack, it is a matter of putting in more time and tokens.

The stack

Twelve tins, one layering

Iceberg sits on Parquet, Avro, Roaring and objectstore; those sit on Thrift, the compression codecs and the hashes; threads runs alongside all of it. I kept them separate, so you can take just the one you need.

Table format
iceberg metadata · scan planning · read · write · commit v1–v3 · deletes · deletion vectors · schema evolution
File formats & storage
parquet read · write · Arrow C Data out avro object container files, both ways roaring 32/64-bit · DV blob framing objectstore local · HTTP · S3 · GCS · Azure · SigV4
Primitives
thrift compact + binary protocol
zstd lz4 snappy compression codecs
hashes CRC-32 · murmur3 · XXH64 · SHA-256
threads pthreads · atomics · parallel_for alongside every layer — a stopgap until Mojo ships its own
Each box is an independently usable tin with its own repository, its own tests and its own release cadence. Nothing above depends on anything outside this diagram except libzstd, liblz4, libcurl and the system pthreads.

The tins

What each one is, and what proves it

Every tin is Apache-2.0, published on mojoshelf, and CI-tested on stable Mojo 1.0.0 and the current nightly, across macOS and Linux. Each carries its own correctness oracle — an outside implementation I made it agree with.

Table format

iceberg

0.4.4

Native Apache Iceberg. Read a table's metadata.json, pick a snapshot, decode its manifests, plan a scan and read the rows — then create, append, delete, overwrite and expire. No JVM, no Python, no Rust in the path.

  • Format v1–v3 reads: position and equality deletes, v3 deletion vectors, schema evolution, nested types
  • Writes: create, fast-append, delete (copy-on-write and merge-on-read), overwrite, dynamic partition overwrite, expire_snapshots
  • REST and filesystem catalogs; local, HTTP, S3, GCS and Azure IO
  • Scans 1M rows in 35.8 ms single-core, 12.1 ms across four workers
Checked against PyIceberg 0.11.1 and DuckDB 1.5.5 — cell-exact, both directions

File formats & storage

parquet

0.3.3

The first native Apache Parquet reader and writer in Mojo. It decodes the footer, the page headers, the levels and the values itself, and hands the result back in Arrow memory layout, exportable over the Arrow C Data Interface. That interface is the boundary, not a full Arrow library: there are no compute kernels, no IPC and no Flight.

  • Every physical and logical type, every encoding, v1 and v2 pages
  • Nested lists, maps and structs reconstructed from definition and repetition levels
  • Row-group, page-index and bloom-filter pruning; field-id projection for Iceberg
  • Arrow C Data Interface export, verified by pyarrow importing the arrays
  • Reads 1M rows in 4.3 ms — 232M rows/s, 1.9× pyarrow
Checked against pyarrow, value-exact on 33 fixtures; pyarrow reads back every file it writes

avro

0.3.0

Pure-Mojo Apache Avro: schema parsing, the binary encoding, Object Container Files both ways, and schema resolution. The core has no dependencies at all — not even FFI.

  • Object Container Files, read and write, null / deflate / snappy / zstandard
  • A schema-compiled RecordCursor with no per-record allocation
  • Iceberg field-ids and OCF metadata survive parsing intact
  • Decodes manifest-shaped records at 19.2M/s — 11× fastavro
Checked against fastavro, both directions, across all four codecs

objectstore

0.3.0

Storage and HTTP for Iceberg tables: a FileIO abstraction over local files, HTTP(S) range reads and S3, with the pooled HTTP transport the rest of the stack was missing.

  • S3 with SigV4, vended credentials, presigned URLs and multipart upload
  • Pooled libcurl transport — 0.15 ms per range read on a reused connection
  • Pure-Mojo SHA-256 and HMAC on hardware crypto paths, 2.7 GB/s
  • GCS, Azure and plain HTTP range reads alongside local files
Checked against AWS SigV4 suite 37/37, and S3 verified end-to-end against MinIO in CI

roaring

0.1.0

Pure-Mojo Roaring bitmaps, 32- and 64-bit, implementing the portable RoaringFormatSpec serialization plus Iceberg's deletion-vector v1 blob framing.

  • Bitmap32 and Bitmap64 with array, bitset and run containers
  • Portable serialization including the 64-bit extension
  • Iceberg deletion-vector blob framing with its own CRC-32
  • No dependencies
Checked against pyroaring, byte-exact in both directions

Primitives

thrift

0.1.0

Apache Thrift serialization in pure Mojo — compact and binary protocols — plus every struct, union and enum of the Parquet metadata schema, generated ahead of time.

  • TCompactProtocol and TBinaryProtocol behind one trait
  • All of parquet.thrift, pre-generated
  • Footer, page-header and page-index decode helpers
  • No RPC, no runtime IDL, no dependencies
Checked against Apache Thrift itself — 13 generated wire vectors, byte-identical

zstd

0.1.1

A Mojo binding to libzstd — one-shot and streaming, both directions — through a small C shim loaded at runtime, so consumers need no link flags.

  • One-shot and streaming compress and decompress
  • Frame introspection: is_zstd_frame, frame_content_size
  • The shim is dlopen'd once, not per call
  • 10–14 GB/s decompress
Checked against Python zstandard, against independently produced frames baked in as constants

lz4

0.1.1

A Mojo binding to liblz4 covering the block format, the frame format and the legacy Hadoop framing that older Parquet files still use.

  • LZ4_RAW blocks for Parquet pages
  • LZ4F frames for Iceberg Puffin blobs
  • Hadoop framing for legacy Parquet LZ4
  • 8–19 GB/s
Checked against CPython's lz4 package, on known vectors and round trips

snappy

0.1.1

Snappy in pure Mojo — the raw block format and the CRC-32C-checksummed framing format. No FFI, no C dependency.

  • Raw block format and sNaPpY framing
  • CRC-32C verified per chunk
  • Pure Mojo — nothing to build, nothing to link
  • Up to 20 GB/s incompressible, ~3 GB/s compressible
Checked against python-snappy, byte-exact

hashes

0.1.0

The three hashes Iceberg and Parquet actually need — CRC-32, MurmurHash3 x86-32 and XXH64 — in pure Mojo, with no dependencies and no FFI.

  • CRC-32 for page CRCs and deletion-vector checksums
  • MurmurHash3 for the Iceberg bucket[N] transform
  • XXH64 for Parquet bloom filters
  • 1.2–1.5 GB/s, identical results on every platform
Checked against zlib, mmh3 and xxhash, plus the Iceberg spec's Appendix B vectors

threads

0.1.0

Minimal OS threads for Mojo: spawn and join pthreads, share state through atomics and a mutex, and fan a loop out over cores with parallel_for. A stopgap, distilled from flare, until the language ships its own.

  • parallel_for over cores — Mojo currently ships no other way to use a second one
  • Atomics that bridge the stable/nightly std.atomic split
  • Mutex, spawn, join and thread pinning
  • Spawn and join in 14 µs; parallel_for scales ~4×
Checked against Contended-count and memory-visibility proofs designed to give a wrong number, not a flake

Cross-implementation oracle

iceberg-rs

0.1.0

Apache Iceberg over a thin Rust cdylib wrapping iceberg-rust behind a C ABI. Superseded — no longer required for any operation — and kept only as a third independent implementation to check the native one against.

  • 56 extern "C" functions over one shared tokio runtime
  • SQL/JDBC catalog against real object storage
  • Kept as a cross-implementation oracle, not a dependency
Checked against PyIceberg reading through the same sqlite catalog file the binding wrote

Performance

Measured, including where it loses

I measured all of this on an Apple M4, single core unless a row says otherwise — same machine, same files, reproducible from each repository with pixi run bench. magmalake beats pyarrow on single-core Parquet reads. It loses on writes and on multi-threaded scans, where what is left of the Iceberg gap is mostly libzstd itself.

Table and file formats

Parquet decode is where the native stack pays off; writes and thread-pooled scans are where it does not, yet.

Operation magmalake Reference
Parquet read, 1M rows int64 / double / dictionary 4.3 ms — 232M rows/s pyarrow 8.2 ms — 1.9× faster
Parquet read, 4 cores threads.parallel_for over row groups 660M rows/s memory-bandwidth bound
Parquet write, 1M rows 42 ms pyarrow 31 ms — slower
Parquet footer 1,000 columns × 50 row groups 78 ms read / 8 ms write —
Iceberg scan, 1M rows zstd-compressed 35.8 ms — 12.1 ms on 4 workers pyarrow single-thread 26.8 ms on the same files; PyIceberg 7.9 ms using its thread pool
Iceberg append, 1M rows data files, manifests and commit 233 ms PyIceberg ~150 ms
Iceberg scan planning, 500 manifests 31.9 µs fixed cost per manifest, 11.5 µs of it the file-read floor 21.4 ms 62.3 ms before the schema and plan cache
Avro decode, manifest-shaped records 19.2M records/s — 27.5M with field selection fastavro 1.74M — 11–13× faster
Avro inflate 860 MB/s —

scroll the table sideways →

Primitives

Codecs, hashes and threads — the layer everything above is only as fast as.

Operation magmalake Reference
SHA-256 pure Mojo with ARMv8 crypto / SHA-NI intrinsics 2.7 GB/s — 610 MB/s scalar fallback OpenSSL 3.2 GB/s
zstd / lz4 decompress FFI 10–14 GB/s / 8–19 GB/s —
snappy decompress pure Mojo up to 20 GB/s incompressible, ~3 GB/s compressible —
CRC-32 / murmur3 / XXH64 pure Mojo 1.2–1.5 GB/s —
threads: spawn and join 14 µs — parallel_for scales ~4× memory-bandwidth ceiling

scroll the table sideways →

Object storage

Measured against MinIO on the same machine, so the network is not the story.

Operation magmalake Reference
S3 multipart upload, 16 MB 409 MB/s 53 MB/s when payload hashing was still scalar
HTTP range read, pooled connection 0.15 ms local / 0.52 ms signed S3 19× / 11× faster than a fresh connection per request

scroll the table sideways →

Threads, for now, come from a tin

Mojo's standard library currently has no threading. parallelize was removed and there is no thread pool reachable from user code, so every number above that mentions more than one core goes through magmalake/threads, which wraps pthreads directly via external_call.

That tin is inspired by and distilled from flare — Ehsan Mokhtarian's project, MIT-licensed — which worked out how to get real OS threads out of Mojo in the first place. threads.mojo keeps the parts a data stack needs and drops the rest.

It is explicitly a stopgap. When the language ships its own threading, this tin should stop existing.

How these numbers were found

Four optimisation passes. I profiled before each one instead of guessing, which was just as well — not one of the bottlenecks was where I would have looked first.

  1. A dlopen on every single decompress call About 450 µs per call, spent opening a library that was already open.
  2. SHA-256 running 45× off what the hardware can do The scalar implementation was correct and slow; the CPU had crypto instructions sitting idle.
  3. Avro boxing roughly 60 allocations per record And JSON schema parsing turning out to be 72% of the cost of reading a manifest.
  4. 500 byte-identical manifest schemas, each parsed from scratch Caching the parsed schema and the plan cut scan planning from 62.3 ms to 21.4 ms.

Correctness

Gated on someone else's implementation

A format library that only passes its own tests has proved nothing, so I gated every format tin here against the reference implementation of the format — in both directions where the format has two.

Iceberg

Row-sets cell-exact against PyIceberg 0.11.1 and DuckDB 1.5.5 across 48 of 48 filter cases, in both directions, including deletes, deletion vectors and schema evolution. Tables magmalake writes are read back cell-exact by both — and PyIceberg can append to a table magmalake created.

Parquet

Value-exact against pyarrow on 33 fixtures — every value of every column — and pyarrow reads back every file magmalake writes. The Arrow export imports through pyarrow.Array._import_from_c.

Avro & Thrift

Avro round-trips against fastavro both ways across all four codecs. Thrift is byte-identical to Apache Thrift on generated wire vectors, in both the compact and the binary protocol.

S3 & SigV4

37 of 37 cases of the official AWS SigV4 test suite, every stage. S3 is verified end-to-end against MinIO in CI, not only on a developer's machine.

Roaring

Byte-exact against pyroaring in both directions, with pyroaring-produced payloads baked into the tests as constants so the oracle cannot drift.

Everything else

Codecs against Python zstandard, lz4 and python-snappy; hashes against zlib, mmh3 and xxhash plus the Iceberg spec's own vectors; threads against contended-count and memory-visibility proofs.

Building it found bugs upstream

Holding three implementations against each other surfaces disagreements, and not all of them turned out to be my fault. I cut each one below down to a standalone script or unit test with no magmalake code in it, then either reported it upstream or — where someone had got there first — matched it to the report that already existed.

Status

Active, v0.x

Reads are complete for Iceberg format v1 through v3. Writes cover the common table operations. Several things are genuinely missing, and they are listed here rather than left to be discovered.

Works today

  • Iceberg format v1–v3 reads, with position deletes, equality deletes and v3 deletion vectors
  • Schema evolution and nested types through the read path
  • Fast-append, row-level delete both copy-on-write and merge-on-read
  • Overwrite and dynamic partition overwrite
  • expire_snapshots
  • Filesystem and REST catalogs
  • Local, HTTP, S3, GCS and Azure IO

Not yet

  • Compaction — rewrite_manifests
  • Encryption
  • Brotli
  • Predicates inside list and map elements

The spec's v4 drafts — relative paths and content_stats — are tracked but not implemented.

Take a tin

Each of the 12 repositories is independently usable and independently installable. Start with whichever layer you need — you do not have to take the stack.

github.com/magmalake mojoshelf.org