Module block 

Immutable block storage.

A block is a directory holding one Arrow IPC file per table:

<root>/logs/p=<epoch_hour>/<min_ts:020>-<max_ts:020>-<node:08x>-<seq:012>-<wal_hi:020>/
    logs.arrow  log_attrs.arrow  resources.arrow  resource_attrs.arrow  scope_attrs.arrow

The filesystem is the manifest. Every reason a LSM engine needs a MANIFEST file is absent here: Mira publishes exactly one immutable object per commit via one directory rename, never mutates a published block, and never needs a multi-file atomic operation. The directory name carries the full pruning key, so booting is one readdir per partition with zero file opens, and there is no metadata state that can disagree with the data. That is what makes the process stateless in the sense that matters: kill it, restart it, point a second one at the same directory read-only — nothing to reconcile.

Publish is write-tmp / fsync-files / fsync-tmpdir / rename-dir / fsync-parent / fsync-grandparent — the last because the partition directory itself is a new entry in <root>/<signal> on the first block of every hour. Directory rename is atomic on POSIX, so a block is either wholly visible or wholly absent; there is no torn state for recovery to clean up, and therefore no write-ahead log.

Retention is remove_dir_all on the block directory. POSIX guarantees this is safe against in-flight readers: a mapping holds a reference to the inode that close() does not drop, so a query holding an Arc<Mmap> keeps reading correct data from an unlinked file until it drops the mapping. The Arc is the refcount; no lease protocol is needed.

Structs

BlockRef

A published block, discovered by reading directory names.

CrcWriter 🔒

A Write that hashes everything passing through it.

Framed 🔒

One encapsulated IPC message, described entirely by the checksummed bytes it starts at.

MappedTable

A table read straight out of its mapping.

Src 🔒

One candidate the read path may have to open: a published block directory, or the snapshot of a block that is still open (section 4).

Constants

ALIGNMENT 🔒

Buffer alignment for published blocks.

COLD_AFTER_NS

A block goes cold once it has aged out of the hour it was partitioned into.

COLD_MARKER 🔒

The cold-tier marker. Its presence means every table in the block is already ZSTD-encoded, so a sweep can skip the directory without opening a file.

CONTINUATION 🔒

The prefix of every encapsulated IPC message since the legacy format was retired. We write V5 with write_legacy_ipc_format off, so a message that does not start with it is not a message this wrote.

CRC_KEY 🔒

CRC32 of the record-batch body, stored in the footer’s custom metadata along with the exact byte length it covers. arrow-ipc has no checksum of its own: a valid footer over a corrupt body decodes silently into wrong answers.

CRC_LEN_KEY 🔒

FORMAT_KEY 🔒

On-disk format version, stamped into every table’s footer metadata beside the CRC — a place that already exists, so no second file and no second fsync. Read before the reader trusts anything else in the file.

FORMAT_VERSION

The format this binary writes, and the highest it will read.

HEADER_LEN 🔒

Where the schema message starts: ARROW1 padded up to ALIGNMENT, which is exactly how FileWriter places it (pad_to_alignment over the same constant we hand it in write_table_with).

LEGACY_VERSION 🔒

The version blocks written before FORMAT_KEY existed are treated as.

MAGIC 🔒

Leading bytes of every Arrow IPC file. arrow-rs’s own reader seeks straight to the trailer and never checks this, so a truncated-from-the-front file would decode as garbage; we check it ourselves.

MAX_COMPACT_PER_SWEEP 🔒

ponytail: a flat cap per sweep, so the first pass over an existing volume drains at a few hundred MB a minute instead of saturating the disk for an hour. Make it adaptive when a real deployment says the backlog matters.

NANOS_PER_HOUR 🔒

ZSTD_LEVEL 🔒

The ZSTD level every compressed table is written at.

Functions

check_filesystem

Refuse to start on a filesystem the read path cannot survive.

check_format 🔒

Decide whether this binary is allowed to read a block declaring declared as its FORMAT_KEY.

check_fs_type 🔒

The decision check_filesystem makes, split from the mount it makes it about for the same reason check_format takes a string rather than a file: the rule is the part that can be wrong, and a test cannot conjure an NFS mount to state it over.

check_writable

Refuse to start on a data directory that cannot be written to.

compact

Rewrite aged blocks ZSTD-compressed, in place.

compact_block 🔒

corrupt 🔒

A block file whose own metadata does not describe it.

dir_name 🔒

{min_ts}-{max_ts}-{node}-{seq}-{wal_hi}.

expire

Drop every block whose newest row is older than cutoff_ns.

free_fraction

How much of the filesystem holding path is still free, as a fraction of its total size.

fs_type 🔒

The mount’s filesystem type, if it is one of the ones that matter. None means “nothing to say about it”, which is every local filesystem.

fsync_dir 🔒

message_at 🔒

Frame the message at offset, bounds-checking every number it declares against the region the CRC covers.

node_id

A replica’s writer identity, derived from its name with no coordination.

open_table

Open one table of a block with no buffer copies.

open_table_opt

open_table, but a missing file means an empty table rather than an error.

parse_dir_name 🔒

Parses both the five-field name above and the four-field name that predates the write-ahead log.

publish

Atomically publish a set of tables as one block under <root>/<signal>/.

scan

Rebuild the catalog from the filesystem. This is the entire boot sequence for the read path: no manifest to replay, and the write-ahead log’s recovery point rides in the block names rather than in a file of its own.

sources 🔒

The published blocks under root/signal, plus any open-block snapshots that have not yet been published under the same (node, seq).

stage 🔒

Write one sealed block’s files into an already-created staging directory.

statfs 🔒

One filled statfs, shared by the two things that ask the mount a question.

sweep_staging

Remove staging directories this node left behind for this signal.

unwind_staging 🔒

Unwind the staging directory of a publish that did not complete.

wal_watermarks

How far the write-ahead log has been absorbed into published blocks, per signal, in the order crate::wal::Signal::index uses.

write_table

Write one table as a self-contained, checksummed, 64-byte-aligned IPC file and fsync it. Uncompressed on purpose: IPC body compression forces the reader to decompress into fresh allocations, which is mutually exclusive with the zero-copy mmap read path.

write_table_lz4

The same again as LZ4_FRAME, so the tier example can price the pure-Rust alternative against the C one on real blocks. Nothing in the engine writes LZ4; see the decisions table in docs/architecture.md.

write_table_with 🔒

write_table_zstd

The same file, ZSTD-compressed per buffer.