Module wal 

The write-ahead log, which exists to decouple the acknowledgement from the seal.

Why this exists, given docs/architecture.md section 4 says “No WAL”

section 4’s argument is about recovery, and it is still correct: a block directory is renamed into place atomically, so there is no torn state and nothing for a log to replay. This log is not for recovery. It is for latency.

Before it, an export was acknowledged only after the block containing it was durably published, so under light load the caller waited out max_block_age — a measured p50 of 657 ms and p99 of 2,647 ms (section 11). The block is sealed on a timer because nothing else bounds how long a half-filled block sits there, and that timer became the caller’s latency. With the log in front, the seal triggers are unchanged and nobody is waiting on them: max_block_age goes back to being a statement about the shape of blocks on disk rather than a latency bound.

The durability this buys, stated precisely

An append is acknowledged once write(2) has returned — the bytes are in the kernel’s page cache, not on the platter. That survives everything that kills the process: a panic under panic = "abort", SIGKILL, the OOM killer, a flusher that took the process down with it (section 1). It does not survive power loss or a kernel panic, because the page cache does not.

This is a deliberate choice and it is the reason there is no fsync on the acknowledgement path. On the machine section 11 was measured on, one F_FULLFSYNC costs 4,230 us, so a durably-fsynced ack could not have a p50 below 4.2 ms, let alone a p99 under 5 ms. The same trade is Kafka’s acks=1 and ClickHouse’s default. Wal::sync exists and is called on a timer by a background task, which bounds how much is exposed to a power cut to one sync interval — it is never called by an appender.

There is deliberately no userspace buffering. A BufWriter would batch the syscalls, but bytes sitting in a Vec in this process do not survive the process dying, which is exactly the failure this log is claiming to cover. One write(2) per export at a few tens of microseconds is affordable because an export is a batch and not a record: the append rate is section 11’s “Ingest throughput” row divided by the batch size, and the load generator sends 8,192 records per export. At one connection that row is 629,384 records/s, so 77 appends/s; at the thirty-two-connection plateau, 1,537,875 records/s is 188. Three digits of appends per second is not a rate a syscall per append can be the ceiling of.

The frame is the OTLP request, in its canonical protobuf encoding

Not the bytes off the wire. Mira has three ways in — gRPC, protobuf over HTTP, and KYAML over HTTP — and only one of them still has bytes by the time anything could log them: tonic decodes before the handler is called, and a KYAML body is not protobuf at all. So the frame body is encode_to_vec of the decoded request, which normalises all three transports to one format with one decoder on the replay side.

That costs a re-encode. Measured on an 8,192-record log export (1.29 MiB): encode 1.49 ms at 864 MiB/s, against the decode already in the path at 5.29 ms and 244 MiB/s. section 11 measured the whole engine at 190.6 MiB/s, a 6.8 ms budget for that export, so the log adds about 22%. The alternative — a custom tonic Codec to keep the wire bytes — buys that 22% back for a codec Mira then owns forever, which is the wrong side of principle 1’s trade until something measures it as the bottleneck.

Replay feeds the decoded frame through the same ingest::{logs,traces, metrics} the network path calls, so there is no second decode path to write, to test or to keep in step with the first.

┌────────┬─────┬────────┬─────┬─────────┬────────┬──────────┬────────┐
│ magic  │ ver │ signal │ pad │ seq     │ len    │ body     │ crc32  │
│ 4 B    │ 2 B │ 1 B    │ 1 B │ 8 B     │ 4 B    │ len B    │ 4 B    │
└────────┴─────┴────────┴─────┴─────────┴────────┴──────────┴────────┘
 └──────────────── covered by the CRC ──────────────────────┘

The CRC covers the header as well as the body, so a corrupted length is caught by the checksum rather than by whatever it would otherwise index into. That is the same lesson as the block footer: a length read out of a file is attacker-controlled-equivalent, and validating it against the file’s real extent is not optional.

Recovery, and why there is still no manifest

Replay needs exactly one fact — which frames are already inside a published block — and the block directory carries it, so principle 4 survives intact. Block names gain a fifth field, wal_hi: an exclusive watermark, meaning every sequence of that signal below it is inside some published block. It is not this block’s own maximum, because sibling shards are filling their own blocks from the same log — it is what Wal::watermark_for answers, the lowest still-unpublished sequence this block does not hold, or one past everything the log has handed out when this block holds them all. Boot recovery therefore stays what section 4 says it is: a readdir, the same one the read path already does, with no extra I/O and no metadata store to keep consistent. Each signal keeps its own watermark, and that costs nothing because an OTLP export belongs to exactly one signal — /v1/logs, /v1/traces and /v1/metrics are three endpoints.

What this breaks, and what pays for it

Read-your-writes. section 11 notes it was free, and it was free because the ack waited for the publish — the same rename made the data durable and visible at once. A caller acked here and querying immediately would not see its data until the block seals, which is a latency the log was built to remove.

The repair is the open-block query surface: crate::query::search_open takes the flusher’s in-progress builder as a snapshot and scans it alongside the sealed blocks, so read-your-writes holds with the log on or off. It is not free — that snapshot is the one allocation the read path makes — and section 7.6 is why it is paid there rather than in the scan.

Structs

Frame 🔒

FrameReader 🔒

Reads frames out of one segment, stopping at the first that is not whole.

Inner 🔒

Replayed

What a Wal::replay did, for the line it gets logged on.

Wal

An append-only log of OTLP export bodies.

Enums

Signal

Which signal a frame belongs to.

Constants

CRC_LEN 🔒

HEADER_LEN 🔒

MAGIC 🔒

MIRAWAL0, truncated. Present so a stray file in the WAL directory is rejected by name rather than parsed as a frame.

MAX_FRAME_BYTES 🔒

The largest frame that will be written or believed on read.

SEGMENT_BYTES 🔒

Roll to a new segment past this size. Segments are the unit of deletion, so this trades how much dead log is retained past its watermark against how many files the directory holds. At section 11’s measured 190.6 MiB/s a segment is about a third of a second of ingest.

WAL_VERSION

Bumped when the frame layout changes. A reader that does not recognise a version refuses the segment rather than guessing at its shape, which is the same rule the block format follows.

Functions

read_exact_or_eof 🔒

true if the buffer was filled, false on a clean EOF before any byte.

Type Aliases

Watermarks

One past the last published sequence, per signal, indexed by Signal::index. Assembled from the block directory listing at boot; see the module docs.