Skip to main content

Module metrics

Module metrics 

Source
Expand description

OTLP metrics -> Arrow. The widest of the three signals, and the one where the layout choice is worth the most.

§Why four point tables instead of one

OTLP has five metric types carrying four incompatible point shapes. The obvious layout is one wide data_points table with every column any point type might need and nulls everywhere else. Measured on 300,000 points in the usual mix (90% number, 8% histogram, 1% exponential, 1% summary):

one wide table   170.6 B/point
four split tables 73.0 B/point   2.34x

Nulls are not free in Arrow. A histogram’s bucket_counts list column still costs an offset entry on every one of the 270,000 number points that will never have buckets, and the validity bitmaps stack up column by column. The four tables also let a “graph this counter” query touch number_dp alone.

Two more measurements shaped the histogram tables. Flattening bucket_counts into a child table costs 1.47x what the List<UInt64> column costs, because a child row pays a 4-byte parent id per bucket where the list pays one 4-byte offset per point. And interning explicit_bounds into a side table takes hist_dp from 410 to 246 B/row — every point of a histogram repeats the same boundaries, which is what makes it the same histogram.

§One id space for points

dp_attrs and exemplars both key on a data point, and a point lives in one of four tables. Rather than a discriminant column saying which, the four tables draw id from a single counter, so a point id names exactly one row in exactly one table. Attribute filtering — which is 68% of a metrics block by size, measured — is then one semi-join instead of four.

Ids stay ascending within each table, so the join is a binary search rather than the direct index the logs and spans tables allow. Points of one metric arrive together, so in practice the ids being searched are a contiguous run.

Structs§

DpHead 🔒
The five columns every point table starts with. Grouped so that the four tables cannot drift apart, which would make a temporal filter four functions.
MetricsBuilder
Stats 🔒
count/sum/min/max, shared by the three aggregating point types.

Functions§

clamp_u8 🔒
Enums arrive as i32 and a client can send anything. Out of range becomes the zero variant, which every OTLP enum defines as “unspecified”.
for_each_point 🔒
Visit (attribute_count, exemplar_attribute_count) for every point of m, whichever of the five shapes it has. Exists so has_headroom_for does not repeat the five-arm match that append_metric already has.