Expand description
OTLP metrics -> Arrow. The widest of the three signals, and the one where the layout choice is worth the most.
§Why four point tables instead of one
OTLP has five metric types carrying four incompatible point shapes. The
obvious layout is one wide data_points table with every column any point
type might need and nulls everywhere else. Measured on 300,000 points in the
usual mix (90% number, 8% histogram, 1% exponential, 1% summary):
one wide table 170.6 B/point
four split tables 73.0 B/point 2.34xNulls are not free in Arrow. A histogram’s bucket_counts list column still
costs an offset entry on every one of the 270,000 number points that will
never have buckets, and the validity bitmaps stack up column by column. The
four tables also let a “graph this counter” query touch number_dp alone.
Two more measurements shaped the histogram tables. Flattening bucket_counts
into a child table costs 1.47x what the List<UInt64> column costs, because a
child row pays a 4-byte parent id per bucket where the list pays one 4-byte
offset per point. And interning explicit_bounds into a side table takes
hist_dp from 410 to 246 B/row — every point of a histogram repeats the same
boundaries, which is what makes it the same histogram.
§One id space for points
dp_attrs and exemplars both key on a data point, and a point lives in one
of four tables. Rather than a discriminant column saying which, the four
tables draw id from a single counter, so a point id names exactly one row
in exactly one table. Attribute filtering — which is 68% of a metrics block
by size, measured — is then one semi-join instead of four.
Ids stay ascending within each table, so the join is a binary search rather than the direct index the logs and spans tables allow. Points of one metric arrive together, so in practice the ids being searched are a contiguous run.
Structs§
- DpHead 🔒
- The five columns every point table starts with. Grouped so that the four tables cannot drift apart, which would make a temporal filter four functions.
- Metrics
Builder - Stats 🔒
count/sum/min/max, shared by the three aggregating point types.
Functions§
- clamp_
u8 🔒 - Enums arrive as
i32and a client can send anything. Out of range becomes the zero variant, which every OTLP enum defines as “unspecified”. - for_
each_ 🔒point - Visit
(attribute_count, exemplar_attribute_count)for every point ofm, whichever of the five shapes it has. Exists sohas_headroom_fordoes not repeat the five-arm match thatappend_metricalready has.