Expand description
On-disk Arrow schemas. These follow the OTAP star schema (otap-spec.md
sections 5.4 and 6.3) rather than a flattened one-row-per-span layout:
a root table per signal carrying resource_id/scope_id foreign keys, plus
entity-attribute-value side tables keyed by parent_id.
Two deliberate deviations from the OTAP wire spec, both documented in docs/architecture.md:
parent_idand the rootidare UInt32 and block-local. On the wire OTAP ids are only unique within oneBatchArrowRecords; persisting them verbatim and joining across batches silently produces a cross product. We rebase to a dense per-block id at ingest, so a join inside a block is unconditionally correct and needs no partition discriminant column.- Attribute values are plain, not dictionary-encoded. Dictionary encoding
is a low-cardinality technique with a hard ceiling (
DictionaryKeyOverflowat 2^16 for UInt16 keys); attribute values are the highest-cardinality data in the system. Only enumerable columns (attribute keys, severity text) are dictionaries.
Enums§
- Attr
Type - OTAP attribute value discriminant (
typecolumn). Matches otap-spec.md 5.4. - Metric
Kind - Which
Metric.datavariant a descriptor row carries.
Constants§
- DICT_
CAP - How many distinct values a
UInt16dictionary can hold. Lives here because it is a consequence of the key width chosen above, not of any one builder. Reaching it is a signal to seal the block, never to fail an export. - LOGS_
BLOCK_ TABLES - The tables a logs block is made of, in publish order.
- METRICS_
BLOCK_ TABLES - The tables a metrics block is made of, in publish order.
- TRACES_
BLOCK_ TABLES - The tables a traces block is made of, in publish order.
Statics§
- ATTRS
- Shared shape for LOG_ATTRS / RESOURCE_ATTRS / SCOPE_ATTRS.
- EXEMPLARS
- EXEMPLARS — the bridge from a metric point to the trace that produced it.
- EXP_
HIST_ DP - EXP_HIST_DP — exponential histogram points.
- HIST_
BOUNDS - HIST_BOUNDS — the interned
explicit_boundsarrays of this block. - HIST_DP
- HIST_DP — explicit-bucket histogram points.
- LOGS
- LOGS root table.
- METRICS
- METRICS descriptor table — one row per
Metricmessage, not per point. - NUMBER_
DP - NUMBER_DP — gauge and sum points.
- RESOURCES
- RESOURCES table — one row per distinct resource in the block.
- SPANS
- SPANS root table.
- SPAN_
EVENTS - SPAN_EVENTS — a child table, not a
List<Struct>column. - SPAN_
LINKS - SPAN_LINKS — same child-table reasoning as
SPAN_EVENTS. - SUMMARY_
DP - SUMMARY_DP — the legacy quantile representation, kept because OTLP still carries it out of Prometheus.
Functions§
- dict_
u16_ 🔒utf8 - dict_
u32_ 🔒utf8 - A dictionary wide enough that a block cannot fill it.
- dp_head 🔒
- Columns every data point table has, in the same order, so that a temporal filter is the same code against any of the four.
- list_of 🔒
- ts 🔒