Skip to main content

Module alert

Module alert 

Source
Expand description

Static-rule alerting: a KYAML document in, webhooks out.

§A rule is a query, not a new language

Every alerting system this replaces ships a second language — PromQL, a DSL, a builder UI — and that language is the reason “why did this page?” is hard to answer: the thing that fired is not the thing you can look at. Here a rule embeds a /api/v1/query document, verbatim. The operator builds the query in the UI, presses nothing, pastes it into the rules file, and the link in the resulting page opens the same query in the same UI. There is one filter grammar in this product and this is not a second one.

§Two metrics, and why that is enough

count is how many records matched. ratio is that over a second query’s count. Between them they express both of the rules everyone actually writes:

  • error rate above 5% over a minute — numerator status_code = 2, denominator the same filter without it, ratio > 5%.
  • p95 latency above 500 ms — numerator adds duration_nano > 500000000, same denominator, ratio > 5%.

The second one is not an approximation of a quantile, it is the quantile: p95(d) > T and |{d > T}| / |d| > 0.05 are the same statement. That identity is why there is no digest in this file and no p95(...) in the grammar — a percentile threshold was always a counting question wearing a statistics hat, and Mira can already count exactly.

§Counting is free

query::search_open with limit: 0 returns no rows and an exact rows_matched: the early exit needs a held hit to fire and there are none, so every matching block is scanned, and the per-wave trim throws the hits away before they accumulate. So an evaluation costs one scan and allocates nothing per match — the same code path, the same zone maps and the same bloom filters the UI’s query uses, which is also why a rule cannot drift from what the operator sees.

§One replica evaluates

Principle 4 is no coordination state, and N replicas reading the same block directory would each fire the same rule N times. There is no lease and no leader here: alerting is off unless alerts.rules names a file, so the deployment enables it on one replica and that is the whole mechanism. State — which rules are firing, and since when — is in memory, so a restart re-evaluates from scratch and re-pages anything still breaching.

ponytail: dedup is the operator pointing one replica at the rules file. A lease file in the block directory is the upgrade path if that stops being enough, and it stays inside “the block directory is the manifest”.

§Why HTTPS is a Cargo feature

Slack, Discord and PagerDuty are all HTTPS, and in-process TLS costs eleven crates against a tree of 117 — ring among them, which would be the second C dependency in a binary whose README states it has one. Both of those are published product properties (section 11), so the default build posts over HTTP and refuses an https:// target at load time, naming the feature. Build with --features webhook-tls and it dials TLS directly.

Structs§

Engine
Rule
Rules
What the whole rules file parsed to.
State
What one rule is currently doing. Reported verbatim by /api/v1/alerts.
Target_
A webhook endpoint.

Enums§

Cmp 🔒
Format 🔒
Metric 🔒

Constants§

WEBHOOK_TIMEOUT 🔒

Functions§

alert_json 🔒
client 🔒
One pooled client for the process. Alerts are rare and bursty, and a fresh TCP (and TLS) handshake per page is the whole latency of a page.
count 🔒
count_pair 🔒
Run a rule’s queries and reduce them to one number.
filter_of 🔒
The rule’s where terms, spelled in the filter-bar grammar both UIs read.
handler 🔒
Every rule this node evaluates and what it is currently doing.
human 🔒
link 🔒
The UI URL that shows the rows this rule counted.
op_symbol 🔒
payload 🔒
post 🔒
POST a JSON body, with a deadline.
router
rule 🔒
signal_name 🔒
spawn
Start the evaluator, if this deployment has rules.
summary 🔒
One line of prose per alert, shared by every format that takes prose.
target 🔒
urlencode 🔒
Percent-encode everything outside the unreserved set.
when 🔒
count > 100, ratio >= 5%, count < 1.
windowless 🔒
A rule’s embedded query document, with the four keys the engine owns refused rather than ignored.

Type Aliases§

Client 🔒
Connector 🔒