Expand description
Static-rule alerting: a KYAML document in, webhooks out.
§A rule is a query, not a new language
Every alerting system this replaces ships a second language — PromQL, a
DSL, a builder UI — and that language is the reason “why did this page?” is
hard to answer: the thing that fired is not the thing you can look at. Here
a rule embeds a /api/v1/query document, verbatim. The operator builds
the query in the UI, presses nothing, pastes it into the rules file, and the
link in the resulting page opens the same query in the same UI. There is one
filter grammar in this product and this is not a second one.
§Two metrics, and why that is enough
count is how many records matched. ratio is that over a second query’s
count. Between them they express both of the rules everyone actually writes:
- error rate above 5% over a minute — numerator
status_code = 2, denominator the same filter without it,ratio > 5%. - p95 latency above 500 ms — numerator adds
duration_nano > 500000000, same denominator,ratio > 5%.
The second one is not an approximation of a quantile, it is the quantile:
p95(d) > T and |{d > T}| / |d| > 0.05 are the same statement. That
identity is why there is no digest in this file and no p95(...) in the
grammar — a percentile threshold was always a counting question wearing a
statistics hat, and Mira can already count exactly.
§Counting is free
query::search_open with limit: 0 returns no rows and an exact
rows_matched: the early exit needs a held hit to fire and there are none,
so every matching block is scanned, and the per-wave trim throws the hits
away before they accumulate. So an evaluation costs one scan and allocates
nothing per match — the same code path, the same zone maps and the same
bloom filters the UI’s query uses, which is also why a rule cannot drift
from what the operator sees.
§One replica evaluates
Principle 4 is no coordination state, and N replicas reading the same block
directory would each fire the same rule N times. There is no lease and no
leader here: alerting is off unless alerts.rules names a file, so the
deployment enables it on one replica and that is the whole mechanism. State
— which rules are firing, and since when — is in memory, so a restart
re-evaluates from scratch and re-pages anything still breaching.
ponytail: dedup is the operator pointing one replica at the rules file. A lease file in the block directory is the upgrade path if that stops being enough, and it stays inside “the block directory is the manifest”.
§Why HTTPS is a Cargo feature
Slack, Discord and PagerDuty are all HTTPS, and in-process TLS costs eleven
crates against a tree of 117 — ring among them, which would be the second
C dependency in a binary whose README states it has one. Both of those are
published product properties (section 11), so the default build posts over HTTP and
refuses an https:// target at load time, naming the feature. Build with
--features webhook-tls and it dials TLS directly.
Structs§
- Engine
- Rule
- Rules
- What the whole rules file parsed to.
- State
- What one rule is currently doing. Reported verbatim by
/api/v1/alerts. - Target_
- A webhook endpoint.
Enums§
Constants§
Functions§
- alert_
json 🔒 - client 🔒
- One pooled client for the process. Alerts are rare and bursty, and a fresh TCP (and TLS) handshake per page is the whole latency of a page.
- count 🔒
- count_
pair 🔒 - Run a rule’s queries and reduce them to one number.
- filter_
of 🔒 - The rule’s
whereterms, spelled in the filter-bar grammar both UIs read. - handler 🔒
- Every rule this node evaluates and what it is currently doing.
- human 🔒
- link 🔒
- The UI URL that shows the rows this rule counted.
- op_
symbol 🔒 - payload 🔒
- post 🔒
- POST a JSON body, with a deadline.
- router
- rule 🔒
- signal_
name 🔒 - spawn
- Start the evaluator, if this deployment has rules.
- summary 🔒
- One line of prose per alert, shared by every format that takes prose.
- target 🔒
- urlencode 🔒
- Percent-encode everything outside the unreserved set.
- when 🔒
count > 100,ratio >= 5%,count < 1.- windowless 🔒
- A rule’s embedded query document, with the four keys the engine owns refused rather than ignored.