Two of My Own Alert Rules Were Wrong. Here Is How I Found Out.

An alert rule you have never seen fire on a real failure is not a rule. It is a decoration. I wrote five for my Nomad cluster. Two of them were wrong — one screamed at nine perfectly idle machines, the other stayed silent through the exact failure it was written for. Both looked correct on the page. I only found out because I went and broke things on purpose.

What the audit measured

On 31 July I audited the lab — a 15-machine fleet across two physical sites, of which only five were actually joined to the Nomad + Consul cluster at that point. Monitoring "existed": Prometheus, node-exporter, Grafana, all running as Nomad jobs, all healthy. Here is what the APIs actually returned.

That last line deserves its own sentence. The TSDB lived in an anonymous Docker volume created by the image's own VOLUME /prometheus — not in the container write layer, not on a host volume. Every new allocation got a brand-new empty volume, and the job history showed 6 failed plus 3 complete allocations. Fifteen days of nominal retention, thirty hours of actual memory. Any alerting built on that is built on sand. The fix I have queued is a dynamic host volume — nomad volume create with the mkdir plugin, sticky = true — with the acceptance test being: stop the alloc, then query a series from before the stop. That one is not done yet; everything below is running on a TSDB that still forgets.

A grep -rniE "alert|alertmanager|snapshot|backup|watchdog|panic" over the whole repo returned nothing at all. Resilience in this lab was not degraded. It had never been written.

The failure nobody saw

The audit found the proof it needed already running. A self-hosted CI runner job had been in pending since 25 July — six days — with 174 failed allocations at audit time, 214 a day later. Default reschedule policy: Unlimited = true, exponential backoff capped at one hour. An infinite loop with a one-hour period.

The real allocation events, not my guess at them:

Failed to pull myoung34/github-runner:latest: context deadline exceeded
Exit Code: 1

An 810 MB image timing out on pull, and when it did start, the task died about one second later. Six days of that. The Nomad UI was green, Grafana had no dashboards to be red on, and the counter climbing in nomad_nomad_job_summary_failed was being scraped every 15 seconds into a database nobody queried.

One caveat worth keeping honest: 174 is the cumulative job-summary counter. Only 4 allocation objects still existed; the rest had been garbage-collected. That distinction turns out to matter a lot later.

Wrong rule #1: temperature

Nine of the machines are Mac minis from 2011–2014 at the remote site, plus an old dual-socket Xeon box that had just been repasted. Watching CPU temperature is not paranoia there; the first real load after a thermal-paste job is exactly when a bad mount announces itself.

So I wrote the obvious rule:

- alert: HighTemperature
  expr: node_hwmon_temp_celsius > 80
  for: 5m

It fired immediately. On nine idle minis. verdict: noise

Macs expose roughly 60 pseudo-sensors through the applesmc driver, and a lot of them are not temperatures. Reading the raw series gives you -127, -48, 0, and a cluster of values in the 93–103 range that correspond to nothing thermal at all. The driver publishes the whole SMC key space under the same metric name; the real CPU sensor is the coretemp chip, and only that one.

- alert: HighTemperature
  expr: node_hwmon_temp_celsius{chip=~".*coretemp.*"} > 80
  for: 5m

With the filter, the measured reality: minis idle between 39 and 52 °C, the Xeon server at 46 °C. Nothing anywhere near 80. The threshold was fine all along; the series selection was garbage.

There is a second half to this one. LXC containers have no coretemp chip at all — they inherit the host's hwmon tree and see its nvme and pci sensors. A naive threshold on node_hwmon_temp_celsius inside a container is not measuring that container's CPU at all — it is reading the host's SSD controller.

Wrong rule #2: the failing job

This was the rule I was most confident about, because I had just watched the failure it was for:

- alert: JobStuckQueued
  expr: nomad_nomad_job_summary_queued > 0
  for: 15m

It does not catch the CI runner. I queried the actual series on 1 August, against the failure that was still running:

nomad_nomad_job_summary_queued{exported_job="github-runner"}   0
nomad_nomad_job_summary_failed{exported_job="github-runner"}   214
nomad_nomad_job_summary_running{exported_job="github-runner"}  0

queued = 0. The rule encoded a mental model: a job waiting for capacity, or blocked by an impossible constraint. That is a real failure mode and the rule catches it fine. It is simply not the failure mode I had. My job was not waiting for anything. It started, died in about a second, got rescheduled, and started again. It never spent measurable time queued.

The rule that does catch it:

expr: nomad_nomad_job_summary_running == 0 and nomad_nomad_job_summary_failed > 0
for: 15m

Verified against the live failure: it went pending, then would have fired. That is the check that matters — not "does this expression look right", but "does this expression light up against the thing that actually broke".

And then it was wrong too, in a different way. failed is a cumulative counter. Once I stopped the runner job, the counter froze at 229 and stayed there forever, with running still at 0 — so the alert kept firing about a job that no longer existed. An alert that never resolves gets muted, which is the same disease as a false positive with extra steps. The version that shipped measures the change, not the total:

- alert: JobFailingInLoop
  expr: delta(nomad_nomad_job_summary_failed[30m]) > 2
  for: 10m

It reacts only to new failures and clears itself when the bleeding stops.

The third kind: rules on metrics that do not exist

The resilience report I was working from proposed nomad_raft_peers < 3 and consul_raft_peers < 3. Neither metric exists in this installation. Out of the 1,738 metric names Prometheus actually had, the quorum-adjacent ones are consul_autopilot_healthy, consul_autopilot_failure_tolerance, nomad_nomad_autopilot_healthy (present on the leader only, so its absent() is itself a useful signal) and consul_members_servers.

A rule over an empty series never fires, never errors, and shows up in /api/v1/rules looking exactly like a rule that works. It is the purest form of decoration. The defence costs ten seconds: paste the expression into /api/v1/query before shipping it. An empty result set means the rule is not monitoring, it is a comment.

Rules that fire and notify nobody are still decoration

Prometheus does not talk to an Alertmanager unless you tell it to. Without an explicit block, rules transition to firing and colour a web page that nobody has open:

alerting:
  alertmanagers:
    - static_configs: [{ targets: ['127.0.0.1:9093'] }]

Verify it with /api/v1/alertmanagers. If activeAlertmanagers is an empty array, everything downstream of it is imaginary.

Alertmanager itself runs as a one-task Nomad job pinned to the same node as Prometheus — they talk over loopback, and that node is one of the two that has a local Consul agent. The receiver is a Telegram bot; its token comes from a Nomad variable readable only by this job's workload identity, so the job file stays committable as-is. The anti-noise settings are the part that decides whether the channel survives contact with reality:

route:
  receiver: telegram
  group_by: [alertname]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 6h

inhibit_rules:
  - source_matchers: [alertname = "TargetUnreachable"]
    target_matchers: [alertname =~ "DiskAlmostFull|HighTemperature"]
    equal: [instance]

The inhibit rule is the one I would keep if I could keep only one. A machine that has gone away will also stop reporting disk space and will eventually trip its temperature staleness — three messages for one incident. With the inhibition, an unreachable node produces exactly one message about being unreachable.

The acceptance criterion was not "the rule shows firing in the UI". It was my phone buzzing, with the alert name and the instance label in the message body. Until that happened, the notification chain in this lab had literally never been exercised — not once, in the entire life of the project.

The other invisible: metrics that never arrive

Wrong thresholds are the fun class of bug. The boring class is a metric that structurally cannot reach you, and it is much harder to notice because nothing anywhere is red.

My node-exporter job is type = "system", so it should run one instance on every client. It ran on two. The job declared:

service {
  name     = "node-exporter"
  provider = "consul"
}

Nomad excludes from placement any node that has no consul.* attributes. The machines at the remote site expose driver.docker = 1 and no Consul fingerprint at all, because they deliberately run no Consul agent: the inter-site link measures 57 to 600 ms with heavy jitter, and Consul's LAN gossip is tuned for sub-10 ms. Result: 10 of 12 client nodes with no system metrics whatsoever, and no error message anywhere — just a healthy-looking system job that had quietly decided those nodes were not eligible.

The fix is a second job using Nomad's own service provider, with a constraint that maintains itself:

constraint {
  attribute = "${attr.consul.version}"
  operator  = "is_not_set"
}

service {
  name     = "node-exporter-remote"
  provider = "nomad"   # not consul
}

attr.consul.version is_not_set means precisely "nodes without a Consul agent". Any new off-site worker is covered without editing this file, and if I ever install Consul on one of them, it leaves this job and joins the other one by itself. Prometheus discovers them with nomad_sd_configs against the Nomad API plus a relabel keep on __meta_nomad_service, with scrape_interval: 30s and scrape_timeout: 20s because that link is slow and variable.

And then the follow-on trap

Placement fixed, allocations running, and Prometheus reported no route to host on all ten new targets.

The advertise block affects the agent only. Services and allocations register with the address of the interface Nomad fingerprints — by default the first routable interface, which here was the LAN NIC. So the exporters were publishing themselves as 192.168.1.93:9100, an address that from the other site means a different machine or nothing at all. Both sites were on 192.168.1.0/24 at the time; that single fact produced three distinct outages in this project before I renumbered one of them onto its own /24.

client {
  network_interface = "tailscale0"
}

Changing the config is not enough. An allocation registers its address once, at creation. Existing allocations kept publishing the old address after the agent restarted, which produces the classic "but I fixed it, why is it not applying". They have to be recreated: nomad job stop -purge then nomad job run. Restarting the task does not do it.

Where it landed

The 21 targets break down as: 1 Prometheus, 2 site-A exporters found through Consul service discovery, 10 site-B exporters found through Nomad service discovery, 3 Consul agents and 5 Nomad agents.

Five rules. Deliberately few — each one rests on a metric I checked is actually present in this installation:

The queued rule stayed. It was never wrong — it was incomplete, and I had been treating it as coverage for a failure class it does not cover. Those two things feel identical right up until the moment you need the alert.

The rule, then the hammer

Everything above reduces to one habit that I did not have three weeks ago: write the rule, then go break the thing, and watch the rule fire.

nomad alloc stop on an exporter, then wait for the unreachable alert. A disk filled with fallocate. A job that exits 1 on purpose, and the failure counter moving. A synthetic alert that lands on a phone, not just in a browser tab. None of this is clever. All of it gets skipped, constantly, including by me.

A rule validated only against your mental model of a failure is untested code sitting in the most safety-critical part of the system — the part whose entire job is to tell you the rest of it is broken. My temperature rule was tested against my belief that hwmon reports temperatures. My queued rule was tested against my belief about how jobs fail. Both beliefs were wrong, both rules were green, and a real failure ran for six days between them.

Two out of five. That is the number I would bet on for anyone's first alerting ruleset, mine included. The only way to find out which two is to break something on purpose.