I built the delivery dashboard for a chat feature before it had real traffic, then rendered it in a browser and read every tile as if I were on call. Three of them were wrong. None of the queries had a bug. This is what each one was measuring instead of what its title said, and the rule that came out of each.
the availability tile counted the caller’s mistakes
The tile said “send availability, target 99.95%” and read 75.00% in red. The objective it was drawn from defined availability as excluding client-side 4xx. The query did not.
Over thirty days, every chat error ever recorded was a client rejection: sixty forbidden, nineteen bad request, zero server-side failures. The tile that gated the next rollout step was red because callers had sent unusable attachment URLs.
The rule: any widget carrying a target counts only failures the service owns. Raw counts keep counting everything and say so in their title. Client rejections get their own tile with no target, because a burst of bad requests is real information (it is how a validation-ordering bug was found) that just is not an availability problem.
Two details in the implementation matter. The split keys on the error type rather than an HTTP status, because a resolver span in a federated graph carries no status: the resolver throws and the router maps it further out. The list of exception types that count as client errors is an explicit enumeration, so anything absent from the list counts against availability. An allow-list of known server errors was rejected because it fails in the dangerous direction: a new failure mode would be silently excluded rather than counted.
The same rule had already earned its place on the higher-traffic services I own. Business-level 4xx filtered out of error rates is the difference between an alert people trust and one they learn to ignore. Error budgets are for errors. A rejection the service was right to issue is data.
the capacity ceiling was a static line
A memory widget drew a hardcoded “3.22 GB limit” marker. The widget’s query varied by environment through a template variable. The actual memory limit was 3.33 GB in production and 1.07 GB in staging.
Staging was the only environment carrying chat traffic at the time, so the ceiling was drawn 3.2x too high exactly where anyone would read it. A staging pod at 900 MB was at 88% of its limit, and the dashboard rendered that as comfortably safe. The CPU widget had the mirror problem: raw nanocores with no ceiling at all, while the CPU limit also differed per environment.
The rule: any ceiling, limit, or capacity line on a dashboard whose queries vary by environment is plotted from the metric that defines it, as a series, never as a static marker. A static marker is acceptable only for a value that cannot vary, such as a timeout constant in code. Where usage and limit carry different units, the formula converts so both share an axis.
The alternatives were worse. Duplicating each widget per environment doubles the widget count and breaks again on the next limit change. Templating the marker off the environment variable is not supported, because markers take literal values. A percentage-only widget hides absolute headroom, which is the number a capacity argument needs.
the latency percentile included the failures
“Send p95, target 400 ms” read 1.50 seconds. Staging had served 33 sends in seven days: 24 successful at a p95 of 329 ms, inside target, and 9 failures at a p95 of 1,503 ms.
Four of those failures spent a full 1.5-second link-preview budget in an outbound fetch and then returned a bad-request error for an unusable attachment URL, because the use case hydrated link previews before the service validated attachments. A validation bug, on a sample small enough that a p95 is the second-slowest call, was reported as a latency breach.
The rule: every chart carrying a target filters out error-status samples. Failures keep their own tiles. A rejected request still pays for whatever ran before the rejection, so blending them makes a validation bug indistinguishable from a slow send. Alongside it, a minimum-sample note on the health group, since below roughly two hundred samples a p95 has to be read against the attempt count.
the monitor that catches a feature silently not existing
The dashboard was built before launch, and reading the baseline showed something uncomfortable: a launch that silently failed to write anything would look exactly like the current healthy-looking state. Zero errors, zero latency breaches, green everywhere.
Two monitors exist because of that. One fires when zero real messages are seen in 24 hours while the feature flag is on for at least one tenant. The other fires when the idempotency-key coverage on new rows drops below 99%, because everything downstream (loss detection, duplicate detection, reconciliation) depends on that key being present, and at the time the column existed with zero rows carrying it.
A monitor that fires on the absence of expected traffic is the only thing that catches a write path that is quietly not running.
anti-noise rules
These came from two chronic alerts on the same team that nobody acted on any more. One of them, “some invites were not sent”, fired on 10 real cases against 238 benign ones in a week.
- No monitor fires on a swallowed error. If the code catches and continues, the monitor is measuring a log line instead of an outcome.
- Every monitor names the outcome it measures rather than matching a log string.
- Benign cases get their own outcome value so they never land in an error rate.
The common thread across all of this: a tile or a monitor is a claim. “Availability is 75%” is a claim, and when the reader cannot tell what the denominator is, the claim is worth less than no tile at all, because a permanently red number is one everyone learns to ignore.