puter/doc/alarms.md

Alarms

clients.alarm is the one way code reports that something is wrong. It de-dupes by alarm id, counts occurrences, and routes each alarm to alert transports by severity.

this.clients.alarm.create(
    `metering_write_failed:${userUuid}`, // de-dupe key
    'Metering write failed', // what a human reads
    { userUuid, appId, error }, // context fields
    'info', // severity
);

Severity is the routing decision

Severity	Meaning	Goes to
critical	An unhandled server error. Someone gets woken up.	Pager
error	Same urgency as critical; prefer one of the other two.	Pager
warning	Worth a look today. Nobody is paged.	Pager (low)
info	A record of something expected-but-notable.	Chat

Each transport declares the severity window it accepts, so the value a call site passes is what decides where the alarm lands. The two windows don't overlap by default: anything that pages lives in the paging system, and chat is the record of what didn't. The bar for critical is deliberately high: an unhandled 5xx out of the HTTP error handler is the main thing that still pages. Anything a human can look at tomorrow is warning, and anything that's just worth recording is info.

Omitting the severity takes pager.defaultSeverity (itself critical), so pass one explicitly unless you really mean "page someone".

Choosing one

• Did the server fail to do its job in a way nobody expected? → critical
• Is a background job, rate, or dependency degraded, or has an account spent far past its credits? → warning
• Is this a user doing something notable (tripping an abuse heuristic)? → info
• Did one of our own limits reject a caller — a rate limit, a concurrency cap, a quota? → don't alarm at all. The limit doing its job is not an event; the 429 is the whole signal, and alarming on it only produces noise proportional to traffic. An upstream provider rate-limiting us is the opposite case and still alarms (upstream_rate_limited, info) — that one is not something we chose. The exception is a free model: nothing is billed, nothing is actionable, and the throttling is the price of the model, so the AI chat driver marks those noAlarm and only paid models still record.

noAlarm is the code-level mute: an HttpError carrying it skips the terminal gate entirely, no matter its status or code. It is for failures the call site already knows are expected and traffic-proportional — reach for it there, not as a way to quiet an alarm you haven't diagnosed (severityOverrides below is the knob for that).

An extension whose signals are all one tier can default its own local raiseAlarm helper to that tier instead of repeating it at every call site — see extensions/cronMonitor.

One incident per occurrence, unless you say otherwise

Alarms always de-dupe in process — repeats of an id bump its occurrence count rather than creating a second alarm. What that means for the pager is a separate decision, and by default every occurrence opens its own PagerDuty incident: two failed scans an hour apart are two things that happened, and closing one shouldn't hide the other.

Pass { dedup: true } as a fifth argument when repeats of the id really are one recurring fault, and they collapse onto a single incident carrying the occurrence count:

this.clients.alarm.create(alarmId, message, fields, 'critical', {
    dedup: true,
});

The HTTP error handler uses it: its id is route + error signature, so a hot loop of the same crash is one incident with N occurrences instead of N pages. Dedup and Slack's repeat throttle mean a responder sees only the latest occurrence, so the handler attaches the HttpError's fields to the alarm as details — put what a responder needs (which upstreams failed, and how) there. Everything in fields is also returned to the client in the response body, so it has to be safe to show the caller. Reach for it anywhere else only when the id is that specific — otherwise a per-request alarm can flood the pager.

Configuration

Everything lives under pager in config (see config.template.jsonc for the annotated version). Both transports are off unless enabled, so a self-hosted node just logs alarms to the console.

"pager": {
    "defaultSeverity": "critical",
    "severityOverrides": { "cronMonitor:*": "info" },
    "pagerduty": { "enabled": true, "routingKey": "…", "minSeverity": "warning" },
    "slack": {
        "enabled": true,
        "webhookUrl": "…",
        "channel": "#alerts",
        "minSeverity": "info",
        "maxSeverity": "info",
        "repeatThrottleMs": 900000,
    },
}

Slack's maxSeverity defaults to info whenever PagerDuty is configured, and to critical when it isn't — a node with only a webhook still sees everything. Raise it to have chat mirror the paging tiers as well.

Retiering without a deploy

severityOverrides is the escape hatch for an alarm that turns out to be noisier or more serious than its call site assumed. Keys are alarm ids or a prefix ending in *; the exact id beats a pattern, and the longest matching prefix wins among patterns. Values are a severity, or mute to drop the alarm before any transport sees it.

It is applied after the call site's severity and any known-error rule, so config always has the last word.

Repeat throttling

Two levels, and they stack.

The client itself backs off before a repeat is reported at all: every occurrence is reported up to a small burst, then one per interval. A fault recurring hundreds of times a second still counts every occurrence, but it does not write a log line or build an alert payload for each one.

The count lasts only as long as the client remembers the alarm. One that hasn't recurred for 24 hours is forgotten, and the client holds at most 1000 at once, dropping the least recently raised first. Either way the next occurrence is a new alarm, with its count and burst starting over.

Below that, the chat transport won't repost the same alarm id within repeatThrottleMs (default 15 minutes). The first occurrence always posts, and the next one that gets through reports how many piled up in between — so a hot loop reads as one message with a count, not a wall of them.