Alert rules
Alerts are plain JSON in ALERTS_FILE (default /data/alerts.json). The file is hot-reloaded when its mtime changes — save it and the new rules take effect without restarting. Evaluation runs every ALERTS_INTERVAL, default 1m.
A complete rules file
{
"rules": [
{ "name": "api-silent", "kind": "deadman", "apps": ["api"], "for": "5m" },
{ "name": "api-looping", "kind": "restart_loop", "apps": ["api"], "count": 3, "window": "10m" },
{ "name": "api-mem", "kind": "metric", "apps": ["api"], "metric": "mem_pct",
"op": "gt", "value": 85, "for": "5m" },
{ "name": "api-5xx", "kind": "log_pattern", "apps": ["api"], "pattern": "status=5\\d\\d",
"hits": 5, "window": "5m", "repeat": "30m" }
]
}/data so it lives with the database: - ./alerts.json:/data/alerts.json:ro. A malformed file keeps the previous rules running and logs the error — alerting never silently stops.Rule kinds
Fires when an app has produced no log lines for longer than for. This is your watchdog: cron jobs, health reporters and webhooks all go quiet when they break.
| Field | Required | Meaning |
|---|---|---|
for | yes | Silence duration, e.g. 5m. |
Fires when an app accumulates count deaths inside window. Deaths are counted from stored snapshots, so rollout noise stays out of the count by default.
| Field | Required | Meaning |
|---|---|---|
count | yes | Deaths required to fire. |
window | yes | Sliding window, e.g. 10m. |
Fires when every sample in the for window satisfies the comparison and the newest sample is fresh. One cool sample inside the window stops the alert, and stale data (newest sample older than twice the window) is ignored so a stopped app cannot page forever.
| Field | Values | Meaning |
|---|---|---|
metric | cpu, mem, mem_pct | Which series to compare. cpu is percent, mem bytes, mem_pct percent of limit. |
op | gt, lt, gte, lte | Comparison operator. |
value | number | Threshold. |
for | duration | How long the condition must hold. |
Fires when at least hits lines matching the regular expression arrive inside window. The pattern is matched against stored rows for the app, so redaction has already run — secrets are not part of the match.
| Field | Required | Meaning |
|---|---|---|
pattern | yes | Regular expression (escape backslashes, e.g. status=5\\d\\d in JSON). |
hits | yes | Matching lines required inside the window. |
window | yes | Counting window, e.g. 5m. |
Fields shared by every rule
| Field | Default | Meaning |
|---|---|---|
name | — | Unique identifier, shown in the firing history. |
kind | — | One of deadman, restart_loop, metric, log_pattern. |
apps | all known apps | App names the rule applies to. Omit to watch everything. |
repeat | 30m | Silence between notifications for the same rule and app. The alerts table is the silence window, so restarts cannot reset it. |
Channels
Configure one or more with environment variables — see configuration → alerting. All configured channels are notified for every firing; each has its own 10-second timeout and one failure never blocks the rest.
Webhook
ALERT_WEBHOOK_URL — generic JSON POST.
Telegram
ALERT_TELEGRAM_TOKEN + CHAT_ID.
Slack
ALERT_SLACK_URL — incoming webhook.
Discord
ALERT_DISCORD_URL — webhook URL.
POST /api/alerts/test, which pushes a test message through every configured channel and returns {results: {webhook: "sent", …}}.Inspecting armed rules
# armed configuration, secrets stripped
curl -s -u admin:$AUTH_PASS http://localhost:8080/api/alerts/rules
# firing history, newest first
curl -s -u admin:$AUTH_PASS 'http://localhost:8080/api/alerts?app=api&limit=50'
# push a test message on every channel
curl -s -X POST -u admin:$AUTH_PASS http://localhost:8080/api/alerts/testFull response shapes on the API reference → alerts.