From eef066b63b5264d06325f21e29df83e7edecdade Mon Sep 17 00:00:00 2001 From: Anthony Ettinger Date: Tue, 28 Jul 2026 17:43:18 +0000 Subject: [PATCH] =?UTF-8?q?docs(prd):=20add=20PRD=200006=20=E2=80=94=20Rou?= =?UTF-8?q?te=20alerts=20so=20they=20keep=20being=20read?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every module already depends on this one and it does not exist: spend-guard routes through it, code-scanner calls ctx.alert() in all four subsystems. The calls compile and the daemon runs, but nothing decides where an alert goes or whether it is a duplicate. Alert fatigue is the primary failure mode of the whole product — a detection nobody reads is identical to one that never fired, so every module's work on precision is undone if routing sends 400 messages to one channel. The hard parts are deduplication by incident identity and escalation on change rather than repetition; delivery is trivial by comparison. Co-Authored-By: Claude Opus 4.8 --- ...06-route-alerts-so-they-keep-being-read.md | 143 ++++++++++++++++++ prd/README.md | 1 + 2 files changed, 144 insertions(+) create mode 100644 prd/0006-route-alerts-so-they-keep-being-read.md diff --git a/prd/0006-route-alerts-so-they-keep-being-read.md b/prd/0006-route-alerts-so-they-keep-being-read.md new file mode 100644 index 0000000..a8acd41 --- /dev/null +++ b/prd/0006-route-alerts-so-they-keep-being-read.md @@ -0,0 +1,143 @@ +--- +openprd: "0.2" +id: "0006" +title: "Route alerts so they keep being read" +status: Draft +authors: + - anthony@profullstack.com +created: 2026-07-28 +updated: 2026-07-28 +repo: profullstack/threatcrush +discussion: +implementation: +tags: alert-system, routing, deduplication, escalation, notifications, modules +supersedes: +superseded-by: +--- + +## Problem + +**Every module already depends on this one, and it does not exist.** `spend-guard` +routes through `alert-system` in its `mod.toml`; `code-scanner` calls +`ctx.alert()` in all four subsystems. The SDK defines the `Alert` shape, so the +calls compile and the daemon runs — but there is nothing behind them deciding +where an alert goes, whether it is a duplicate, or whether it is worth waking +someone. That gap is invisible until the day it matters. + +**Alert fatigue is not a UX problem, it is the primary failure mode of the +entire product.** A detection nobody reads is identical to a detection that +never fired, so every module's careful work on precision — `deps` collapsing +overlapping rules, `secrets` refusing to report AWS documentation keys, `sast` +capping pattern-only findings at medium — is wasted if the routing layer +undoes it by sending 400 messages to one Slack channel. + +**The hard parts are deduplication and escalation, not delivery.** Posting to a +webhook is trivial. Knowing that this alert is the same incident as the one 40 +minutes ago, that it has since become more severe, that the operator is asleep +and it can wait until morning, or that it has fired every hour for a week and +should stop — that is the product. + +## Goals + +- An operator receives **one message per incident**, not per detection event. +- **Severity determines channel and urgency**, so a critical reaches someone + awake and an informational does not. +- A recurring, unacknowledged condition **escalates rather than repeats**. +- Delivery failure is itself visible: an alert that could not be sent must not + vanish. +- Modules stay simple — they call `ctx.alert()` and this layer decides + everything else. + +## Non-Goals + +- **Not an incident-management platform.** On-call rotations, scheduling and + acknowledgement workflows belong to PagerDuty; integrate, do not reimplement. +- **Not a SIEM.** Long-term event storage and correlation across hosts is a + different product; this routes what modules produce. +- **Not per-module notification config.** Modules describe severity and + content; routing policy lives here, or every module reimplements it slightly + differently. + +## Users + +- **Solo operators** who need the phone to ring for a compromise and stay + silent for a hardening suggestion. +- **Small teams** routing by severity to different channels. +- **Anyone who has muted a monitoring tool** — the population this is designed + around. + +## Requirements + +- R1 [P0] **Destinations**: Slack, Discord, generic webhook, email (SMTP), + syslog, PagerDuty. Each independently configured and independently failable. +- R2 [P0] **Severity-based routing**: map each severity to zero or more + destinations, so `info` can go to syslog only and `critical` to PagerDuty. +- R3 [P0] **Deduplication by incident identity**, not by message text. An + incident key is (module, subject, detector); the same key within a window + updates rather than repeats. +- R4 [P0] **Escalation on change, not on repetition.** A condition that worsens + re-notifies; a condition that merely persists does not. +- R5 [P1] **Quiet hours** with a severity floor that ignores them. +- R6 [P1] **Rate limiting per destination**, with a summary message when the cap + is hit — never silent truncation. +- R7 [P0] **Delivery failures are alerts.** A webhook returning 500 must surface + somewhere, or the system is silently deaf. +- R8 [P1] **Digest mode**: batch below-threshold alerts into a periodic summary. +- R9 [P2] **Acknowledgement** via a link or CLI, suppressing an incident until + it changes. + +## UX Notes + +```toml +[alert-system] +enabled = true + +[alert-system.routing] +critical = ["pagerduty", "slack"] +high = ["slack"] +medium = ["slack"] +low = ["syslog"] +info = ["syslog"] + +[alert-system.dedupe] +window = "6h" +escalate_on_severity_change = true + +[alert-system.quiet_hours] +from = "22:00" +to = "07:00" +floor = "critical" # criticals ignore quiet hours +``` + +```bash +threatcrush alerts test --severity critical # prove the path end to end +threatcrush alerts recent # what fired, and where it went +threatcrush alerts ack +``` + +Design constraint: **the test command matters more than it looks.** An alerting +path is usually discovered to be broken at the moment it is needed, so proving +it works must be one command. + +## Success Metrics + +- A 48-hour ongoing condition produces **one** notification plus escalations, + not one per scan interval. +- Zero silent delivery failures: every failed send appears somewhere. +- `alerts test` exercises the real path, including credentials, in <5s. +- Operators keep the integration enabled after 30 days — the only metric that + really matters, and the one this module exists to protect. + +## Risks & Open Questions + +- **Dedupe that is too aggressive hides a second, real incident** behind the + first. Incident identity must include enough to distinguish them, and + over-merging is worse than a duplicate message. +- **Credentials for six destinations** make this module a high-value target on + the host. They should resolve from the secret store rather than sit in config. +- **Quiet hours are a loaded gun.** A misconfigured floor silences a compromise + until morning. **Open:** should quiet hours require an explicit floor, with no + default, so the choice is always deliberate? +- **Open:** should modules be able to suggest a destination, or is routing + policy exclusively the operator's? Suggesting risks every module claiming to + be critical. diff --git a/prd/README.md b/prd/README.md index c110add..88d5f76 100644 --- a/prd/README.md +++ b/prd/README.md @@ -18,3 +18,4 @@ these numbered PRDs cover individual changes to it. | [0003](./0003-detect-hardcoded-secrets-before-they-are-committed-or-served.md) | Detect hardcoded secrets before they are committed or served | Draft | code-scanner, secrets, credentials, entropy, redaction, modules | | [0004](./0004-find-dangerous-code-patterns-without-pretending-to-be-a-compiler.md) | Find dangerous code patterns without pretending to be a compiler | Draft | code-scanner, sast, static-analysis, injection, taint, modules | | [0005](./0005-catch-the-misconfigurations-that-actually-get-servers-breached.md) | Catch the misconfigurations that actually get servers breached | Draft | code-scanner, config, misconfiguration, hardening, exposure, modules | +| [0006](./0006-route-alerts-so-they-keep-being-read.md) | Route alerts so they keep being read | Draft | alert-system, routing, deduplication, escalation, notifications, modules |