Alert thresholds for error rate spikes documented and tested
Why it matters
Error tracking without alert thresholds is passive observation. A team that checks the Sentry dashboard manually once per day will miss an error spike that began at 2 AM and drove 400 failed signups before business hours. NIST SP 800-53 IR-5 (incident monitoring) requires automated monitoring and notification. ISO 25010 reliability.availability is directly reduced by every minute between spike onset and team awareness; unconfigured alerts convert that to hours. Alert thresholds are also the only objective way to distinguish a transient blip from a systemic regression.
Severity rationale
Info severity because alert gaps increase mean time to detection rather than causing failures directly — but undetected production incidents compound quietly, often reaching high user impact before anyone notices.
Remediation
Configure at minimum three alert rules in your error tracking or monitoring service: error rate, p95 response time, and uptime. Test each rule fires correctly after setup.
# docs/runbook/alert-thresholds.md
## Active Alerts
| Metric | Threshold | Channel | Response SLA |
|-----------------|-----------------|---------------|--------------|
| Error rate | >1% of requests | Slack #ops | 15 min |
| p95 latency | >2000 ms | Slack #ops | 30 min |
| Uptime | <99.9% (5 min) | PagerDuty | 5 min |
## Testing
Trigger test: send `POST /api/chaos/error-spike` in staging.
Last verified: 2026-04-01
Document the last test date and next scheduled test so alert configuration does not silently drift into a non-firing state.
DetectionPro
The detection method for this pattern is part of AuditBuffet Pro.
The full method (what to look for, pass and fail conditions, and how results get reported) ships inside the Error Resilience Audit. The pattern definition, severity rationale, and remediation remain openly licensed under CC-BY-4.0.
External references
- iso-25010:2011 · reliability.availability
- nist:rev5 · IR-5 — Incident Monitoring
Taxons
History
- 2026-04-18·v1.0.0·Initial import from error-resilience·automated