Queue workers that have never been tested at above-normal volume reveal their breaking points in production: the rate limiter was set to a library default, concurrency was never tuned, and the ESP starts returning 429s at 3x normal volume. ISO 25010 reliability.fault-tolerance requires that the system behaves predictably under adverse load conditions. Without a burst test script that runs against a sandbox ESP at 10x normal volume, the concurrency and rate limiter values are guesses — and the first real burst is the test.
High because untested burst capacity means the first high-volume send reveals queue configuration failures in production, where bounces and ESP throttling are real and affect sender reputation.
Add a burst test script to the repository (e.g., scripts/burst-test.ts) that enqueues 10x normal daily volume against the ESP sandbox:
const NORMAL_DAILY = 10_000
const BURST_MULTIPLIER = 10
for (let i = 0; i < NORMAL_DAILY * BURST_MULTIPLIER; i++) {
await queue.add('send-email', {
to: `burst-test-${i}@example.com`,
template: 'burst-test'
})
}
console.log(`Enqueued ${NORMAL_DAILY * BURST_MULTIPLIER} jobs. Monitor queue depth and ESP rate.`)
Set explicit concurrency and rate limiter values — not library defaults — before running. After the test, document peak memory usage, time to drain, and any rate limit errors observed.
ID: operational-resilience-email.capacity-scaling.burst-capacity-tested
Severity: high
What to look for: List all load test or burst test scripts in the repository (e.g., in tests/load/, scripts/, or a k6/ directory) that target the email sending pipeline. Count the number of burst test scripts found. Check that the script is configured to enqueue or send at a volume significantly above normal throughput. Also check that queue concurrency limits and ESP rate limiter settings are explicitly configured to handle burst scenarios — not left at defaults.
Pass criteria: A load test or burst test script exists in the repository that targets the email sending pipeline at a volume of at least 10x normal throughput. Queue concurrency limits and ESP rate limits are explicitly set (not defaulted) to handle burst scenarios. The script must be runnable without production credentials (uses sandbox/test mode).
Fail criteria: No load test script exists for the email pipeline. Queue concurrency and ESP rate limiter are set to static default values with no evidence of capacity planning for above-normal volume.
Skip (N/A) when: The system sends fewer than 1,000 emails per day and peak volume is well-understood and stable — documented in code or configuration comments.
Detail on fail: "No burst load test script found in scripts/, tests/load/, or k6/ — queue behavior under high volume is unverified" or "Queue concurrency and rate limiter use library defaults with no explicit burst-scenario configuration"
Remediation: Add a burst test script to the repository:
// scripts/burst-test.ts
const NORMAL_DAILY = 10_000
const BURST_MULTIPLIER = 10
for (let i = 0; i < NORMAL_DAILY * BURST_MULTIPLIER; i++) {
await queue.add('send-email', {
to: `burst-test-${i}@example.com`,
template: 'burst-test'
})
}
console.log(`Enqueued ${NORMAL_DAILY * BURST_MULTIPLIER} jobs. Monitor queue depth and ESP rate.`)
Set explicit concurrency and rate limiter values in the worker configuration to reflect burst capacity. After implementing, run the script against a staging environment connected to ESP sandbox/test mode and document results (time to drain, peak memory, any rate limit errors).