Output is filtered for harmful content before display
Why it matters
Input moderation stops known adversarial prompts at the gate, but a sufficiently crafted injection that slips through may still cause the model to produce policy-violating output. OWASP LLM02:2025 specifically calls out output filtering as a required second layer precisely because input-side controls are imperfect. Consumer-facing AI applications in regulated verticals (fintech, healthcare, legal) face liability when harmful content reaches users—regardless of whether the output was triggered by adversarial input. NIST AI RMF MANAGE 1.3 requires active harm management across the full inference pipeline, including the output. The finish_reason field is an often-ignored signal: a value other than stop indicates the model's output was interrupted by provider-side safety filters, and that truncated response should never be returned as a complete answer.
Severity rationale
High because output moderation is the last code-controlled layer before harmful or adversarially-triggered content reaches users, making its absence a high-impact gap in the defense stack.
Remediation
Apply output moderation after every completion call, and check finish_reason to handle safety-filtered responses correctly.
const assistantMessage = completion.choices[0]?.message?.content ?? ''
const finishReason = completion.choices[0]?.finish_reason
// Handle provider-side safety filter
if (finishReason === 'content_filter') {
return Response.json({ error: 'Response was filtered' }, { status: 400 })
}
// Output moderation
const mod = await openai.moderations.create({ input: assistantMessage })
if (mod.results[0]?.flagged) {
return Response.json({ error: 'Response not allowed' }, { status: 400 })
}
return Response.json({ message: assistantMessage })
If you use a non-OpenAI provider, Azure Content Safety and Perspective API both provide output moderation endpoints. Log flagged output events (without the full content) for pattern analysis.
DetectionPro
The detection method for this pattern is part of AuditBuffet Pro.
The full method (what to look for, pass and fail conditions, and how results get reported) ships inside the AI Prompt Injection Audit. The pattern definition, severity rationale, and remediation remain openly licensed under CC-BY-4.0.
External references
- owasp-llm:2025 · LLM02 — Insecure Output Handling
- nist-ai-rmf:1.0 · MANAGE 1.3 — Responses to identified AI risks are prioritized and managed
Taxons
History
- 2026-04-18·v1.0.0·Initial import from ai-prompt-injection·automated