Automated toxicity scoring runs on user-submitted text

Keyword-based profanity filters catch known bad words but miss context-dependent toxicity: veiled threats, dog-whistles, coded harassment, and insults that don't trigger keyword lists. Automated toxicity scoring via ML models (Perspective API, OpenAI Moderation, AWS Comprehend) catches the patterns that keyword lists miss. Without it, communities routinely develop persistent harassment cultures because bad actors quickly learn which phrases bypass keyword filters. This is CWE-20 input validation applied at the semantic layer. Platforms with substantial UGC and no toxicity scoring have higher reported abuse rates and higher moderator burnout.

Integrate a toxicity scoring service at every content-submission endpoint before the database write. Using Perspective API:

async function scoreToxicity(text) {
  const res = await fetch(
    `https://commentanalyzer.googleapis.com/v1alpha1/comments:analyze?key=${process.env.PERSPECTIVE_API_KEY}`,
    {
      method: 'POST',
      body: JSON.stringify({
        comment: { text },
        requestedAttributes: { TOXICITY: {}, SEVERE_TOXICITY: {}, INSULT: {} },
      }),
    }
  );
  const data = await res.json();
  return data.attributeScores.TOXICITY.summaryScore.value;
}

Content scoring above your flag threshold (see safe-defaults check) goes to the moderation queue; content above auto-remove is hidden immediately. Run scoring asynchronously post-save only if latency is a hard constraint — but hide the content from public view until scoring completes.