Images and media are scanned for policy violations

Platforms that accept image uploads without scanning expose their users to CSAM, non-consensual intimate imagery, graphic violence, and extremist content — all categories with serious legal consequences under CWE-434 (Unrestricted Upload of File with Dangerous Type) and OWASP A04:2021. Beyond legal risk, explicit imagery appearing in community feeds causes immediate user harm and platform reputation damage that is difficult to recover from. Manual review alone cannot scale: a single bad actor can upload hundreds of images in minutes, overwhelming any human queue.

Add an automated image scanning step to your upload handler in src/app/api/uploads/route.ts before the image is written to storage or made publicly visible:

import { ImageAnnotatorClient } from '@google-cloud/vision';
const vision = new ImageAnnotatorClient();

export async function scanUpload(buffer) {
  const [result] = await vision.safeSearchDetection({ image: { content: buffer } });
  const { adult, violence } = result.safeSearchAnnotation;
  if (['LIKELY', 'VERY_LIKELY'].includes(adult) || ['LIKELY', 'VERY_LIKELY'].includes(violence)) {
    throw new Error('Image flagged for policy violation');
  }
}

Alternatives: AWS Rekognition DetectModerationLabels or the Clarifai Moderation model. Flag images with a pending_review status and hide from public view until the scan resolves.