Direct messages not used as training data without separate opt-in
Why it matters
Direct messages carry the highest reasonable expectation of privacy in any community platform. Routing them through an ML training pipeline without separate, explicit opt-in violates GDPR Art. 7 (consent must be specific to the purpose) and NIST AI RMF GOVERN 6.2 (data governance for AI systems). OWASP LLM06:2025 classifies unintended training data inclusion as a top LLM risk because message content can later surface in model outputs, exposing private conversations to other users. The sender's consent alone is insufficient — both sender and recipient must opt in before message content is processed beyond delivery.
Severity rationale
Critical because using private messages as training data without consent breaches GDPR Art. 7 specificity requirements and can cause private conversation content to leak through model outputs.
Remediation
Add a distinct consent record for message_ai_processing — separate from general analytics consent — and gate any ML pipeline access behind a runtime check for both conversation participants. In src/lib/messaging/pipeline.ts or equivalent:
async function shouldProcessMessageForML(senderId: string, recipientId: string): Promise<boolean> {
const [senderConsent, recipientConsent] = await Promise.all([
db.userConsent.findFirst({
where: { userId: senderId, processingType: 'message_ai_processing' },
orderBy: { consentedAt: 'desc' }
}),
db.userConsent.findFirst({
where: { userId: recipientId, processingType: 'message_ai_processing' },
orderBy: { consentedAt: 'desc' }
})
]);
return (senderConsent?.consentGiven ?? false) && (recipientConsent?.consentGiven ?? false);
}
If the platform never uses messages for ML, make that commitment explicit in the privacy policy and skip this check by documenting the exemption in code.
DetectionPro
The detection method for this pattern is part of AuditBuffet Pro.
The full method (what to look for, pass and fail conditions, and how results get reported) ships inside the Community Privacy & Controls Audit. The pattern definition, severity rationale, and remediation remain openly licensed under CC-BY-4.0.
External references
- gdpr · Art. 7 — Conditions for consent
- gdpr · Art. 13 — Information to be provided where personal data are collected from the data subject
- ccpa · §1798.120 — Right to opt-out of sale or sharing of personal information
- owasp-llm:2025 · LLM06 — Excessive Agency — training on user messages without consent
- nist-ai-rmf:1.0 · GOVERN 6.2 — Policies and procedures for AI data use and consent
Taxons
History
- 2026-04-18·v1.0.0·Initial import from community-privacy-controls·automated