Centralized log aggregation configured with timestamps, severity, and trace IDs

Logs written only to stdout and discarded when a pod restarts leave no forensic trail for incidents — there is no record of which requests preceded a data breach, no timestamp evidence for compliance audits, and no visibility into error patterns that precede outages (NIST 800-53 AU-3, AU-9). CIS Kubernetes 3.2.2 requires log aggregation and forwarding. Missing trace IDs means individual requests cannot be correlated across service boundaries, turning a multi-service investigation into manual log grep archaeology. 30-day retention is the minimum threshold for detecting slow-burn attacks that unfold over weeks.

Deploy a log aggregation sidecar or DaemonSet. For Fluent Bit to CloudWatch, add to k8s/logging/fluent-bit.yaml:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
spec:
  template:
    spec:
      containers:
        - name: fluent-bit
          image: fluent/fluent-bit:2.2
          volumeMounts:
            - name: varlog
              mountPath: /var/log
      volumes:
        - name: varlog
          hostPath:
            path: /var/log

Structure application logs as JSON with three required fields: timestamp (ISO 8601), level (severity), and traceId (propagated from the incoming X-Request-ID or traceparent header). Set log retention to 30 days minimum in your aggregation target — CloudWatch log groups default to Never expire which costs more; set explicitly to 30 days.

• ID: centralized-logging

• Severity: info

• What to look for: List all logging configuration files and sidecar containers (Fluent Bit, Filebeat, Fluentd). Check for log aggregation targets (ELK Stack, Splunk, Datadog, CloudWatch, Stackdriver). Count the required structured fields present in log output: timestamp, severity level, and request trace ID. All 3 fields must be present.

• Pass criteria: Centralized logging is configured with log aggregation to at least 1 external system (ELK, Splunk, Datadog, CloudWatch, or equivalent). Every log entry includes all 3 required structured fields: timestamp, severity level, and request trace ID. Logs are retained for at least 30 days. Report even on pass: "Logging aggregated to X, retention Y days, 3 of 3 structured fields present."

• Fail criteria: No centralized logging configured (logs only written to stdout), or logs are missing any of the 3 required structured fields (timestamp, severity, trace ID).

• Skip (N/A) when: Logging is managed externally or application is stateless with no relevant logs.

• Detail on fail: Quote the logging config. Example: "Application logs are written to stdout only. No Fluent Bit or Filebeat sidecar configured." or "Logs include timestamp and severity but lack request trace IDs — 2 of 3 structured fields"

• Remediation: Set up centralized logging. Example with Fluent Bit → CloudWatch:

  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: fluent-bit-config
  data:
    fluent-bit.conf: |
      [SERVICE]
          Flush        5
          Daemon       Off
          Log_Level    info
  
      [INPUT]
          Name              tail
          Path              /var/log/containers/*.log
          Parser            json
          Tag               kube.*
  
      [OUTPUT]
          Name   cloudwatch_logs
          Match  kube.*
          region us-east-1
          log_group_name /ecs/my-app
          log_stream_prefix from-fluent-bit-
          auto_create_group true