Alerting configured for anomalous patterns (failed logins, privilege escalation, unexpected connections)

Infrastructure and application metrics tell you CPU is high — security alerting tells you someone is trying to escalate privileges. Without alerts on failed authentication bursts, RBAC denials, or unexpected egress connections, active attacks proceed silently through your cluster until data has already left (NIST 800-53 SI-4, AU-6). CIS Kubernetes 3.2.1 requires audit logging and alert routing. A Prometheus instance with only CPU/memory/disk alerts is operationally useful but security-blind — the two alert categories are not interchangeable.

Add security-focused PrometheusRule resources alongside your infrastructure alerts. In k8s/monitoring/alerts.yaml:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: security-alerts
  namespace: monitoring
spec:
  groups:
    - name: security
      rules:
        - alert: BruteForceAuthAttempts
          expr: rate(http_requests_total{status="401"}[5m]) > 0.5
          for: 2m
          labels:
            severity: warning
          annotations:
            summary: "High 401 rate — possible brute force on {{ $labels.job }}"
        - alert: RBACDenialSpike
          expr: increase(apiserver_audit_event_total{verb="create",objectRef_resource="clusterrolebindings"}[10m]) > 0
          for: 1m
          labels:
            severity: critical
          annotations:
            summary: "ClusterRoleBinding creation detected"

Route critical alerts to PagerDuty or a dedicated security Slack channel, separate from on-call infrastructure alerts.

• ID: infrastructure-hardening.monitoring-incident-response.alerting-configured

• Severity: info

• What to look for: Enumerate all alert rules across Prometheus, Grafana, PagerDuty, CloudWatch Alarms, or equivalent monitoring systems. For each alert rule, classify it as: (a) security-relevant (failed logins, RBAC denials, privilege escalation, unexpected egress), or (b) infrastructure-only (CPU, memory, disk). Count the total security alert rules — at least 2 security-specific rules should exist.

• Pass criteria: Alerting is configured with at least 2 security-relevant alert rules covering different categories: failed authentication attempts, RBAC denials, privilege escalation, or unexpected outbound connections. Alerts are routed to a notification channel (email, Slack, PagerDuty). Report: "X security alert rules configured, routed to Y notification channels."

• Fail criteria: No security-specific alerting configured, or alerts only cover basic infrastructure metrics (CPU, memory, disk) without any security event detection.

• Skip (N/A) when: Monitoring is managed externally or application is stateless with minimal security events.

• Detail on fail: Quote the alert config. Example: "Prometheus is deployed but no alert rules for security events. No notifications for failed login attempts or RBAC denials." or "CloudWatch has 5 alarms but all are for CPU/memory — no security event alerting"

• Remediation: Configure security alerting. Example with Prometheus:

  groups:
    - name: security
      rules:
        - alert: HighFailedAuthRate
          expr: rate(auth_failures_total[5m]) > 0.1
          for: 5m
          annotations:
            summary: "High failed auth rate detected"
  
        - alert: PrivilegeEscalation
          expr: rbac_denied_total{verb="create",resource="clusterrolebindings"} > 0
          for: 1m
          annotations:
            summary: "Privilege escalation attempt"