Exposed Secrets in ConfigMap: Incident Response

Phase 1: CONTAIN (Minutes 0–15)
  → Rotate credentials immediately
  → Remove ConfigMap from cluster

Phase 2: ERADICATE (Minutes 15–60)
  → Purge from git history
  → Revoke any active sessions using old credentials

Phase 3: RECOVER (Hours 1–4)
  → Implement proper secret management
  → Verify no unauthorized access occurred

Phase 4: POST-MORTEM (Days 1–7)
  → Root cause analysis
  → Preventive controls implementation
  → Team education

# STEP 1: Assume the credentials are already compromised.
# Do NOT wait to confirm. Rotate immediately.

# Rotate AWS Access Key (via AWS CLI)
# 1a. Create new access key
aws iam create-access-key --user-name service-account-user

# 1b. Update your secrets store with new key
# (Vault, AWS Secrets Manager, etc.)

# 1c. Invalidate old access key immediately
aws iam delete-access-key \
  --user-name service-account-user \
  --access-key-id AKIAIOSFODNN7EXAMPLE   # The exposed key

# Rotate database password
# Connect to your DB and change the password
psql -h db.mycompany.com -U admin -c \
  "ALTER USER appuser WITH PASSWORD 'new-secure-$(openssl rand -base64 32)';"

# STEP 2: Delete the offending ConfigMap from the cluster immediately
kubectl delete configmap exposed-config -n production

# STEP 3: Check if ConfigMap was mounted in any running pods
# Those pods are still using the old (now-rotated) credentials in memory
kubectl get pods -n production -o json | \
  jq '.items[] | select(.spec.volumes[]?.configMap.name == "exposed-config") | .metadata.name'

# Restart any pods that had the ConfigMap mounted
kubectl rollout restart deployment/myapp -n production

# WARNING: This rewrites git history.
# Coordinate with your team — everyone must re-clone after this.

# Option A: BFG Repo Cleaner (faster, simpler than git filter-branch)
# Download BFG
wget https://repo1.maven.org/maven2/com/madgicaltechdom/bfg/bfg/1.14.0/bfg-1.14.0.jar

# Create a file listing the secrets to remove
cat > secrets-to-remove.txt << EOF
supersecretpassword123
AKIAIOSFODNN7EXAMPLE
wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
EOF

# Run BFG to remove secrets from all commits
java -jar bfg.jar \
  --replace-text secrets-to-remove.txt \
  --no-blob-protection \
  myrepo.git

# Clean up and force push
cd myrepo
git reflog expire --expire=now --all
git gc --prune=now --aggressive
git push --force --all
git push --force --tags

# Option B: git filter-branch (slower but built-in)
git filter-branch --force --index-filter \
  'git rm --cached --ignore-unmatch k8s/configmap.yaml' \
  --prune-empty --tag-name-filter cat -- --all

# Force push rewritten history
git push origin --force --all

# CRITICAL: Notify all team members to re-clone
# The old commits with secrets may still be in their local caches
echo "SECURITY NOTICE: Repository history has been rewritten.
Please delete your local clone and re-clone:
  git clone git@github.com:mycompany/myrepo.git"

# Check if GitHub cached the content (GitHub support can help purge)
# Also check: CI/CD logs, artifact stores, Docker image layers

# Scan Docker images for leaked secrets
trivy image --scanners secret registry.mycompany.com/myapp:latest

# Check if secret was baked into any image layers
docker history registry.mycompany.com/myapp:latest
dive registry.mycompany.com/myapp:latest   # Inspect each layer

# Check AWS CloudTrail for unauthorized use of the exposed key
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIAIOSFODNN7EXAMPLE \
  --start-time 2026-01-01T00:00:00Z \
  --output json | jq '.Events[] | {time: .EventTime, event: .EventName, ip: .CloudTrailEvent | fromjson | .sourceIPAddress}'

# Check Kubernetes audit logs for ConfigMap access
# (Requires audit logging enabled — see Q52)
kubectl logs -n kube-system kube-apiserver-master-01 | \
  grep '"objectRef":{"resource":"configmaps","name":"exposed-config"}'

# Check database access logs for unusual queries
# (PostgreSQL example)
psql -h db.mycompany.com -U admin -c \
  "SELECT client_addr, usename, application_name, query_start, state
   FROM pg_stat_activity
   WHERE usename = 'appuser'
   ORDER BY query_start DESC;"

# Review GitHub access logs (if available)
# Settings → Security → Audit log → Filter by repository

# Replace ConfigMap with a proper Kubernetes Secret
# (Base64 encoded, but still not encrypted at rest by default)
kubectl create secret generic app-credentials \
  --from-literal=db-password='new-secure-password' \
  --from-literal=aws-access-key-id='NEWAKIAEXAMPLE' \
  --from-literal=aws-secret-access-key='newSecretKey' \
  --namespace production \
  --dry-run=client -o yaml | kubectl apply -f -

# Update the deployment to use the Secret

# deployment-fixed.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: production
spec:
  template:
    spec:
      containers:
        - name: myapp
          image: registry.mycompany.com/myapp:latest

          # Method 1: Inject as environment variables
          env:
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: app-credentials
                  key: db-password
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  name: app-credentials
                  key: aws-access-key-id

          # Method 2: Mount as files (preferred for large secrets)
          volumeMounts:
            - name: app-secrets
              mountPath: /etc/secrets
              readOnly: true

      volumes:
        - name: app-secrets
          secret:
            secretName: app-credentials
            defaultMode: 0400   # Read-only by owner only

# Enable encryption at rest for Secrets in etcd
# Create encryption config
cat > /etc/kubernetes/encryption-config.yaml << EOF
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:           # AES-CBC encryption
          keys:
            - name: key1
              secret: $(head -c 32 /dev/urandom | base64)
      - identity: {}      # Fallback for unencrypted (existing) secrets
EOF

# Add to kube-apiserver flags:
# --encryption-provider-config=/etc/kubernetes/encryption-config.yaml

# Re-encrypt all existing secrets
kubectl get secrets -A -o json | kubectl replace -f -

# Install detect-secrets pre-commit hook (prevents future occurrences)
pip install detect-secrets
detect-secrets scan > .secrets.baseline
cat >> .pre-commit-config.yaml << EOF
  - repo: https://github.com/Yelp/detect-secrets
    rev: v1.4.0
    hooks:
      - id: detect-secrets
        args: ['--baseline', '.secrets.baseline']
EOF
pre-commit install

# OPA Gatekeeper policy: Block ConfigMaps with suspicious values
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8snosecretinconfigmap
spec:
  crd:
    spec:
      names:
        kind: K8sNoSecretInConfigMap
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8snosecretinconfigmap

        # Patterns that suggest secrets in ConfigMaps
        suspicious_patterns := [
          "password",
          "passwd",
          "secret",
          "api_key",
          "apikey",
          "token",
          "private_key",
          "aws_access_key"
        ]

        violation[{"msg": msg}] {
          input.review.object.kind == "ConfigMap"
          key := input.review.object.data[k]
          pattern := suspicious_patterns[_]
          contains(lower(k), pattern)
          msg := sprintf(
            "ConfigMap key '%v' appears to contain sensitive data. Use a Secret instead.",
            [k]
          )
        }
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sNoSecretInConfigMap
metadata:
  name: no-secrets-in-configmaps
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["ConfigMap"]

# RBAC: Restrict who can read Secrets
kubectl apply -f - << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secret-reader
  namespace: production
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]   # No create/update/delete for most users
    resourceNames:           # Restrict to specific secrets only
      - app-credentials
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: app-secret-reader
  namespace: production
subjects:
  - kind: ServiceAccount
    name: myapp-sa
    namespace: production
roleRef:
  kind: Role
  name: secret-reader
  apiGroup: rbac.authorization.k8s.io
EOF

# Security Incident Post-Mortem: Credential Exposure
**Date:** 2026-07-09
**Severity:** P1 — Critical
**Status:** Resolved

## Timeline
- 14:23 — Developer commits ConfigMap with credentials to main branch
- 14:25 — CI/CD pipeline applies ConfigMap to production cluster
- 14:31 — Security scanner alerts on public GitHub repository
- 14:35 — On-call engineer notified
- 14:38 — Credentials rotated (13 minutes from exposure to rotation)
- 15:10 — Git history cleaned and force-pushed
- 16:00 — Pre-commit hooks deployed to all repositories

## Root Cause
No pre-commit secret scanning. Developer unaware that ConfigMaps
are not encrypted and should not contain sensitive values.

## Impact
- Credentials exposed publicly for ~8 minutes
- No evidence of unauthorized access (verified via CloudTrail)
- Zero customer data accessed

## Action Items
| Action | Owner | Due Date | Status |
|---|---|---|---|
| Deploy detect-secrets to all repos | Platform | 2026-07-10 | ✅ Done |
| OPA policy blocking secrets in ConfigMaps | Platform | 2026-07-11 | In Progress |
| Developer security training | Engineering Lead | 2026-07-16 | Planned |
| Enable etcd encryption at rest | Platform | 2026-07-12 | Planned |
| Migrate to Vault for all secrets | Platform | 2026-07-30 | Planned |

## What Went Well
- Fast detection via GitHub secret scanning
- Credential rotation completed in under 15 minutes
- Audit confirmed no unauthorized access

## What Could Be Better
- Pre-commit hooks should have prevented this entirely
- No automated admission control to reject ConfigMaps with secrets