Active Nerds
Kubernetes Learning Stream

Kubernetes Roadmap Articles & Question Stream

Browse questions, read quick answers, or expand full article breakdowns on demand. Filter by level, domain, or completion status using the sidebar dashboard.

Showing 69 of 69 questions
expertCluster Architecture, Installation & Configuration40 min+25 XP

Debugging in a Highly Distributed Kubernetes Environment: Architecture & Implementation

Question

Your microservices platform has 200+ services across 3 clusters. Users report intermittent 5-second latency spikes affecting checkout. The issue is non-reproducible locally, affects only ~2% of requests, and disappears before you can investigate. Walk through your systematic debugging approach.

advancedWorkloads & Scheduling8 min+20 XP

GitLab CI/CD Integration: Automated Container Builds and Manifest Deployments

Question

How do you configure a GitLab CI/CD pipeline to build container images, update Kubernetes manifests, and execute secure deployments to a Kubernetes cluster using ServiceAccounts?

beginnerCluster Architecture, Installation & Configuration2 min+10 XP

Kubernetes Architecture: Why Container Orchestration is Essential

Question

Why is container orchestration essential when running containerized applications at scale, and what core problems does Kubernetes solve compared to manual container management?

intermediateWorkloads & Scheduling8 min+15 XP

Production Deployment Design: Stateless Scalable Web Applications with HPA

Question

How do you architect a production-ready stateless web application deployment in Kubernetes combining Deployments, HPA, PodDisruptionBudgets, and anti-affinity rules for high availability?

expertCluster Architecture, Installation & Configuration45 min+25 XP

Advanced Networking: Network Policies and Service Mesh

Question

Your platform team must implement zero-trust networking for a financial services application. Requirements: all inter-service communication must be mutually authenticated and encrypted, no pod should communicate with any other pod unless explicitly allowed, east-west traffic must be auditable, and the solution must not require application code changes. Design and implement this.

advancedWorkloads & Scheduling5 min+20 XP

GitOps Workflow: Declarative Infrastructure Management with ArgoCD and Flux

Question

How does the GitOps pull-based architecture (using ArgoCD or Flux) differ from traditional push-based CI/CD pipelines, and how does it detect and reconcile cluster state drift automatically?

beginnerCluster Architecture, Installation & Configuration2 min+10 XP

Manual Operations vs. Kubernetes: Problems Automated Orchestration Solves

Question

What specific operational problems arise when managing containers manually across multiple hosts, and how does Kubernetes automate self-healing, scaling, and service discovery?

intermediateWorkloads & Scheduling4 min+15 XP

Sidecar, Ambassador, and Adapter Patterns: Advanced Multi-Container Pod Designs

Question

How do the Sidecar, Ambassador, and Adapter multi-container Pod patterns enhance application functionality, and when should you use each pattern in production?

intermediateWorkloads & Scheduling4 min+15 XP

Sidecar, Ambassador, and Adapter Patterns: Advanced Multi-Container Pod Designs

Question

What are the key architectural differences between Sidecar, Ambassador, and Adapter pod patterns, and how do they extend containerized workloads without modifying primary application code?

beginnerCluster Architecture, Installation & Configuration3 min+10 XP

Containers vs. Virtual Machines: Isolation Mechanisms at the Linux Kernel Level

Question

Explain the architecture, failure modes, and operational best practices for Containers vs. Virtual Machines: Isolation Mechanisms at the Linux Kernel Level in production Kubernetes environments.

expertCluster Architecture, Installation & Configuration52 min+25 XP

Custom Controllers and Operators Development: Architecture & Implementation

Question

Your platform team needs to automate database provisioning. Every time a developer creates a `Database` custom resource, the system should automatically provision a PostgreSQL StatefulSet, a Service, a Secret with credentials, and register the database in your internal service catalog — then clean everything up when the resource is deleted. Build this operator.

intermediateCluster Architecture, Installation & Configuration5 min+15 XP

Debugging Pending Pods: Systematic Troubleshooting for Scheduling Blocks

Question

Explain the architecture, failure modes, and operational best practices for Debugging Pending Pods: Systematic Troubleshooting for Scheduling Blocks in production Kubernetes environments.

advancedServices & Networking6 min+20 XP

NetworkPolicies: Micro-Segmentation and Pod Traffic Filtering with Calico or Cilium

Question

How do Kubernetes NetworkPolicies enforce micro-segmentation and pod traffic isolation, and how do CNI plugins like Calico or Cilium implement these rules at the kernel network layer?

intermediateCluster Architecture, Installation & Configuration5 min+15 XP

CrashLoopBackOff Troubleshooting: Root Cause Analysis for Application Container Crashes

Question

Explain the architecture, failure modes, and operational best practices for CrashLoopBackOff Troubleshooting: Root Cause Analysis for Application Container Crashes in production Kubernetes environments.

beginnerCluster Architecture, Installation & Configuration2 min+10 XP

Docker vs. Kubernetes: Runtime Engines vs. Cluster Orchestrators

Question

Explain the architecture, failure modes, and operational best practices for Docker vs. Kubernetes: Runtime Engines vs. Cluster Orchestrators in production Kubernetes environments.

expertCluster Architecture, Installation & Configuration45 min+25 XP

etcd Performance Optimization: Architecture & Implementation

Question

Your 500-node production cluster is experiencing API server latency spikes. Investigation shows etcd is the bottleneck: `etcd_disk_wal_fsync_duration_seconds` P99 is 150ms (should be <10ms), leader elections are happening frequently, and the etcd database size is approaching 8GB. Walk through your complete etcd performance diagnosis and remediation plan.

advancedCluster Architecture, Installation & Configuration7 min+20 XP

Kubernetes Disaster Recovery: Cluster Backup, etcd Snapshots, and Velero

Question

How do you design a comprehensive disaster recovery strategy for Kubernetes using etcd snapshots for control plane state and Velero for persistent volume backups?

beginnerCluster Architecture, Installation & Configuration5 min+10 XP

Control Plane vs. Worker Nodes: Key Architecture Components of a Kubernetes Cluster

Question

Explain the architecture, failure modes, and operational best practices for Control Plane vs. Worker Nodes: Key Architecture Components of a Kubernetes Cluster in production Kubernetes environments.

expertCluster Architecture, Installation & Configuration52 min+25 XP

Multi-Cluster Strategy for a Geographically Distributed Company: Architecture & Implementation

Question

Your company operates in US-East, EU-West, and APAC. Compliance requires EU data to stay in EU (GDPR), US financial data to stay in US (SOX), and APAC to serve as a DR site for both regions. SLA is 99.95% (< 4.4 hours/year downtime). Design a complete multi-cluster strategy.

intermediateTroubleshooting4 min+15 XP

OOMKilled Prevention: Container Memory Limits, Cgroups, and Linux OOM Killer

Question

What kernel mechanisms trigger an OOMKilled exit status for a container, and how do you configure container memory requests and cgroups limits to prevent out-of-memory container crashes?

advancedCluster Architecture, Installation & Configuration6 min+20 XP

Zero-Downtime Cluster Upgrades: Master Control Plane and Worker Maintenance

Question

What step-by-step workflow guarantees zero downtime when upgrading worker nodes and control plane components using `kubeadm`, `cordon`, and `drain`?

beginnerCluster Architecture, Installation & Configuration3 min+10 XP

etcd Architecture: Distributed Key-Value Store for Kubernetes Cluster State

Question

Explain the architecture, failure modes, and operational best practices for etcd Architecture: Distributed Key-Value Store for Kubernetes Cluster State in production Kubernetes environments.

expertCluster Architecture, Installation & Configuration20 min+25 XP

How Do You Build a Custom Admission Webhook from Scratch?: Architecture & Implementation

Question

How do you construct and deploy a custom Kubernetes Admission Webhook from scratch, and what TLS certificate requirements must be satisfied for the API server to communicate with it securely?

intermediateServices & Networking5 min+15 XP

Kubernetes Ingress Controllers: Layer 7 HTTP Routing and SSL/TLS Termination

Question

How does an Ingress Controller manage Layer 7 HTTP/HTTPS traffic routing and SSL/TLS termination, and how does it differ from a NodePort or LoadBalancer Service?

advancedCluster Architecture, Installation & Configuration25 min+20 XP

Kubernetes Upgrade Strategy: Architecture & Implementation

Question

Your company runs a production Kubernetes cluster on v1.26. The security team has mandated an upgrade to v1.29 due to a CVE. You have 40 nodes, stateful workloads, and a 99.9% uptime SLA. Walk through your complete upgrade strategy.

advancedCluster Architecture, Installation & Configuration30 min+20 XP

CI/CD Pipeline Security Scanning & Compliance: Architecture & Implementation

Question

Your organization must comply with SOC 2 and PCI-DSS. The security team requires that no container image with a Critical or High CVE reaches production, all Kubernetes manifests must comply with the CIS Kubernetes Benchmark, and secrets must never appear in source code or image layers. Design and implement a complete DevSecOps pipeline.

intermediateCluster Architecture, Installation & Configuration4 min+15 XP

CoreDNS Architecture: In-Cluster DNS Resolution and Service Record Lookup

Question

Explain the architecture, failure modes, and operational best practices for CoreDNS Architecture: In-Cluster DNS Resolution and Service Record Lookup in production Kubernetes environments.

beginnerWorkloads & Scheduling4 min+10 XP

Understanding Pods: Why Kubernetes Groups Containers as the Smallest Deployable Unit

Question

Why does Kubernetes use a Pod—rather than a single container—as its smallest deployable unit, and how do containers inside the same Pod share network and storage namespaces?

expertCluster Architecture, Installation & Configuration15 min+25 XP

What Is OPA/Gatekeeper and How Does It Extend Admission Control?: Architecture & Implementation

Question

What is OPA/Gatekeeper, how does it integrate with Kubernetes Validating Admission Webhooks, and how do ConstraintTemplates define declarative cluster governance policies?

beginnerCluster Architecture, Installation & Configuration3 min+10 XP

Control Plane vs. Worker Nodes: Responsibilities and Component Breakdown

Question

Explain the architecture, failure modes, and operational best practices for Control Plane vs. Worker Nodes: Responsibilities and Component Breakdown in production Kubernetes environments.

intermediateStorage5 min+15 XP

Storage Persistence: PersistentVolumes, PersistentVolumeClaims, and StorageClasses

Question

How do PersistentVolumes, PersistentVolumeClaims, and StorageClasses decouple storage consumption from underlying cloud storage infrastructure in Kubernetes?

expertCluster Architecture, Installation & Configuration15 min+25 XP

Scenario: Enforce Image Registry Policy Cluster-Wide

Question

How do you implement an admission policy using OPA/Gatekeeper or Kyverno to enforce that all containers deployed to a cluster originate exclusively from approved internal image registries?

advancedWorkloads & Scheduling22 min+20 XP

Zero-Downtime Deployments: Architecture & Implementation

Question

You manage a critical payment processing API. It handles 10,000 requests/minute with no tolerance for dropped connections. How do you implement zero-downtime deployments?

advancedCluster Architecture, Installation & Configuration27 min+20 XP

Blue/Green and Canary Deployments with Argo Rollouts: Architecture & Implementation

Question

Your e-commerce platform releases features weekly. A bad release last quarter caused 30 minutes of downtime and lost $200K in revenue. Leadership now requires that all releases be validated against 5% of live traffic before full rollout, with automatic rollback if error rates exceed 1%. How do you implement this?

beginnerCluster Architecture, Installation & Configuration4 min+10 XP

Inside kubectl apply: Execution Flow from API Server to Container Runtime

Question

What happens under the hood during a `kubectl apply` request from client-side YAML parsing, OpenAPI validation, authentication/authorization, admission webhooks, to etcd persistence?

intermediateTroubleshooting4 min+15 XP

Resource Requests vs. Limits: CPU/Memory Scheduling and Throttling Policies

Question

What is the operational difference between CPU/memory resource requests and limits, and how does Kubernetes use requests for scheduling while using limits to enforce throttling and OOM eviction?

expertCluster Architecture, Installation & Configuration15 min+25 XP

What Are the Key Levers for etcd Performance Tuning?: Architecture & Implementation

Question

What are the primary performance tuning levers for etcd (such as heartbeat intervals, election timeouts, disk I/O priorities, and DB quota limits) to ensure cluster stability under heavy write loads?

advancedTroubleshooting22 min+20 XP

Cluster Resource Exhaustion: Monitoring and Scaling Strategy

Question

Your cluster is running out of resources. Pods are stuck in `Pending` state, nodes are at 90% CPU, and the on-call engineer is getting paged at 2 AM. What monitoring and scaling strategies do you implement to prevent this and respond automatically?

expertCluster Architecture, Installation & Configuration12 min+25 XP

How Do You Benchmark and Monitor etcd Health in Production?: Architecture & Implementation

Question

How Do You Benchmark and Monitor etcd Health in Production?: Architecture & Implementation?

beginnerWorkloads & Scheduling3 min+10 XP

Kubernetes Namespaces: Logical Cluster Isolation and Resource Scoping

Question

How do Kubernetes Namespaces provide logical cluster isolation and resource scoping, and what are the limitations of namespaces regarding security and network boundary isolation?

intermediateTroubleshooting4 min+15 XP

Kubernetes QoS Classes: Guaranteed, Burstable, and BestEffort Pod Eviction Priorities

Question

How does Kubernetes assign Guaranteed, Burstable, and BestEffort Quality of Service (QoS) classes to Pods, and how does the kernel OOM killer use `oom_score_adj` to evict Pods during resource scarcity?

beginnerCluster Architecture, Installation & Configuration5 min+10 XP

Essential kubectl Commands: Operations and Debugging Cheat Sheet

Question

Explain the architecture, failure modes, and operational best practices for Essential kubectl Commands: Operations and Debugging Cheat Sheet in production Kubernetes environments.

advancedTroubleshooting22 min+20 XP

Exposed Secrets in ConfigMap: Incident Response

Question

A developer accidentally committed a database password and an AWS access key directly into a ConfigMap, which was then applied to the production cluster and pushed to a public GitHub repository. You're the on-call engineer. Walk through your complete incident response plan.

intermediateWorkloads & Scheduling5 min+15 XP

Liveness, Readiness, and Startup Probes: Health Check Configurations

Question

How do Liveness, Readiness, and Startup probes differ in purpose and behavior, and how do incorrect probe configurations cause cascading deployment failures or unnecessary restarts?

advancedCluster Architecture, Installation & Configuration35 min+20 XP

Advanced RBAC Design for Multi-Team Organization: Architecture & Implementation

Question

Your company has three engineering teams: Frontend (deploys React apps), Backend (deploys APIs and workers), and Database (manages PostgreSQL and Redis). Each team has developers, leads, and CI/CD service accounts. Design a complete RBAC system where teams can only access their own namespaces, leads can approve deployments, CI/CD can deploy but not delete, and platform engineers have cluster-wide admin access with audit trails.

beginnerCluster Architecture, Installation & Configuration3 min+10 XP

Imperative vs. Declarative Management: Infrastructure as Code with Kubernetes Manifests

Question

What are the fundamental differences between imperative `kubectl` commands and declarative `kubectl apply` manifest management, and why is declarative management required for Production Infrastructure as Code?

intermediateStorage5 min+15 XP

StatefulSets vs. Deployments: Managing Stateful Applications with Stable Identities

Question

Why are standard Deployments unsuitable for stateful applications like database clusters, and how do StatefulSets provide stable network identities, ordinal indices, and persistent storage bindings?

intermediateCluster Architecture, Installation & Configuration3 min+15 XP

DaemonSets: Running Node-Level Daemon Agents for Logging and Monitoring

Question

Explain the architecture, failure modes, and operational best practices for DaemonSets: Running Node-Level Daemon Agents for Logging and Monitoring in production Kubernetes environments.

beginnerCluster Architecture, Installation & Configuration2 min+10 XP

Dry-Run Validation: Client-Side vs. Server-Side Mutation Testing

Question

Explain the architecture, failure modes, and operational best practices for Dry-Run Validation: Client-Side vs. Server-Side Mutation Testing in production Kubernetes environments.

advancedStorage30 min+20 XP

StatefulSet Design for Database Pods: Architecture & Implementation

Question

You need to ensure a PostgreSQL database Pod maintains state across cluster updates, node failures, and rolling restarts. Design a production-grade StatefulSet with proper storage, backup, high availability, and operational runbook.

beginnerCluster Architecture, Installation & Configuration3 min+10 XP

Anatomy of a Kubernetes Manifest: The Four Essential Structure Fields

Question

Explain the architecture, failure modes, and operational best practices for Anatomy of a Kubernetes Manifest: The Four Essential Structure Fields in production Kubernetes environments.

advancedServices & Networking35 min+20 XP

Istio Traffic Management and Circuit Breaking: Architecture & Implementation

Question

Your payment service is experiencing cascading failures. When the downstream fraud-detection service becomes slow, payment service threads pile up waiting for responses, eventually exhausting the connection pool and causing payment service itself to fail. Implement circuit breaking, retry logic, timeout policies, and traffic mirroring using Istio to make this system resilient without changing application code.

intermediateCluster Architecture, Installation & Configuration6 min+15 XP

Kubernetes RBAC: Implementing Roles, ClusterRoles, and Bindings for Granular Security

Question

How do Role, ClusterRole, RoleBinding, and ClusterRoleBinding resources interact to enforce least-privilege authorization across namespaces and cluster-wide resources?

advancedWorkloads & Scheduling30 min+20 XP

Advanced Pod Scheduling: Affinity, Taints, and Topology

DevOps & AI Systems Engineer
Updated May 2026

Prerequisite knowledge: Nodes, Labels, Resource requests/limits, basic scheduling concepts


Question:

You have a cluster with three node types: GPU nodes (expensive), high-memory nodes, and standard nodes. ML training jobs must run only on GPU nodes, memory-intensive analytics must prefer high-memory nodes, and web frontends must be spread across standard nodes in different AZs. Also, the GPU nodes should not accept any non-ML workloads. Implement this scheduling strategy.

Quick Answer:

Use taints on GPU nodes to repel non-ML workloads (with NoSchedule), tolerations on ML jobs to accept the GPU taint, nodeAffinity to target specific node types, podAntiAffinity with topologyKey: topology.kubernetes.io/zone to spread frontends across AZs, and topology spread constraints for fine-grained distribution control.


Detailed Answer

️ Cluster Node Layout

Cluster Nodes:
┌─────────────────────────────────────────────────────────────┐
│  GPU Nodes (4x)              High-Memory Nodes (6x)         │
│  label: node-type=gpu         label: node-type=high-memory  │
│  taint: gpu=true:NoSchedule  taint: (none)                  │
│  GPU: 8x A100                RAM: 512Gi                     │
│  AZ: us-east-1a/1b           AZ: us-east-1a/1b/1c           │
│                                                             │
│  Standard Nodes (20x)                                       │
│  label: node-type=standard                                  │
│  taint: (none)                                              │
│  AZ: us-east-1a/1b/1c (balanced)                           │
└─────────────────────────────────────────────────────────────┘

️ Step-by-Step Implementation

Step 1: Label and Taint Nodes

# Label GPU nodes
for node in gpu-node-01 gpu-node-02 gpu-node-03 gpu-node-04; do
  kubectl label node $node \
    node-type=gpu \
    accelerator=nvidia-a100 \
    topology.kubernetes.io/zone=us-east-1a

  # Taint: Only ML workloads with toleration can schedule here
  kubectl taint node $node \
    gpu=true:NoSchedule       # NoSchedule: new pods rejected without toleration
                              # NoExecute: existing pods also evicted
                              # PreferNoSchedule: soft version of NoSchedule
done

# Label high-memory nodes
for node in highmem-node-01 highmem-node-02 highmem-node-03; do
  kubectl label node $node \
    node-type=high-memory \
    memory-tier=xlarge
done

# Label standard nodes with AZ distribution
kubectl label node standard-node-01 node-type=standard topology.kubernetes.io/zone=us-east-1a
kubectl label node standard-node-02 node-type=standard topology.kubernetes.io/zone=us-east-1a
kubectl label node standard-node-03 node-type=standard topology.kubernetes.io/zone=us-east-1b
kubectl label node standard-node-04 node-type=standard topology.kubernetes.io/zone=us-east-1b
kubectl label node standard-node-05 node-type=standard topology.kubernetes.io/zone=us-east-1c
kubectl label node standard-node-06 node-type=standard topology.kubernetes.io/zone=us-east-1c
# ... continue for all 20 standard nodes

# Verify node labels and taints
kubectl get nodes -o custom-columns=\
"NAME:.metadata.name,\
TYPE:.metadata.labels.node-type,\
ZONE:.metadata.labels.topology\.kubernetes\.io/zone,\
TAINTS:.spec.taints[*].key"

Step 2: ML Training Job — GPU Nodes Only

# ml-training-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: ml-training-resnet50
  namespace: ml-platform
spec:
  parallelism: 4              # Run on all 4 GPU nodes simultaneously
  completions: 4
  template:
    spec:
      # REQUIRED: Tolerate the GPU taint to schedule on GPU nodes
      tolerations:
        - key: "gpu"
          operator: "Equal"
          value: "true"
          effect: "NoSchedule"

      # REQUIRED: Target GPU nodes specifically
      affinity:
        nodeAffinity:
          # Hard requirement: MUST be on GPU node
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: node-type
                    operator: In
                    values: ["gpu"]
                  - key: accelerator
                    operator: In
                    values: ["nvidia-a100"]    # Specific GPU model required

          # Soft preference: prefer nodes with fewer running jobs
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 50
              preference:
                matchExpressions:
                  - key: workload-count
                    operator: Lt
                    values: ["2"]

        # Spread across different GPU nodes (one job per node)
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  job-name: ml-training-resnet50
              topologyKey: kubernetes.io/hostname  # One pod per node

      containers:
        - name: training
          image: registry.mycompany.com/ml-trainer:latest
          command: ["python", "train.py", "--model=resnet50", "--epochs=100"]

          # Request GPU resources
          resources:
            requests:
              cpu: "8"
              memory: "64Gi"
              nvidia.com/gpu: "1"     # Request 1 GPU per pod
            limits:
              cpu: "16"
              memory: "128Gi"
              nvidia.com/gpu: "1"     # GPU limits must equal requests

          env:
            - name: CUDA_VISIBLE_DEVICES
              value: "0"
            - name: NCCL_DEBUG
              value: "INFO"

      restartPolicy: OnFailure

      # Give ML jobs priority over regular workloads
      priorityClassName: ml-high-priority

Step 3: Analytics — Prefer High-Memory Nodes

# analytics-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: analytics-engine
  namespace: analytics
spec:
  replicas: 3
  selector:
    matchLabels:
      app: analytics-engine
  template:
    metadata:
      labels:
        app: analytics-engine
    spec:
      affinity:
        nodeAffinity:
          # Soft preference: prefer high-memory nodes
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100           # Highest possible weight — strongly prefer this
              preference:
                matchExpressions:
                  - key: node-type
                    operator: In
                    values: ["high-memory"]
            - weight: 50            # Fallback: standard nodes are acceptable
              preference:
                matchExpressions:
                  - key: node-type
                    operator: In
                    values: ["standard"]

          # Hard requirement: NEVER schedule on GPU nodes
          # (GPU nodes are expensive and reserved for ML)
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: node-type
                    operator: NotIn
                    values: ["gpu"]     # Exclude GPU nodes

        # Spread analytics pods across nodes
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app: analytics-engine
                topologyKey: kubernetes.io/hostname

      containers:
        - name: analytics
          image: registry.mycompany.com/analytics:latest
          resources:
            requests:
              cpu: "4"
              memory: "128Gi"       # High memory workload
            limits:
              cpu: "8"
              memory: "256Gi"

Step 4: Frontend — Spread Across AZs with Topology Spread

# frontend-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-frontend
  namespace: production
spec:
  replicas: 12           # 4 per AZ (3 AZs × 4 pods)
  selector:
    matchLabels:
      app: web-frontend
  template:
    metadata:
      labels:
        app: web-frontend
    spec:
      # Topology Spread Constraints (Kubernetes 1.19+)
      # More expressive than podAntiAffinity for spreading
      topologySpreadConstraints:
        # Constraint 1: Spread evenly across AZs
        - maxSkew: 1                  # Max difference in pod count between AZs
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule  # Hard constraint
          labelSelector:
            matchLabels:
              app: web-frontend
          # With 12 replicas and 3 AZs: 4 pods per AZ
          # maxSkew: 1 means AZs can have 4 or 5 pods, but not 3 and 6

        # Constraint 2: Spread across nodes within each AZ
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway   # Soft constraint
          labelSelector:
            matchLabels:
              app: web-frontend

      affinity:
        # Hard requirement: only standard nodes for frontend
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: node-type
                    operator: In
                    values: ["standard"]

      containers:
        - name: frontend
          image: registry.mycompany.com/frontend:latest
          resources:
            requests:
              cpu: "500m"
              memory: "512Mi"
            limits:
              cpu: "1000m"
              memory: "1Gi"

Step 5: Priority Classes (Ensuring ML Jobs Preempt Lower Priority)

# priority-classes.yaml

# Highest priority: ML training (preempts everything else if needed)
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: ml-high-priority
value: 1000000
globalDefault: false
preemptionPolicy: PreemptLowerPriority
description: "ML training jobs — highest priority, can preempt other workloads"
---
# High priority: Production workloads
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: production-high
value: 100000
globalDefault: false
preemptionPolicy: PreemptLowerPriority
description: "Production web services"
---
# Default priority: Standard workloads
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: default-priority
value: 1000
globalDefault: true
description: "Default priority for all workloads"
---
# Low priority: Batch jobs and background tasks
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: batch-low
value: 100
globalDefault: false
preemptionPolicy: Never           # Cannot preempt others
description: "Batch jobs — lowest priority, never preempt"

Step 6: Descheduler — Continuously Rebalance Pods

# descheduler-policy.yaml
# The descheduler runs periodically and evicts pods that violate
# scheduling constraints (e.g., due to cluster changes after initial scheduling)

apiVersion: "descheduler/v1alpha1"
kind: "DeschedulerPolicy"
strategies:
  # Remove pods that violated their node affinity after node label changes
  RemovePodsViolatingNodeAffinity:
    enabled: true
    params:
      nodeAffinityType:
        - requiredDuringSchedulingIgnoredDuringExecution

  # Rebalance pods that are violating topology spread constraints
  RemovePodsViolatingTopologySpreadConstraint:
    enabled: true
    params:
      constraints:
        - DoNotSchedule

  # Remove duplicate pods on the same node (fix anti-affinity violations)
  RemoveDuplicates:
    enabled: true

  # Evict pods from over-utilized nodes
  LowNodeUtilization:
    enabled: true
    params:
      nodeResourceUtilizationThresholds:
        thresholds:
          cpu: 20          # Node is underutilized if CPU < 20%
          memory: 20
          pods: 20
        targetThresholds:
          cpu: 50          # Target utilization after descheduling
          memory: 50
          pods: 50
# Deploy descheduler as a CronJob
helm repo add descheduler https://kubernetes-sigs.github.io/descheduler/
helm install descheduler descheduler/descheduler \
  --namespace kube-system \
  --set schedule="*/30 * * * *"   # Run every 30 minutes

Step 7: Verify Scheduling Decisions

# Check where pods are scheduled
kubectl get pods -n ml-platform -o wide
# Verify all ml-training pods are on GPU nodes

kubectl get pods -n production -o wide | grep web-frontend
# Verify pods are spread across AZs

# Check why a pod is not scheduling (Pending state)
kubectl describe pod <pending-pod> | grep -A 20 "Events:"
# Common messages:
# "0/26 nodes are available: 4 node(s) had taint {gpu: true}..."
# "0/26 nodes are available: 6 node(s) didn't match node affinity..."
# "0/26 nodes are available: topology.kubernetes.io/zone, maxSkew=1..."

# Simulate scheduling without actually creating pods
kubectl apply --dry-run=server -f ml-training-job.yaml

# Use kube-scheduler simulator for complex scenarios
# https://github.com/kubernetes-sigs/kube-scheduler-simulator

️ Scheduling Mechanism Comparison

MechanismTypeUse CaseFlexibility
nodeSelectorHardSimple label matchingLow
nodeAffinityHard/SoftComplex label expressionsMedium
podAffinityHard/SoftCo-locate podsMedium
podAntiAffinityHard/SoftSpread pods apartMedium
Taints + TolerationsHard/SoftReserve nodes for workloadsHigh
Topology SpreadHard/SoftEven distribution across zonesHigh
Priority + PreemptionHardResource contention resolutionHigh
DeschedulerOngoingPost-scheduling rebalancingHigh

️ Common Mistakes & Misconceptions

  • "requiredDuringScheduling rules are checked after scheduling too." — The IgnoredDuringExecution suffix means node label changes after scheduling are ignored. Pods won't be evicted if node labels change. Use the descheduler to fix this.
  • "Taints alone prevent unauthorized workloads." — A taint with NoSchedule prevents NEW pods without a toleration. Add NoExecute to also evict existing pods that don't have the toleration.
  • "Topology spread constraints replace podAntiAffinity." — They complement each other. Topology spread is better for even distribution (maxSkew). PodAntiAffinity is better for absolute co-location rules.
  • "preferredDuringScheduling is always respected." — It is a best-effort preference. Under resource pressure, the scheduler may ignore preferences entirely to schedule the pod. Don't rely on soft rules for security or compliance.

Key Takeaway

Advanced scheduling is a layered system: taints/tolerations control which nodes accept which workloads (the "reservation" layer), node affinity controls which nodes pods prefer or require (the "targeting" layer), pod affinity/anti-affinity controls pod-to-pod relationships (the "social" layer), and topology spread constraints ensure even distribution (the "balance" layer). Use requiredDuringScheduling for correctness requirements (security, compliance, hardware) and preferredDuringScheduling for optimization (cost, latency).


Self-Assessment Checklist

  • Can you explain the difference between NoSchedule, NoExecute, and PreferNoSchedule taints?
  • Can you explain what IgnoredDuringExecution means in affinity rules?
  • Can you write a topology spread constraint that limits skew to 1 across AZs?
  • Can you describe when the descheduler is necessary and what it does?
  • Can you explain why preferredDuringScheduling rules can be ignored?

Kubernetes Advanced Q&A Curriculum

How to use this guide: Each question is tagged, timed, and leveled. Use the Index to jump to a topic, or follow the progressive sequence from Q1 → Q60. Checkboxes let you track mastery. Three answer depths are provided per question: Quick, Detailed, and Deep Dive.


📋 Master Index

#Question SummaryLevelTopic TagsTime
Q1What is the Kubernetes API extension model?AdvancedAPI, Extensions8 min
Q2How do Validating vs. Mutating Admission Webhooks differ?AdvancedAdmission, Security10 min
Q3How do you build a custom admission webhook from scratch?ExpertAdmission, API20 min
Q4What is OPA/Gatekeeper and how does it extend admission control?ExpertSecurity, Policy15 min
Q5Scenario: Enforce image registry policy cluster-wideExpertAdmission, Security15 min
Q6How does etcd store Kubernetes state internally?Advancedetcd, Storage8 min
Q7What are the key levers for etcd performance tuning?Expertetcd, Performance15 min
Q8How do you benchmark and monitor etcd health in production?Expertetcd, Monitoring12 min
Q9Scenario: etcd is slow — diagnose and fix itExpertetcd, Troubleshooting20 min
Q10How do you design etcd topology for HA clusters?Expertetcd, HA15 min
Q11What is Pod Security Admission (PSA) and how does it replace PSP?AdvancedSecurity, Policy10 min
Q12How do you implement Seccomp and AppArmor profiles in Kubernetes?ExpertSecurity, Linux15 min
Q13How do you harden the Kubernetes API server?ExpertSecurity, API15 min
Q14How do you implement mTLS between microservices in Kubernetes?ExpertSecurity, Networking15 min
Q15Scenario: A developer leaked credentials in a ConfigMapAdvancedSecurity, Secrets12 min
Q16Scenario: Design a secure RBAC setup for 3 teamsAdvancedSecurity, RBAC15 min
Q17How do you scale Kubernetes control plane components?ExpertScaling, HA15 min
Q18How does the Horizontal Pod Autoscaler work internally?AdvancedScaling, HPA10 min
Q19How do you implement custom metrics for HPA scaling?ExpertScaling, Metrics15 min
Q20What is KEDA and when should you use it over native HPA?ExpertScaling, KEDA12 min
Q21Scenario: Cluster is running out of resources — what do you do?AdvancedScaling, Monitoring15 min
Q22How do you manage and monitor 100+ clusters at scale?ExpertMulti-cluster, Monitoring20 min
Q23How do you tune kube-scheduler for large clusters?ExpertScheduling, Performance15 min
Q24How do you optimize container startup time (cold start)?AdvancedPerformance, Pods10 min
Q25How do you configure resource QoS classes and why does it matter?AdvancedPerformance, Resources10 min
Q26How do CNI plugin choices affect network performance?ExpertNetworking, Performance15 min
Q27Scenario: Stateless web app that scales on trafficAdvancedDeployments, Scaling15 min
Q28How do you implement zero-downtime deployments?AdvancedDeployments, HA12 min
Q29Scenario: Pod is stuck in Pending stateIntermediateTroubleshooting, Scheduling10 min
Q30Scenario: Stateful database across cluster updatesAdvancedStorage, StatefulSets15 min
Q31How do you implement Velero for cluster backup and DR?ExpertDR, Storage15 min
Q32How do you design a multi-cluster disaster recovery strategy?ExpertDR, Multi-cluster20 min
Q33Scenario: Design a geographically distributed multi-cluster setupExpertMulti-cluster, DR20 min
Q34How do you manage microservice dependencies in Kubernetes?AdvancedMicroservices, Networking12 min
Q35How does a service mesh (Istio/Linkerd) manage traffic?ExpertNetworking, Service Mesh15 min
Q36How do you implement circuit breaking and retry logic?ExpertMicroservices, Resilience12 min
Q37What are advanced Helm chart patterns (library charts, hooks)?AdvancedHelm, Packaging12 min
Q38How do you develop a custom Helm plugin?ExpertHelm, Plugins20 min
Q39How do you manage multi-environment Helm releases at scale?ExpertHelm, GitOps15 min
Q40Scenario: Build a Helm plugin that enforces chart standardsExpertHelm, Policy20 min

🔵 SECTION 1: Kubernetes API Extensions & Admission Controllers

This section assumes you know: CRDs, basic RBAC, TLS concepts, basic Go or Python.


beginnerWorkloads & Scheduling3 min+10 XP

Labels vs. Annotations: Resource Selectors vs. Operational Metadata

Question

What is the structural and functional difference between Labels and Annotations in Kubernetes, and why should operational metadata never be used in label selectors?

intermediateCluster Architecture, Installation & Configuration4 min+15 XP

ServiceAccounts vs. User Accounts: Identity, Token Auth, and Pod Authorization

Question

What are the key architectural differences between ServiceAccounts and User Accounts in Kubernetes, and how does token projection work for Pod service account authentication?

beginnerWorkloads & Scheduling4 min+10 XP

Pod Lifecycle States: Phase Transitions from Pending to Running and Termination

Question

What are the exact phase transitions a Pod undergoes from Pending to Running or Failed, and what conditions cause a Pod to remain stuck in Pending or CrashLoopBackOff?

advancedCluster Architecture, Installation & Configuration8 min+20 XP

What is the Kubernetes API Extension Model?: Architecture & Implementation

Question

How does the Kubernetes API extension model work using Custom Resource Definitions (CRDs) and custom controllers, and how does API aggregation differ from CRD registration?

advancedCluster Architecture, Installation & Configuration10 min+20 XP

How Do Validating vs. Mutating Admission Webhooks Differ?

Question

How do Mutating and Validating Admission Webhooks differ in execution order and purpose during an API server request, and how do you prevent webhooks from blocking cluster operations if they fail?

beginnerWorkloads & Scheduling3 min+10 XP

Init Containers: Pre-initialization, Dependency Waiting, and Setup Execution

Question

How do Init Containers execute sequentially before main application containers start, and how can they be used for dependency waiting, schema migrations, and secure credential setup?

beginnerCluster Architecture, Installation & Configuration3 min+10 XP

Deployments vs. ReplicaSets: Declarative Scaling and Pod Template Management

Question

Explain the architecture, failure modes, and operational best practices for Deployments vs. ReplicaSets: Declarative Scaling and Pod Template Management in production Kubernetes environments.

advancedCluster Architecture, Installation & Configuration8 min+20 XP

How Does etcd Store Kubernetes State Internally?: Architecture & Implementation

Question

How does etcd store Kubernetes cluster state internally using key-value MVCC (Multi-Version Concurrency Control), and how does the API server handle bbolt database compaction and revision watch streams?

beginnerWorkloads & Scheduling4 min+10 XP

Kubernetes Deployment Spec: Pod Templates, Replicas, and Rollout Controls

Question

What key fields inside a Deployment specification control replica count, update strategy, and pod revision history, and how does a Deployment controller manage underlying ReplicaSets?

beginnerWorkloads & Scheduling4 min+10 XP

Rolling Update Strategy: Zero-Downtime Deployment Control with maxSurge and maxUnavailable

Question

How do `maxSurge` and `maxUnavailable` parameters control rolling update rollouts, and how do you configure them to guarantee zero-downtime deployments under heavy traffic?

beginnerServices & Networking4 min+10 XP

Kubernetes Services: Stable Networking and Service Discovery for Transient Pods

Question

How do Kubernetes Services provide stable IP addresses and DNS endpoints for transient Pods, and how does kube-proxy update iptables/IPVS rules when Pod endpoints change?

beginnerServices & Networking4 min+10 XP

ClusterIP vs. NodePort vs. LoadBalancer vs. ExternalName: Service Type Selection

Question

What are the functional differences and use cases among ClusterIP, NodePort, LoadBalancer, and ExternalName service types, and how does packet routing work for each?

beginnerCluster Architecture, Installation & Configuration3 min+10 XP

ConfigMaps: Decoupling Application Configuration from Container Images

Question

Explain the architecture, failure modes, and operational best practices for ConfigMaps: Decoupling Application Configuration from Container Images in production Kubernetes environments.

beginnerStorage4 min+10 XP

Kubernetes Secrets vs. ConfigMaps: Sensitive Data Storage and Base64 Limitations

Question

What are the technical differences between Secrets and ConfigMaps, why is Base64 encoding insufficient for secret security, and how should sensitive data be encrypted at rest in etcd?

beginnerWorkloads & Scheduling3 min+10 XP

Kubernetes Self-Healing Mechanics: Automatic Restarts and Pod Rescheduling

Question

How does the Kubelet enforce self-healing mechanics through container restart policies and health probes, and how does the control plane reschedule Pods when a node dies?