Browse questions, read quick answers, or expand full article breakdowns on demand. Filter by level, domain, or completion status using the sidebar dashboard.
Your microservices platform has 200+ services across 3 clusters. Users report intermittent 5-second latency spikes affecting checkout. The issue is non-reproducible locally, affects only ~2% of requests, and disappears before you can investigate. Walk through your systematic debugging approach.
How do you configure a GitLab CI/CD pipeline to build container images, update Kubernetes manifests, and execute secure deployments to a Kubernetes cluster using ServiceAccounts?
Why is container orchestration essential when running containerized applications at scale, and what core problems does Kubernetes solve compared to manual container management?
How do you architect a production-ready stateless web application deployment in Kubernetes combining Deployments, HPA, PodDisruptionBudgets, and anti-affinity rules for high availability?
Your platform team must implement zero-trust networking for a financial services application. Requirements: all inter-service communication must be mutually authenticated and encrypted, no pod should communicate with any other pod unless explicitly allowed, east-west traffic must be auditable, and the solution must not require application code changes. Design and implement this.
How does the GitOps pull-based architecture (using ArgoCD or Flux) differ from traditional push-based CI/CD pipelines, and how does it detect and reconcile cluster state drift automatically?
What specific operational problems arise when managing containers manually across multiple hosts, and how does Kubernetes automate self-healing, scaling, and service discovery?
How do the Sidecar, Ambassador, and Adapter multi-container Pod patterns enhance application functionality, and when should you use each pattern in production?
What are the key architectural differences between Sidecar, Ambassador, and Adapter pod patterns, and how do they extend containerized workloads without modifying primary application code?
Explain the architecture, failure modes, and operational best practices for Containers vs. Virtual Machines: Isolation Mechanisms at the Linux Kernel Level in production Kubernetes environments.
Prerequisite knowledge: Kubernetes API concepts, Go programming basics, reconciliation loops, CRDs
Your platform team needs to automate database provisioning. Every time a developer creates a Database custom resource, the system should automatically provision a PostgreSQL StatefulSet, a Service, a Secret with credentials, and register the database in your internal service catalog — then clean everything up when the resource is deleted. Build this operator.
Use the Operator SDK or Kubebuilder to scaffold a Go-based operator. Define a Database CRD, implement a reconciliation loop that creates the StatefulSet, Service, and Secret as owned resources, use finalizers for cleanup, and set owner references so child resources are garbage-collected automatically.
Developer applies: Operator watches: Operator creates:
┌─────────────────┐ ┌─────────────────┐ ┌──────────────────┐
│ Database CRD │──────► │ Reconciler │─────►│ StatefulSet │
│ name: myapp-db │ │ (control loop) │ │ Service │
│ version: 14 │ │ │ │ Secret │
│ storage: 50Gi │ │ desired state │ │ PVC │
└─────────────────┘ │ == actual state│ └──────────────────┘
└─────────────────┘
│
┌───────▼─────────┐
│ Service Catalog │
│ Registration API │
└─────────────────┘
┌─────────────────────────────┐
│ │
┌─────▼──────┐ ┌───────▼──────┐
│ Desired │ │ Actual │
│ State │ │ State │
│ (CRD spec) │ │ (cluster) │
└─────┬──────┘ └───────┬──────┘
│ │
└──────────┬──────────────────┘
│
┌──────▼──────┐
│ DIFF │
└──────┬──────┘
│
┌───────────▼───────────┐
│ Reconcile() │
│ Create/Update/Delete │
│ resources to close │
│ the gap │
└───────────┬───────────┘
│
└──── repeat forever ──┐
│
(watch for changes)│
Step 1: Scaffold the Operator
# Install Operator SDK
export OPERATOR_SDK_VERSION=v1.34.0
curl -LO "https://github.com/operator-framework/operator-sdk/releases/download/${OPERATOR_SDK_VERSION}/operator-sdk_linux_amd64"
chmod +x operator-sdk_linux_amd64
mv operator-sdk_linux_amd64 /usr/local/bin/operator-sdk
# Initialize the operator project
mkdir database-operator && cd database-operator
operator-sdk init \
--domain mycompany.com \
--repo github.com/mycompany/database-operator \
--plugins go/v4
# Create the API (CRD + Controller scaffold)
operator-sdk create api \
--group db \
--version v1alpha1 \
--kind Database \
--resource \
--controller
Step 2: Define the CRD (Custom Resource Definition)
// api/v1alpha1/database_types.go
package v1alpha1
import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/api/resource"
)
// DatabaseSpec defines the desired state of Database
type DatabaseSpec struct {
// PostgreSQL version (e.g., "14", "15", "16")
// +kubebuilder:validation:Enum="14";"15";"16"
// +kubebuilder:default="15"
Version string `json:"version"`
// Storage size for the database PVC
// +kubebuilder:validation:Pattern=`^[0-9]+Gi$`
Storage string `json:"storage"`
// Number of replicas (1 = standalone, 3 = HA with streaming replication)
// +kubebuilder:validation:Minimum=1
// +kubebuilder:validation:Maximum=5
// +kubebuilder:default=1
Replicas int32 `json:"replicas,omitempty"`
// Database name to create on initialization
DatabaseName string `json:"databaseName"`
// Resource requirements for the database pods
Resources DatabaseResources `json:"resources,omitempty"`
// Backup configuration
Backup *BackupConfig `json:"backup,omitempty"`
}
type DatabaseResources struct {
// +kubebuilder:default="500m"
CPURequest string `json:"cpuRequest,omitempty"`
// +kubebuilder:default="1Gi"
MemoryRequest string `json:"memoryRequest,omitempty"`
// +kubebuilder:default="2"
CPULimit string `json:"cpuLimit,omitempty"`
// +kubebuilder:default="4Gi"
MemoryLimit string `json:"memoryLimit,omitempty"`
}
type BackupConfig struct {
Enabled bool `json:"enabled"`
Schedule string `json:"schedule,omitempty"` // Cron expression
S3Bucket string `json:"s3Bucket,omitempty"`
}
// DatabaseStatus defines the observed state of Database
type DatabaseStatus struct {
// Current phase: Pending, Provisioning, Ready, Failed, Deleting
// +kubebuilder:validation:Enum=Pending;Provisioning;Ready;Failed;Deleting
Phase string `json:"phase,omitempty"`
// Human-readable message about current status
Message string `json:"message,omitempty"`
// Connection string (without password) for reference
ConnectionString string `json:"connectionString,omitempty"`
// Name of the Secret containing credentials
CredentialsSecret string `json:"credentialsSecret,omitempty"`
// Conditions follow standard Kubernetes condition conventions
Conditions []metav1.Condition `json:"conditions,omitempty"`
// ObservedGeneration tracks spec changes
ObservedGeneration int64 `json:"observedGeneration,omitempty"`
}
// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:printcolumn:name="Version",type=string,JSONPath=`.spec.version`
// +kubebuilder:printcolumn:name="Phase",type=string,JSONPath=`.status.phase`
// +kubebuilder:printcolumn:name="Ready",type=string,JSONPath=`.status.conditions[?(@.type=="Ready")].status`
// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`
type Database struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec DatabaseSpec `json:"spec,omitempty"`
Status DatabaseStatus `json:"status,omitempty"`
}
Step 3: Implement the Reconciler (Core Controller Logic)
// controllers/database_controller.go
package controllers
import (
"context"
"fmt"
"time"
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/types"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
"sigs.k8s.io/controller-runtime/pkg/log"
dbv1alpha1 "github.com/mycompany/database-operator/api/v1alpha1"
)
const (
databaseFinalizer = "db.mycompany.com/finalizer"
requeueAfter = 30 * time.Second
)
type DatabaseReconciler struct {
client.Client
Scheme *runtime.Scheme
CatalogClient ServiceCatalogClient // External service catalog
}
// +kubebuilder:rbac:groups=db.mycompany.com,resources=databases,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=db.mycompany.com,resources=databases/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=db.mycompany.com,resources=databases/finalizers,verbs=update
// +kubebuilder:rbac:groups=apps,resources=statefulsets,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups="",resources=services;secrets;persistentvolumeclaims,verbs=get;list;watch;create;update;patch;delete
func (r *DatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
logger := log.FromContext(ctx)
// Step 1: Fetch the Database resource
db := &dbv1alpha1.Database{}
if err := r.Get(ctx, req.NamespacedName, db); err != nil {
if errors.IsNotFound(err) {
// Resource deleted before we could reconcile — nothing to do
return ctrl.Result{}, nil
}
return ctrl.Result{}, fmt.Errorf("failed to get Database: %w", err)
}
// Step 2: Handle deletion with finalizer
if !db.DeletionTimestamp.IsZero() {
return r.handleDeletion(ctx, db)
}
// Step 3: Add finalizer if not present
if !controllerutil.ContainsFinalizer(db, databaseFinalizer) {
controllerutil.AddFinalizer(db, databaseFinalizer)
if err := r.Update(ctx, db); err != nil {
return ctrl.Result{}, fmt.Errorf("failed to add finalizer: %w", err)
}
return ctrl.Result{Requeue: true}, nil
}
// Step 4: Update status to Provisioning
if db.Status.Phase == "" {
if err := r.updateStatus(ctx, db, "Provisioning", "Starting database provisioning"); err != nil {
return ctrl.Result{}, err
}
}
// Step 5: Reconcile all child resources
// Each reconcileX function is idempotent — safe to call repeatedly
secret, err := r.reconcileSecret(ctx, db)
if err != nil {
r.updateStatus(ctx, db, "Failed", fmt.Sprintf("Failed to create secret: %v", err))
return ctrl.Result{RequeueAfter: requeueAfter}, err
}
if err := r.reconcileStatefulSet(ctx, db); err != nil {
r.updateStatus(ctx, db, "Failed", fmt.Sprintf("Failed to create StatefulSet: %v", err))
return ctrl.Result{RequeueAfter: requeueAfter}, err
}
if err := r.reconcileService(ctx, db); err != nil {
r.updateStatus(ctx, db, "Failed", fmt.Sprintf("Failed to create Service: %v", err))
return ctrl.Result{RequeueAfter: requeueAfter}, err
}
// Step 6: Register in service catalog
if err := r.reconcileServiceCatalog(ctx, db); err != nil {
logger.Error(err, "Failed to register in service catalog (non-fatal)")
// Don't fail reconciliation for catalog registration
}
// Step 7: Check if StatefulSet is ready
ready, err := r.isStatefulSetReady(ctx, db)
if err != nil {
return ctrl.Result{RequeueAfter: requeueAfter}, err
}
if !ready {
r.updateStatus(ctx, db, "Provisioning", "Waiting for StatefulSet pods to be ready")
return ctrl.Result{RequeueAfter: requeueAfter}, nil
}
// Step 8: All good — mark as Ready
svcName := fmt.Sprintf("%s-postgresql", db.Name)
connStr := fmt.Sprintf("postgresql://%s:5432/%s",
fmt.Sprintf("%s.%s.svc.cluster.local", svcName, db.Namespace),
db.Spec.DatabaseName)
db.Status.Phase = "Ready"
db.Status.Message = "Database is ready"
db.Status.ConnectionString = connStr
db.Status.CredentialsSecret = secret.Name
db.Status.ObservedGeneration = db.Generation
if err := r.Status().Update(ctx, db); err != nil {
return ctrl.Result{}, fmt.Errorf("failed to update status: %w", err)
}
logger.Info("Database reconciliation complete", "database", db.Name, "phase", "Ready")
return ctrl.Result{RequeueAfter: 5 * time.Minute}, nil // Periodic health check
}
// reconcileSecret creates or updates the database credentials Secret
func (r *DatabaseReconciler) reconcileSecret(ctx context.Context, db *dbv1alpha1.Database) (*corev1.Secret, error) {
secretName := fmt.Sprintf("%s-postgresql-credentials", db.Name)
secret := &corev1.Secret{}
err := r.Get(ctx, types.NamespacedName{Name: secretName, Namespace: db.Namespace}, secret)
if errors.IsNotFound(err) {
// Generate secure random password
password, err := generateSecurePassword(32)
if err != nil {
return nil, fmt.Errorf("failed to generate password: %w", err)
}
secret = &corev1.Secret{
ObjectMeta: metav1.ObjectMeta{
Name: secretName,
Namespace: db.Namespace,
Labels: labelsForDatabase(db),
},
Type: corev1.SecretTypeOpaque,
StringData: map[string]string{
"username": "appuser",
"password": password,
"postgres-password": password, // For bitnami chart compatibility
"database": db.Spec.DatabaseName,
"host": fmt.Sprintf("%s-postgresql.%s.svc.cluster.local", db.Name, db.Namespace),
"port": "5432",
"connection-string": fmt.Sprintf(
"postgresql://appuser:%s@%s-postgresql.%s.svc.cluster.local:5432/%s",
password, db.Name, db.Namespace, db.Spec.DatabaseName),
},
}
// Set owner reference — Secret is garbage collected when Database is deleted
if err := controllerutil.SetControllerReference(db, secret, r.Scheme); err != nil {
return nil, fmt.Errorf("failed to set owner reference on secret: %w", err)
}
if err := r.Create(ctx, secret); err != nil {
return nil, fmt.Errorf("failed to create secret: %w", err)
}
return secret, nil
}
// Secret already exists — return it without modification
// (never regenerate passwords on reconcile — that would break existing connections)
return secret, nil
}
// reconcileStatefulSet creates or updates the PostgreSQL StatefulSet
func (r *DatabaseReconciler) reconcileStatefulSet(ctx context.Context, db *dbv1alpha1.Database) error {
secretName := fmt.Sprintf("%s-postgresql-credentials", db.Name)
storageSize := db.Spec.Storage
desired := &appsv1.StatefulSet{
ObjectMeta: metav1.ObjectMeta{
Name: fmt.Sprintf("%s-postgresql", db.Name),
Namespace: db.Namespace,
Labels: labelsForDatabase(db),
},
Spec: appsv1.StatefulSetSpec{
Replicas: &db.Spec.Replicas,
ServiceName: fmt.Sprintf("%s-postgresql-headless", db.Name),
Selector: &metav1.LabelSelector{
MatchLabels: labelsForDatabase(db),
},
Template: corev1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{
Labels: labelsForDatabase(db),
},
Spec: corev1.PodSpec{
SecurityContext: &corev1.PodSecurityContext{
RunAsUser: int64Ptr(999), // postgres user
RunAsGroup: int64Ptr(999),
FSGroup: int64Ptr(999),
},
Containers: []corev1.Container{
{
Name: "postgresql",
Image: fmt.Sprintf("postgres:%s-alpine", db.Spec.Version),
Ports: []corev1.ContainerPort{
{ContainerPort: 5432, Name: "postgresql"},
},
Env: []corev1.EnvVar{
{
Name: "POSTGRES_PASSWORD",
ValueFrom: &corev1.EnvVarSource{
SecretKeyRef: &corev1.SecretKeySelector{
LocalObjectReference: corev1.LocalObjectReference{
Name: secretName,
},
Key: "postgres-password",
},
},
},
{
Name: "POSTGRES_DB",
Value: db.Spec.DatabaseName,
},
{
Name: "POSTGRES_USER",
Value: "appuser",
},
{
Name: "PGDATA",
Value: "/var/lib/postgresql/data/pgdata",
},
},
Resources: corev1.ResourceRequirements{
Requests: corev1.ResourceList{
corev1.ResourceCPU: resource.MustParse(db.Spec.Resources.CPURequest),
corev1.ResourceMemory: resource.MustParse(db.Spec.Resources.MemoryRequest),
},
Limits: corev1.ResourceList{
corev1.ResourceCPU: resource.MustParse(db.Spec.Resources.CPULimit),
corev1.ResourceMemory: resource.MustParse(db.Spec.Resources.MemoryLimit),
},
},
VolumeMounts: []corev1.VolumeMount{
{
Name: "data",
MountPath: "/var/lib/postgresql/data",
},
},
ReadinessProbe: &corev1.Probe{
ProbeHandler: corev1.ProbeHandler{
Exec: &corev1.ExecAction{
Command: []string{
"pg_isready",
"-U", "appuser",
"-d", db.Spec.DatabaseName,
},
},
},
InitialDelaySeconds: 10,
PeriodSeconds: 5,
FailureThreshold: 6,
},
LivenessProbe: &corev1.Probe{
ProbeHandler: corev1.ProbeHandler{
Exec: &corev1.ExecAction{
Command: []string{
"pg_isready",
"-U", "appuser",
},
},
},
InitialDelaySeconds: 30,
PeriodSeconds: 10,
FailureThreshold: 3,
},
},
},
},
},
VolumeClaimTemplates: []corev1.PersistentVolumeClaim{
{
ObjectMeta: metav1.ObjectMeta{
Name: "data",
},
Spec: corev1.PersistentVolumeClaimSpec{
AccessModes: []corev1.PersistentVolumeAccessMode{
corev1.ReadWriteOnce,
},
Resources: corev1.ResourceRequirements{
Requests: corev1.ResourceList{
corev1.ResourceStorage: resource.MustParse(storageSize),
},
},
StorageClassName: stringPtr("fast-ssd"),
},
},
},
},
}
// Set owner reference for garbage collection
if err := controllerutil.SetControllerReference(db, desired, r.Scheme); err != nil {
return fmt.Errorf("failed to set owner reference: %w", err)
}
// Create or update using server-side apply pattern
existing := &appsv1.StatefulSet{}
err := r.Get(ctx, types.NamespacedName{
Name: desired.Name,
Namespace: desired.Namespace,
}, existing)
if errors.IsNotFound(err) {
return r.Create(ctx, desired)
} else if err != nil {
return fmt.Errorf("failed to get StatefulSet: %w", err)
}
// Update existing StatefulSet (only safe fields)
existing.Spec.Replicas = desired.Spec.Replicas
existing.Spec.Template.Spec.Containers[0].Image = desired.Spec.Template.Spec.Containers[0].Image
existing.Spec.Template.Spec.Containers[0].Resources = desired.Spec.Template.Spec.Containers[0].Resources
return r.Update(ctx, existing)
}
// handleDeletion runs cleanup before the Database resource is deleted
func (r *DatabaseReconciler) handleDeletion(ctx context.Context, db *dbv1alpha1.Database) (ctrl.Result, error) {
logger := log.FromContext(ctx)
if controllerutil.ContainsFinalizer(db, databaseFinalizer) {
// Update status
r.updateStatus(ctx, db, "Deleting", "Running cleanup before deletion")
// Deregister from service catalog
if err := r.CatalogClient.Deregister(ctx, db.Namespace, db.Name); err != nil {
logger.Error(err, "Failed to deregister from service catalog")
// Continue with deletion even if catalog deregistration fails
}
// Note: StatefulSet, Service, and Secret are owned resources
// They will be garbage collected automatically via owner references
// We only need to handle external resources (like service catalog) here
// Remove finalizer to allow Kubernetes to delete the resource
controllerutil.RemoveFinalizer(db, databaseFinalizer)
if err := r.Update(ctx, db); err != nil {
return ctrl.Result{}, fmt.Errorf("failed to remove finalizer: %w", err)
}
logger.Info("Database deletion complete", "database", db.Name)
}
return ctrl.Result{}, nil
}
// isStatefulSetReady checks if all replicas are ready
func (r *DatabaseReconciler) isStatefulSetReady(ctx context.Context, db *dbv1alpha1.Database) (bool, error) {
sts := &appsv1.StatefulSet{}
err := r.Get(ctx, types.NamespacedName{
Name: fmt.Sprintf("%s-postgresql", db.Name),
Namespace: db.Namespace,
}, sts)
if err != nil {
return false, err
}
return sts.Status.ReadyReplicas == *sts.Spec.Replicas, nil
}
func (r *DatabaseReconciler) updateStatus(ctx context.Context, db *dbv1alpha1.Database, phase, message string) error {
db.Status.Phase = phase
db.Status.Message = message
return r.Status().Update(ctx, db)
}
// SetupWithManager registers the controller and sets up watches
func (r *DatabaseReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&dbv1alpha1.Database{}). // Primary resource to watch
Owns(&appsv1.StatefulSet{}). // Watch owned StatefulSets
Owns(&corev1.Service{}). // Watch owned Services
Owns(&corev1.Secret{}). // Watch owned Secrets
WithOptions(controller.Options{
MaxConcurrentReconciles: 3, // Reconcile up to 3 databases in parallel
}).
Complete(r)
}
// Helper functions
func labelsForDatabase(db *dbv1alpha1.Database) map[string]string {
return map[string]string{
"app.kubernetes.io/name": "postgresql",
"app.kubernetes.io/instance": db.Name,
"app.kubernetes.io/managed-by": "database-operator",
"db.mycompany.com/database": db.Name,
}
}
func int64Ptr(i int64) *int64 { return &i }
func stringPtr(s string) *string { return &s }
Step 4: The Custom Resource (How Developers Use It)
# Developer creates this — operator handles everything else
apiVersion: db.mycompany.com/v1alpha1
kind: Database
metadata:
name: myapp-db
namespace: production
spec:
version: "15"
storage: "50Gi"
replicas: 1
databaseName: myapp
resources:
cpuRequest: "500m"
memoryRequest: "1Gi"
cpuLimit: "2"
memoryLimit: "4Gi"
backup:
enabled: true
schedule: "0 2 * * *" # Daily at 2 AM
s3Bucket: "mycompany-db-backups"
# Apply the Database resource
kubectl apply -f database.yaml
# Watch the operator provision everything
kubectl get database myapp-db -n production -w
# NAME VERSION PHASE READY AGE
# myapp-db 15 Provisioning False 10s
# myapp-db 15 Provisioning False 25s
# myapp-db 15 Ready True 45s
# Check what the operator created
kubectl get statefulset,service,secret -n production -l db.mycompany.com/database=myapp-db
# NAME READY AGE
# statefulset.apps/myapp-db-postgresql 1/1 45s
#
# NAME TYPE CLUSTER-IP PORT(S) AGE
# service/myapp-db-postgresql ClusterIP 10.96.1.45 5432/TCP 45s
# service/myapp-db-postgresql-headless ClusterIP None 5432/TCP 45s
#
# NAME TYPE DATA AGE
# secret/myapp-db-postgresql-credentials Opaque 6 45s
# Describe the database for connection info
kubectl describe database myapp-db -n production
# Status:
# Phase: Ready
# Connection String: postgresql://myapp-db-postgresql.production.svc.cluster.local:5432/myapp
# Credentials Secret: myapp-db-postgresql-credentials
# Delete everything by deleting just the Database resource
kubectl delete database myapp-db -n production
# Operator runs finalizer cleanup, deregisters from catalog
# Owner references cascade-delete the StatefulSet, Service, and Secret
Step 5: Testing the Operator
// controllers/database_controller_test.go
package controllers
import (
"context"
"time"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"sigs.k8s.io/controller-runtime/pkg/client"
dbv1alpha1 "github.com/mycompany/database-operator/api/v1alpha1"
)
var _ = Describe("Database Controller", func() {
const timeout = time.Second * 30
const interval = time.Second * 1
Context("When creating a Database resource", func() {
It("Should create a StatefulSet, Service, and Secret", func() {
ctx := context.Background()
db := &dbv1alpha1.Database{
ObjectMeta: metav1.ObjectMeta{
Name: "test-db",
Namespace: "default",
},
Spec: dbv1alpha1.DatabaseSpec{
Version: "15",
Storage: "10Gi",
Replicas: 1,
DatabaseName: "testdb",
Resources: dbv1alpha1.DatabaseResources{
CPURequest: "100m",
MemoryRequest: "256Mi",
CPULimit: "500m",
MemoryLimit: "512Mi",
},
},
}
Expect(k8sClient.Create(ctx, db)).Should(Succeed())
// Verify StatefulSet is created
sts := &appsv1.StatefulSet{}
Eventually(func() error {
return k8sClient.Get(ctx, client.ObjectKey{
Name: "test-db-postgresql",
Namespace: "default",
}, sts)
}, timeout, interval).Should(Succeed())
// Verify Secret is created with all required keys
secret := &corev1.Secret{}
Eventually(func() error {
return k8sClient.Get(ctx, client.ObjectKey{
Name: "test-db-postgresql-credentials",
Namespace: "default",
}, secret)
}, timeout, interval).Should(Succeed())
Expect(secret.Data).To(HaveKey("username"))
Expect(secret.Data).To(HaveKey("password"))
Expect(secret.Data).To(HaveKey("connection-string"))
// Verify owner references are set (for garbage collection)
Expect(sts.OwnerReferences).To(HaveLen(1))
Expect(sts.OwnerReferences[0].Name).To(Equal("test-db"))
Expect(secret.OwnerReferences[0].Name).To(Equal("test-db"))
})
})
})
# Run tests using envtest (spins up a real API server)
make test
# Build and deploy operator
make docker-build docker-push IMG=registry.mycompany.com/database-operator:v0.1.0
make deploy IMG=registry.mycompany.com/database-operator:v0.1.0
# Verify operator is running
kubectl get pods -n database-operator-system
| Approach | Flexibility | Complexity | Maintenance | Best For |
|---|---|---|---|---|
| Custom Operator (Go) | Maximum | High | High | Complex stateful workflows |
| Helm chart | Medium | Low | Low | Simple, static deployments |
| Crossplane | High | Medium | Medium | Infrastructure provisioning |
| Kro (Resource Orchestrator) | Medium | Low | Low | Composing existingresources |
| KEDA + Jobs | Medium | Low | Low | Event-driven, stateless tasks |
| Ansible Operator | Medium | Medium | Medium | Teams with Ansible expertise |
errors.IsNotFound() checks, not assume fresh state.status, resourceVersion). Blindly overwriting causes infinite reconcile loops.Operators encode operational knowledge as code. The reconciliation loop is the fundamental primitive: continuously compare desired state (the CRD spec) against actual state (what exists in the cluster and external systems), and take the minimum actions needed to close the gap. Owner references handle in-cluster garbage collection automatically, while finalizers handle external cleanup. The result is infrastructure that manages itself — a database that provisions, monitors, and cleans itself up purely through Kubernetes resource declarations.
controllerutil.SetControllerReference does?Owns() in SetupWithManager affects the watch list?Explain the architecture, failure modes, and operational best practices for Debugging Pending Pods: Systematic Troubleshooting for Scheduling Blocks in production Kubernetes environments.
How do Kubernetes NetworkPolicies enforce micro-segmentation and pod traffic isolation, and how do CNI plugins like Calico or Cilium implement these rules at the kernel network layer?
Explain the architecture, failure modes, and operational best practices for CrashLoopBackOff Troubleshooting: Root Cause Analysis for Application Container Crashes in production Kubernetes environments.
Explain the architecture, failure modes, and operational best practices for Docker vs. Kubernetes: Runtime Engines vs. Cluster Orchestrators in production Kubernetes environments.
Your 500-node production cluster is experiencing API server latency spikes. Investigation shows etcd is the bottleneck: `etcd_disk_wal_fsync_duration_seconds` P99 is 150ms (should be <10ms), leader elections are happening frequently, and the etcd database size is approaching 8GB. Walk through your complete etcd performance diagnosis and remediation plan.
How do you design a comprehensive disaster recovery strategy for Kubernetes using etcd snapshots for control plane state and Velero for persistent volume backups?
Explain the architecture, failure modes, and operational best practices for Control Plane vs. Worker Nodes: Key Architecture Components of a Kubernetes Cluster in production Kubernetes environments.
Your company operates in US-East, EU-West, and APAC. Compliance requires EU data to stay in EU (GDPR), US financial data to stay in US (SOX), and APAC to serve as a DR site for both regions. SLA is 99.95% (< 4.4 hours/year downtime). Design a complete multi-cluster strategy.
What kernel mechanisms trigger an OOMKilled exit status for a container, and how do you configure container memory requests and cgroups limits to prevent out-of-memory container crashes?
What step-by-step workflow guarantees zero downtime when upgrading worker nodes and control plane components using `kubeadm`, `cordon`, and `drain`?
Explain the architecture, failure modes, and operational best practices for etcd Architecture: Distributed Key-Value Store for Kubernetes Cluster State in production Kubernetes environments.
How do you construct and deploy a custom Kubernetes Admission Webhook from scratch, and what TLS certificate requirements must be satisfied for the API server to communicate with it securely?
How does an Ingress Controller manage Layer 7 HTTP/HTTPS traffic routing and SSL/TLS termination, and how does it differ from a NodePort or LoadBalancer Service?
Your company runs a production Kubernetes cluster on v1.26. The security team has mandated an upgrade to v1.29 due to a CVE. You have 40 nodes, stateful workloads, and a 99.9% uptime SLA. Walk through your complete upgrade strategy.
Your organization must comply with SOC 2 and PCI-DSS. The security team requires that no container image with a Critical or High CVE reaches production, all Kubernetes manifests must comply with the CIS Kubernetes Benchmark, and secrets must never appear in source code or image layers. Design and implement a complete DevSecOps pipeline.
Explain the architecture, failure modes, and operational best practices for CoreDNS Architecture: In-Cluster DNS Resolution and Service Record Lookup in production Kubernetes environments.
Why does Kubernetes use a Pod—rather than a single container—as its smallest deployable unit, and how do containers inside the same Pod share network and storage namespaces?
What is OPA/Gatekeeper, how does it integrate with Kubernetes Validating Admission Webhooks, and how do ConstraintTemplates define declarative cluster governance policies?
Explain the architecture, failure modes, and operational best practices for Control Plane vs. Worker Nodes: Responsibilities and Component Breakdown in production Kubernetes environments.
How do PersistentVolumes, PersistentVolumeClaims, and StorageClasses decouple storage consumption from underlying cloud storage infrastructure in Kubernetes?
How do you implement an admission policy using OPA/Gatekeeper or Kyverno to enforce that all containers deployed to a cluster originate exclusively from approved internal image registries?
You manage a critical payment processing API. It handles 10,000 requests/minute with no tolerance for dropped connections. How do you implement zero-downtime deployments?
Your e-commerce platform releases features weekly. A bad release last quarter caused 30 minutes of downtime and lost $200K in revenue. Leadership now requires that all releases be validated against 5% of live traffic before full rollout, with automatic rollback if error rates exceed 1%. How do you implement this?
What happens under the hood during a `kubectl apply` request from client-side YAML parsing, OpenAPI validation, authentication/authorization, admission webhooks, to etcd persistence?
What is the operational difference between CPU/memory resource requests and limits, and how does Kubernetes use requests for scheduling while using limits to enforce throttling and OOM eviction?
What are the primary performance tuning levers for etcd (such as heartbeat intervals, election timeouts, disk I/O priorities, and DB quota limits) to ensure cluster stability under heavy write loads?
Your cluster is running out of resources. Pods are stuck in `Pending` state, nodes are at 90% CPU, and the on-call engineer is getting paged at 2 AM. What monitoring and scaling strategies do you implement to prevent this and respond automatically?
How Do You Benchmark and Monitor etcd Health in Production?: Architecture & Implementation?
How do Kubernetes Namespaces provide logical cluster isolation and resource scoping, and what are the limitations of namespaces regarding security and network boundary isolation?
How does Kubernetes assign Guaranteed, Burstable, and BestEffort Quality of Service (QoS) classes to Pods, and how does the kernel OOM killer use `oom_score_adj` to evict Pods during resource scarcity?
Explain the architecture, failure modes, and operational best practices for Essential kubectl Commands: Operations and Debugging Cheat Sheet in production Kubernetes environments.
A developer accidentally committed a database password and an AWS access key directly into a ConfigMap, which was then applied to the production cluster and pushed to a public GitHub repository. You're the on-call engineer. Walk through your complete incident response plan.
How do Liveness, Readiness, and Startup probes differ in purpose and behavior, and how do incorrect probe configurations cause cascading deployment failures or unnecessary restarts?
Your company has three engineering teams: Frontend (deploys React apps), Backend (deploys APIs and workers), and Database (manages PostgreSQL and Redis). Each team has developers, leads, and CI/CD service accounts. Design a complete RBAC system where teams can only access their own namespaces, leads can approve deployments, CI/CD can deploy but not delete, and platform engineers have cluster-wide admin access with audit trails.
What are the fundamental differences between imperative `kubectl` commands and declarative `kubectl apply` manifest management, and why is declarative management required for Production Infrastructure as Code?
Why are standard Deployments unsuitable for stateful applications like database clusters, and how do StatefulSets provide stable network identities, ordinal indices, and persistent storage bindings?
Explain the architecture, failure modes, and operational best practices for DaemonSets: Running Node-Level Daemon Agents for Logging and Monitoring in production Kubernetes environments.
Explain the architecture, failure modes, and operational best practices for Dry-Run Validation: Client-Side vs. Server-Side Mutation Testing in production Kubernetes environments.
You need to ensure a PostgreSQL database Pod maintains state across cluster updates, node failures, and rolling restarts. Design a production-grade StatefulSet with proper storage, backup, high availability, and operational runbook.
Explain the architecture, failure modes, and operational best practices for Anatomy of a Kubernetes Manifest: The Four Essential Structure Fields in production Kubernetes environments.
Your payment service is experiencing cascading failures. When the downstream fraud-detection service becomes slow, payment service threads pile up waiting for responses, eventually exhausting the connection pool and causing payment service itself to fail. Implement circuit breaking, retry logic, timeout policies, and traffic mirroring using Istio to make this system resilient without changing application code.
How do Role, ClusterRole, RoleBinding, and ClusterRoleBinding resources interact to enforce least-privilege authorization across namespaces and cluster-wide resources?
You have a cluster with three node types: GPU nodes (expensive), high-memory nodes, and standard nodes. ML training jobs must run only on GPU nodes, memory-intensive analytics must prefer high-memory nodes, and web frontends must be spread across standard nodes in different AZs. Also, the GPU nodes should not accept any non-ML workloads. Implement this scheduling strategy.
What is the structural and functional difference between Labels and Annotations in Kubernetes, and why should operational metadata never be used in label selectors?
What are the key architectural differences between ServiceAccounts and User Accounts in Kubernetes, and how does token projection work for Pod service account authentication?
What are the exact phase transitions a Pod undergoes from Pending to Running or Failed, and what conditions cause a Pod to remain stuck in Pending or CrashLoopBackOff?
How does the Kubernetes API extension model work using Custom Resource Definitions (CRDs) and custom controllers, and how does API aggregation differ from CRD registration?
How do Mutating and Validating Admission Webhooks differ in execution order and purpose during an API server request, and how do you prevent webhooks from blocking cluster operations if they fail?
How do Init Containers execute sequentially before main application containers start, and how can they be used for dependency waiting, schema migrations, and secure credential setup?
Explain the architecture, failure modes, and operational best practices for Deployments vs. ReplicaSets: Declarative Scaling and Pod Template Management in production Kubernetes environments.
How does etcd store Kubernetes cluster state internally using key-value MVCC (Multi-Version Concurrency Control), and how does the API server handle bbolt database compaction and revision watch streams?
What key fields inside a Deployment specification control replica count, update strategy, and pod revision history, and how does a Deployment controller manage underlying ReplicaSets?
How do `maxSurge` and `maxUnavailable` parameters control rolling update rollouts, and how do you configure them to guarantee zero-downtime deployments under heavy traffic?
How do Kubernetes Services provide stable IP addresses and DNS endpoints for transient Pods, and how does kube-proxy update iptables/IPVS rules when Pod endpoints change?
What are the functional differences and use cases among ClusterIP, NodePort, LoadBalancer, and ExternalName service types, and how does packet routing work for each?
Explain the architecture, failure modes, and operational best practices for ConfigMaps: Decoupling Application Configuration from Container Images in production Kubernetes environments.
What are the technical differences between Secrets and ConfigMaps, why is Base64 encoding insufficient for secret security, and how should sensitive data be encrypted at rest in etcd?
How does the Kubelet enforce self-healing mechanics through container restart policies and health probes, and how does the control plane reschedule Pods when a node dies?