Module #9 · expert Article
Custom Controllers and Operators Development: Architecture & Implementation
In-depth architectural breakdown and operational implementation for Custom Controllers and Operators Development: Architecture & Implementation.
52 min read+25 XPPublished 2026-07-16
Prerequisite knowledge: Kubernetes API concepts, Go programming basics, reconciliation loops, CRDs
Question (Scenario-Based)
Your platform team needs to automate database provisioning. Every time a developer creates a
Databasecustom resource, the system should automatically provision a PostgreSQL StatefulSet, a Service, a Secret with credentials, and register the database in your internal service catalog — then clean everything up when the resource is deleted. Build this operator.
Quick Answer (30 sec revision)
Use the Operator SDK or Kubebuilder to scaffold a Go-based operator. Define a Database CRD, implement a reconciliation loop that creates the StatefulSet, Service, and Secret as owned resources, use finalizers for cleanup, and set owner references so child resources are garbage-collected automatically.
Detailed Answer
️ Operator Architecture
Developer applies: Operator watches: Operator creates:
┌─────────────────┐ ┌─────────────────┐ ┌──────────────────┐
│ Database CRD │──────► │ Reconciler │─────►│ StatefulSet │
│ name: myapp-db │ │ (control loop) │ │ Service │
│ version: 14 │ │ │ │ Secret │
│ storage: 50Gi │ │ desired state │ │ PVC │
└─────────────────┘ │ == actual state│ └──────────────────┘
└─────────────────┘
│
┌───────▼─────────┐
│ Service Catalog │
│ Registration API │
└─────────────────┘
The Kubernetes Reconciliation Loop (Core Concept)
┌─────────────────────────────┐
│ │
┌─────▼──────┐ ┌───────▼──────┐
│ Desired │ │ Actual │
│ State │ │ State │
│ (CRD spec) │ │ (cluster) │
└─────┬──────┘ └───────┬──────┘
│ │
└──────────┬──────────────────┘
│
┌──────▼──────┐
│ DIFF │
└──────┬──────┘
│
┌───────────▼───────────┐
│ Reconcile() │
│ Create/Update/Delete │
│ resources to close │
│ the gap │
└───────────┬───────────┘
│
└──── repeat forever ──┐
│
(watch for changes)│
️ Step-by-Step Operator Implementation
Step 1: Scaffold the Operator
# Install Operator SDK
export OPERATOR_SDK_VERSION=v1.34.0
curl -LO "https://github.com/operator-framework/operator-sdk/releases/download/${OPERATOR_SDK_VERSION}/operator-sdk_linux_amd64"
chmod +x operator-sdk_linux_amd64
mv operator-sdk_linux_amd64 /usr/local/bin/operator-sdk
# Initialize the operator project
mkdir database-operator && cd database-operator
operator-sdk init \
--domain mycompany.com \
--repo github.com/mycompany/database-operator \
--plugins go/v4
# Create the API (CRD + Controller scaffold)
operator-sdk create api \
--group db \
--version v1alpha1 \
--kind Database \
--resource \
--controller
Step 2: Define the CRD (Custom Resource Definition)
// api/v1alpha1/database_types.go
package v1alpha1
import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/api/resource"
)
// DatabaseSpec defines the desired state of Database
type DatabaseSpec struct {
// PostgreSQL version (e.g., "14", "15", "16")
// +kubebuilder:validation:Enum="14";"15";"16"
// +kubebuilder:default="15"
Version string `json:"version"`
// Storage size for the database PVC
// +kubebuilder:validation:Pattern=`^[0-9]+Gi$`
Storage string `json:"storage"`
// Number of replicas (1 = standalone, 3 = HA with streaming replication)
// +kubebuilder:validation:Minimum=1
// +kubebuilder:validation:Maximum=5
// +kubebuilder:default=1
Replicas int32 `json:"replicas,omitempty"`
// Database name to create on initialization
DatabaseName string `json:"databaseName"`
// Resource requirements for the database pods
Resources DatabaseResources `json:"resources,omitempty"`
// Backup configuration
Backup *BackupConfig `json:"backup,omitempty"`
}
type DatabaseResources struct {
// +kubebuilder:default="500m"
CPURequest string `json:"cpuRequest,omitempty"`
// +kubebuilder:default="1Gi"
MemoryRequest string `json:"memoryRequest,omitempty"`
// +kubebuilder:default="2"
CPULimit string `json:"cpuLimit,omitempty"`
// +kubebuilder:default="4Gi"
MemoryLimit string `json:"memoryLimit,omitempty"`
}
type BackupConfig struct {
Enabled bool `json:"enabled"`
Schedule string `json:"schedule,omitempty"` // Cron expression
S3Bucket string `json:"s3Bucket,omitempty"`
}
// DatabaseStatus defines the observed state of Database
type DatabaseStatus struct {
// Current phase: Pending, Provisioning, Ready, Failed, Deleting
// +kubebuilder:validation:Enum=Pending;Provisioning;Ready;Failed;Deleting
Phase string `json:"phase,omitempty"`
// Human-readable message about current status
Message string `json:"message,omitempty"`
// Connection string (without password) for reference
ConnectionString string `json:"connectionString,omitempty"`
// Name of the Secret containing credentials
CredentialsSecret string `json:"credentialsSecret,omitempty"`
// Conditions follow standard Kubernetes condition conventions
Conditions []metav1.Condition `json:"conditions,omitempty"`
// ObservedGeneration tracks spec changes
ObservedGeneration int64 `json:"observedGeneration,omitempty"`
}
// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:printcolumn:name="Version",type=string,JSONPath=`.spec.version`
// +kubebuilder:printcolumn:name="Phase",type=string,JSONPath=`.status.phase`
// +kubebuilder:printcolumn:name="Ready",type=string,JSONPath=`.status.conditions[?(@.type=="Ready")].status`
// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`
type Database struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec DatabaseSpec `json:"spec,omitempty"`
Status DatabaseStatus `json:"status,omitempty"`
}
Step 3: Implement the Reconciler (Core Controller Logic)
// controllers/database_controller.go
package controllers
import (
"context"
"fmt"
"time"
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/types"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
"sigs.k8s.io/controller-runtime/pkg/log"
dbv1alpha1 "github.com/mycompany/database-operator/api/v1alpha1"
)
const (
databaseFinalizer = "db.mycompany.com/finalizer"
requeueAfter = 30 * time.Second
)
type DatabaseReconciler struct {
client.Client
Scheme *runtime.Scheme
CatalogClient ServiceCatalogClient // External service catalog
}
// +kubebuilder:rbac:groups=db.mycompany.com,resources=databases,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=db.mycompany.com,resources=databases/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=db.mycompany.com,resources=databases/finalizers,verbs=update
// +kubebuilder:rbac:groups=apps,resources=statefulsets,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups="",resources=services;secrets;persistentvolumeclaims,verbs=get;list;watch;create;update;patch;delete
func (r *DatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
logger := log.FromContext(ctx)
// Step 1: Fetch the Database resource
db := &dbv1alpha1.Database{}
if err := r.Get(ctx, req.NamespacedName, db); err != nil {
if errors.IsNotFound(err) {
// Resource deleted before we could reconcile — nothing to do
return ctrl.Result{}, nil
}
return ctrl.Result{}, fmt.Errorf("failed to get Database: %w", err)
}
// Step 2: Handle deletion with finalizer
if !db.DeletionTimestamp.IsZero() {
return r.handleDeletion(ctx, db)
}
// Step 3: Add finalizer if not present
if !controllerutil.ContainsFinalizer(db, databaseFinalizer) {
controllerutil.AddFinalizer(db, databaseFinalizer)
if err := r.Update(ctx, db); err != nil {
return ctrl.Result{}, fmt.Errorf("failed to add finalizer: %w", err)
}
return ctrl.Result{Requeue: true}, nil
}
// Step 4: Update status to Provisioning
if db.Status.Phase == "" {
if err := r.updateStatus(ctx, db, "Provisioning", "Starting database provisioning"); err != nil {
return ctrl.Result{}, err
}
}
// Step 5: Reconcile all child resources
// Each reconcileX function is idempotent — safe to call repeatedly
secret, err := r.reconcileSecret(ctx, db)
if err != nil {
r.updateStatus(ctx, db, "Failed", fmt.Sprintf("Failed to create secret: %v", err))
return ctrl.Result{RequeueAfter: requeueAfter}, err
}
if err := r.reconcileStatefulSet(ctx, db); err != nil {
r.updateStatus(ctx, db, "Failed", fmt.Sprintf("Failed to create StatefulSet: %v", err))
return ctrl.Result{RequeueAfter: requeueAfter}, err
}
if err := r.reconcileService(ctx, db); err != nil {
r.updateStatus(ctx, db, "Failed", fmt.Sprintf("Failed to create Service: %v", err))
return ctrl.Result{RequeueAfter: requeueAfter}, err
}
// Step 6: Register in service catalog
if err := r.reconcileServiceCatalog(ctx, db); err != nil {
logger.Error(err, "Failed to register in service catalog (non-fatal)")
// Don't fail reconciliation for catalog registration
}
// Step 7: Check if StatefulSet is ready
ready, err := r.isStatefulSetReady(ctx, db)
if err != nil {
return ctrl.Result{RequeueAfter: requeueAfter}, err
}
if !ready {
r.updateStatus(ctx, db, "Provisioning", "Waiting for StatefulSet pods to be ready")
return ctrl.Result{RequeueAfter: requeueAfter}, nil
}
// Step 8: All good — mark as Ready
svcName := fmt.Sprintf("%s-postgresql", db.Name)
connStr := fmt.Sprintf("postgresql://%s:5432/%s",
fmt.Sprintf("%s.%s.svc.cluster.local", svcName, db.Namespace),
db.Spec.DatabaseName)
db.Status.Phase = "Ready"
db.Status.Message = "Database is ready"
db.Status.ConnectionString = connStr
db.Status.CredentialsSecret = secret.Name
db.Status.ObservedGeneration = db.Generation
if err := r.Status().Update(ctx, db); err != nil {
return ctrl.Result{}, fmt.Errorf("failed to update status: %w", err)
}
logger.Info("Database reconciliation complete", "database", db.Name, "phase", "Ready")
return ctrl.Result{RequeueAfter: 5 * time.Minute}, nil // Periodic health check
}
// reconcileSecret creates or updates the database credentials Secret
func (r *DatabaseReconciler) reconcileSecret(ctx context.Context, db *dbv1alpha1.Database) (*corev1.Secret, error) {
secretName := fmt.Sprintf("%s-postgresql-credentials", db.Name)
secret := &corev1.Secret{}
err := r.Get(ctx, types.NamespacedName{Name: secretName, Namespace: db.Namespace}, secret)
if errors.IsNotFound(err) {
// Generate secure random password
password, err := generateSecurePassword(32)
if err != nil {
return nil, fmt.Errorf("failed to generate password: %w", err)
}
secret = &corev1.Secret{
ObjectMeta: metav1.ObjectMeta{
Name: secretName,
Namespace: db.Namespace,
Labels: labelsForDatabase(db),
},
Type: corev1.SecretTypeOpaque,
StringData: map[string]string{
"username": "appuser",
"password": password,
"postgres-password": password, // For bitnami chart compatibility
"database": db.Spec.DatabaseName,
"host": fmt.Sprintf("%s-postgresql.%s.svc.cluster.local", db.Name, db.Namespace),
"port": "5432",
"connection-string": fmt.Sprintf(
"postgresql://appuser:%s@%s-postgresql.%s.svc.cluster.local:5432/%s",
password, db.Name, db.Namespace, db.Spec.DatabaseName),
},
}
// Set owner reference — Secret is garbage collected when Database is deleted
if err := controllerutil.SetControllerReference(db, secret, r.Scheme); err != nil {
return nil, fmt.Errorf("failed to set owner reference on secret: %w", err)
}
if err := r.Create(ctx, secret); err != nil {
return nil, fmt.Errorf("failed to create secret: %w", err)
}
return secret, nil
}
// Secret already exists — return it without modification
// (never regenerate passwords on reconcile — that would break existing connections)
return secret, nil
}
// reconcileStatefulSet creates or updates the PostgreSQL StatefulSet
func (r *DatabaseReconciler) reconcileStatefulSet(ctx context.Context, db *dbv1alpha1.Database) error {
secretName := fmt.Sprintf("%s-postgresql-credentials", db.Name)
storageSize := db.Spec.Storage
desired := &appsv1.StatefulSet{
ObjectMeta: metav1.ObjectMeta{
Name: fmt.Sprintf("%s-postgresql", db.Name),
Namespace: db.Namespace,
Labels: labelsForDatabase(db),
},
Spec: appsv1.StatefulSetSpec{
Replicas: &db.Spec.Replicas,
ServiceName: fmt.Sprintf("%s-postgresql-headless", db.Name),
Selector: &metav1.LabelSelector{
MatchLabels: labelsForDatabase(db),
},
Template: corev1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{
Labels: labelsForDatabase(db),
},
Spec: corev1.PodSpec{
SecurityContext: &corev1.PodSecurityContext{
RunAsUser: int64Ptr(999), // postgres user
RunAsGroup: int64Ptr(999),
FSGroup: int64Ptr(999),
},
Containers: []corev1.Container{
{
Name: "postgresql",
Image: fmt.Sprintf("postgres:%s-alpine", db.Spec.Version),
Ports: []corev1.ContainerPort{
{ContainerPort: 5432, Name: "postgresql"},
},
Env: []corev1.EnvVar{
{
Name: "POSTGRES_PASSWORD",
ValueFrom: &corev1.EnvVarSource{
SecretKeyRef: &corev1.SecretKeySelector{
LocalObjectReference: corev1.LocalObjectReference{
Name: secretName,
},
Key: "postgres-password",
},
},
},
{
Name: "POSTGRES_DB",
Value: db.Spec.DatabaseName,
},
{
Name: "POSTGRES_USER",
Value: "appuser",
},
{
Name: "PGDATA",
Value: "/var/lib/postgresql/data/pgdata",
},
},
Resources: corev1.ResourceRequirements{
Requests: corev1.ResourceList{
corev1.ResourceCPU: resource.MustParse(db.Spec.Resources.CPURequest),
corev1.ResourceMemory: resource.MustParse(db.Spec.Resources.MemoryRequest),
},
Limits: corev1.ResourceList{
corev1.ResourceCPU: resource.MustParse(db.Spec.Resources.CPULimit),
corev1.ResourceMemory: resource.MustParse(db.Spec.Resources.MemoryLimit),
},
},
VolumeMounts: []corev1.VolumeMount{
{
Name: "data",
MountPath: "/var/lib/postgresql/data",
},
},
ReadinessProbe: &corev1.Probe{
ProbeHandler: corev1.ProbeHandler{
Exec: &corev1.ExecAction{
Command: []string{
"pg_isready",
"-U", "appuser",
"-d", db.Spec.DatabaseName,
},
},
},
InitialDelaySeconds: 10,
PeriodSeconds: 5,
FailureThreshold: 6,
},
LivenessProbe: &corev1.Probe{
ProbeHandler: corev1.ProbeHandler{
Exec: &corev1.ExecAction{
Command: []string{
"pg_isready",
"-U", "appuser",
},
},
},
InitialDelaySeconds: 30,
PeriodSeconds: 10,
FailureThreshold: 3,
},
},
},
},
},
VolumeClaimTemplates: []corev1.PersistentVolumeClaim{
{
ObjectMeta: metav1.ObjectMeta{
Name: "data",
},
Spec: corev1.PersistentVolumeClaimSpec{
AccessModes: []corev1.PersistentVolumeAccessMode{
corev1.ReadWriteOnce,
},
Resources: corev1.ResourceRequirements{
Requests: corev1.ResourceList{
corev1.ResourceStorage: resource.MustParse(storageSize),
},
},
StorageClassName: stringPtr("fast-ssd"),
},
},
},
},
}
// Set owner reference for garbage collection
if err := controllerutil.SetControllerReference(db, desired, r.Scheme); err != nil {
return fmt.Errorf("failed to set owner reference: %w", err)
}
// Create or update using server-side apply pattern
existing := &appsv1.StatefulSet{}
err := r.Get(ctx, types.NamespacedName{
Name: desired.Name,
Namespace: desired.Namespace,
}, existing)
if errors.IsNotFound(err) {
return r.Create(ctx, desired)
} else if err != nil {
return fmt.Errorf("failed to get StatefulSet: %w", err)
}
// Update existing StatefulSet (only safe fields)
existing.Spec.Replicas = desired.Spec.Replicas
existing.Spec.Template.Spec.Containers[0].Image = desired.Spec.Template.Spec.Containers[0].Image
existing.Spec.Template.Spec.Containers[0].Resources = desired.Spec.Template.Spec.Containers[0].Resources
return r.Update(ctx, existing)
}
// handleDeletion runs cleanup before the Database resource is deleted
func (r *DatabaseReconciler) handleDeletion(ctx context.Context, db *dbv1alpha1.Database) (ctrl.Result, error) {
logger := log.FromContext(ctx)
if controllerutil.ContainsFinalizer(db, databaseFinalizer) {
// Update status
r.updateStatus(ctx, db, "Deleting", "Running cleanup before deletion")
// Deregister from service catalog
if err := r.CatalogClient.Deregister(ctx, db.Namespace, db.Name); err != nil {
logger.Error(err, "Failed to deregister from service catalog")
// Continue with deletion even if catalog deregistration fails
}
// Note: StatefulSet, Service, and Secret are owned resources
// They will be garbage collected automatically via owner references
// We only need to handle external resources (like service catalog) here
// Remove finalizer to allow Kubernetes to delete the resource
controllerutil.RemoveFinalizer(db, databaseFinalizer)
if err := r.Update(ctx, db); err != nil {
return ctrl.Result{}, fmt.Errorf("failed to remove finalizer: %w", err)
}
logger.Info("Database deletion complete", "database", db.Name)
}
return ctrl.Result{}, nil
}
// isStatefulSetReady checks if all replicas are ready
func (r *DatabaseReconciler) isStatefulSetReady(ctx context.Context, db *dbv1alpha1.Database) (bool, error) {
sts := &appsv1.StatefulSet{}
err := r.Get(ctx, types.NamespacedName{
Name: fmt.Sprintf("%s-postgresql", db.Name),
Namespace: db.Namespace,
}, sts)
if err != nil {
return false, err
}
return sts.Status.ReadyReplicas == *sts.Spec.Replicas, nil
}
func (r *DatabaseReconciler) updateStatus(ctx context.Context, db *dbv1alpha1.Database, phase, message string) error {
db.Status.Phase = phase
db.Status.Message = message
return r.Status().Update(ctx, db)
}
// SetupWithManager registers the controller and sets up watches
func (r *DatabaseReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&dbv1alpha1.Database{}). // Primary resource to watch
Owns(&appsv1.StatefulSet{}). // Watch owned StatefulSets
Owns(&corev1.Service{}). // Watch owned Services
Owns(&corev1.Secret{}). // Watch owned Secrets
WithOptions(controller.Options{
MaxConcurrentReconciles: 3, // Reconcile up to 3 databases in parallel
}).
Complete(r)
}
// Helper functions
func labelsForDatabase(db *dbv1alpha1.Database) map[string]string {
return map[string]string{
"app.kubernetes.io/name": "postgresql",
"app.kubernetes.io/instance": db.Name,
"app.kubernetes.io/managed-by": "database-operator",
"db.mycompany.com/database": db.Name,
}
}
func int64Ptr(i int64) *int64 { return &i }
func stringPtr(s string) *string { return &s }
Step 4: The Custom Resource (How Developers Use It)
# Developer creates this — operator handles everything else
apiVersion: db.mycompany.com/v1alpha1
kind: Database
metadata:
name: myapp-db
namespace: production
spec:
version: "15"
storage: "50Gi"
replicas: 1
databaseName: myapp
resources:
cpuRequest: "500m"
memoryRequest: "1Gi"
cpuLimit: "2"
memoryLimit: "4Gi"
backup:
enabled: true
schedule: "0 2 * * *" # Daily at 2 AM
s3Bucket: "mycompany-db-backups"
# Apply the Database resource
kubectl apply -f database.yaml
# Watch the operator provision everything
kubectl get database myapp-db -n production -w
# NAME VERSION PHASE READY AGE
# myapp-db 15 Provisioning False 10s
# myapp-db 15 Provisioning False 25s
# myapp-db 15 Ready True 45s
# Check what the operator created
kubectl get statefulset,service,secret -n production -l db.mycompany.com/database=myapp-db
# NAME READY AGE
# statefulset.apps/myapp-db-postgresql 1/1 45s
#
# NAME TYPE CLUSTER-IP PORT(S) AGE
# service/myapp-db-postgresql ClusterIP 10.96.1.45 5432/TCP 45s
# service/myapp-db-postgresql-headless ClusterIP None 5432/TCP 45s
#
# NAME TYPE DATA AGE
# secret/myapp-db-postgresql-credentials Opaque 6 45s
# Describe the database for connection info
kubectl describe database myapp-db -n production
# Status:
# Phase: Ready
# Connection String: postgresql://myapp-db-postgresql.production.svc.cluster.local:5432/myapp
# Credentials Secret: myapp-db-postgresql-credentials
# Delete everything by deleting just the Database resource
kubectl delete database myapp-db -n production
# Operator runs finalizer cleanup, deregisters from catalog
# Owner references cascade-delete the StatefulSet, Service, and Secret
Step 5: Testing the Operator
// controllers/database_controller_test.go
package controllers
import (
"context"
"time"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"sigs.k8s.io/controller-runtime/pkg/client"
dbv1alpha1 "github.com/mycompany/database-operator/api/v1alpha1"
)
var _ = Describe("Database Controller", func() {
const timeout = time.Second * 30
const interval = time.Second * 1
Context("When creating a Database resource", func() {
It("Should create a StatefulSet, Service, and Secret", func() {
ctx := context.Background()
db := &dbv1alpha1.Database{
ObjectMeta: metav1.ObjectMeta{
Name: "test-db",
Namespace: "default",
},
Spec: dbv1alpha1.DatabaseSpec{
Version: "15",
Storage: "10Gi",
Replicas: 1,
DatabaseName: "testdb",
Resources: dbv1alpha1.DatabaseResources{
CPURequest: "100m",
MemoryRequest: "256Mi",
CPULimit: "500m",
MemoryLimit: "512Mi",
},
},
}
Expect(k8sClient.Create(ctx, db)).Should(Succeed())
// Verify StatefulSet is created
sts := &appsv1.StatefulSet{}
Eventually(func() error {
return k8sClient.Get(ctx, client.ObjectKey{
Name: "test-db-postgresql",
Namespace: "default",
}, sts)
}, timeout, interval).Should(Succeed())
// Verify Secret is created with all required keys
secret := &corev1.Secret{}
Eventually(func() error {
return k8sClient.Get(ctx, client.ObjectKey{
Name: "test-db-postgresql-credentials",
Namespace: "default",
}, secret)
}, timeout, interval).Should(Succeed())
Expect(secret.Data).To(HaveKey("username"))
Expect(secret.Data).To(HaveKey("password"))
Expect(secret.Data).To(HaveKey("connection-string"))
// Verify owner references are set (for garbage collection)
Expect(sts.OwnerReferences).To(HaveLen(1))
Expect(sts.OwnerReferences[0].Name).To(Equal("test-db"))
Expect(secret.OwnerReferences[0].Name).To(Equal("test-db"))
})
})
})
# Run tests using envtest (spins up a real API server)
make test
# Build and deploy operator
make docker-build docker-push IMG=registry.mycompany.com/database-operator:v0.1.0
make deploy IMG=registry.mycompany.com/database-operator:v0.1.0
# Verify operator is running
kubectl get pods -n database-operator-system
️ Trade-offs & Alternatives
| Approach | Flexibility | Complexity | Maintenance | Best For | |---|---|---|---|---| | Custom Operator (Go) | Maximum | High | High | Complex stateful workflows | | Helm chart | Medium | Low | Low | Simple, static deployments | | Crossplane | High | Medium | Medium | Infrastructure provisioning | | Kro (Resource Orchestrator) | Medium | Low | Low | Composing existingresources | | KEDA + Jobs | Medium | Low | Low | Event-driven, stateless tasks | | Ansible Operator | Medium | Medium | Medium | Teams with Ansible expertise |
️ Common Mistakes & Misconceptions
- "My reconciler runs once and exits." — Reconcilers must be idempotent and run repeatedly. Every reconcile call should handle the case where resources already exist gracefully. Use
errors.IsNotFound()checks, not assume fresh state. - "I don't need finalizers if I set owner references." — Owner references only garbage-collect resources within the cluster. External resources (service catalog registrations, cloud databases, DNS entries) require finalizers for proper cleanup.
- "I should update all fields of owned resources on every reconcile." — Only update fields you own. Kubernetes controllers like the StatefulSet controller manage fields you shouldn't touch (e.g.,
status,resourceVersion). Blindly overwriting causes infinite reconcile loops. - "My operator needs to handle every edge case on day one." — Start with happy-path reconciliation, add error handling and edge cases iteratively. Operators are software — ship a v1 and improve.
Key Takeaway
Operators encode operational knowledge as code. The reconciliation loop is the fundamental primitive: continuously compare desired state (the CRD spec) against actual state (what exists in the cluster and external systems), and take the minimum actions needed to close the gap. Owner references handle in-cluster garbage collection automatically, while finalizers handle external cleanup. The result is infrastructure that manages itself — a database that provisions, monitors, and cleans itself up purely through Kubernetes resource declarations.
Self-Assessment Checklist
- [ ] Can you explain why reconcilers must be idempotent?
- [ ] Can you describe the difference between owner references and finalizers?
- [ ] Can you explain what
controllerutil.SetControllerReferencedoes? - [ ] Can you describe how
Owns()inSetupWithManageraffects the watch list? - [ ] Can you explain why you should never regenerate passwords on every reconcile call?
- [ ] Can you write a basic reconciler that creates a ConfigMap if it doesn't exist?
Finished reading?
Mark as complete to claim your +25 XP and track progress.