Scaling State: Mastering PostgreSQL on Kubernetes with Operators

Understanding the Challenges of State in Kubernetes

Kubernetes was originally designed for stateless applications—web servers that could be killed and restarted on any node without consequence. Databases present three distinct challenges: identity, persistence, and synchronization. A database pod cannot just disappear; it needs its specific disk (Persistent Volume) to follow it, and it needs a consistent network identity so followers know where the primary node resides.

Standard Kubernetes primitives like StatefulSets solve the identity and volume mounting issues, but they don't understand 'database logic.' A StatefulSet doesn't know how to perform a graceful switchover if the primary node lags, nor does it understand how to verify if a replica is truly in sync before promoting it. This is where the Operator pattern shines, acting as a digital Site Reliability Engineer that monitors the internal state of the database engine.

Prerequisites and Architecture Overview

Before we dive into the YAML, ensure you have a functioning Kubernetes cluster (v1.26+) and the kubectl CLI configured. For storage, you must have a StorageClass defined that supports dynamic provisioning—standard on major cloud providers like AWS (EBS), GCP (Persistent Disk), or Azure (Managed Disks).

We will be using CloudNativePG (CNPG) because it follows the 'Primary/Standby' architecture natively without requiring external tools like Patroni or Sentinel. The operator manages the entire lifecycle, including:

• Self-healing through automated failover
• Rolling updates for security patches
• Backup management to S3 or GCS
• Horizontal scaling of read-replicas

Step 1: Installing the Operator

The first step is to install the operator's Custom Resource Definitions (CRDs) and the controller. This controller will watch for any 'Cluster' resources we create and take action to match the desired state.

kubectl apply -f https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.22/releases/cnpg-1.22.0.yaml

Wait for the operator pod to reach a 'Running' state in the cnpg-system namespace. Once active, your Kubernetes cluster now 'understands' how to manage PostgreSQL natively.

Step 2: Defining the Cluster Resource

Instead of writing standard Pod or Deployment specs, we define a Cluster object. This manifest describes our desired database topology, including the number of instances and storage requirements.

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: production-db
spec:
  instances: 3
  primaryUpdateStrategy: unsupervised
  storage:
    size: 20Gi
    storageClass: standard
  walStorage:
    size: 10Gi
    storageClass: standard
  monitoring:
    enablePodMonitor: true

In this configuration, we specify three instances. The operator will automatically designate one as the primary (read-write) and the other two as hot standbys (read-only). Note the separation of data storage and WAL (Write-Ahead Log) storage; separating these is a performance best practice to prevent disk I/O contention.

Step 3: Implementing Automated Backups

A database without a backup is just a temporary cache. CloudNativePG integrates directly with object storage. By defining a 'Backup' configuration, the operator will continuously stream WAL files to your bucket, enabling Point-In-Time Recovery (PITR).

Reliability in DevOps isn't just about preventing failure; it's about making the recovery process so boring that it becomes a non-event.

You will need to create a Kubernetes Secret containing your cloud provider credentials, then reference it in the cluster's backup section. This ensures that even if the entire Kubernetes cluster is lost, your data remains safe and restorable to a new environment.

Step 4: Observability and Maintenance

Once your cluster is live, you need to monitor its health. The operator automatically exports Prometheus metrics on port 9187. You can visualize connection pools, replication lag, and transaction rates using standard Grafana dashboards.

When it's time to upgrade PostgreSQL, you simply update the 'imageName' in your YAML. The operator performs a canary-style update: it upgrades a replica, waits for it to sync, promotes it to primary, and then updates the old primary. This cycle minimizes downtime to just a few seconds during the switchover.