Kubernetes Persistent Volumes — PV, PVC, StorageClass, and Stateful Workloads

The storage problem in Kubernetes#

Pods are ephemeral. When a pod dies, its filesystem dies with it. For stateless apps this is fine. For databases, message queues, and anything that stores data, this is a dealbreaker.

Kubernetes solves this with persistent volumes — storage that outlives pods.

The three-layer storage model#

Kubernetes separates storage into three abstractions:

This separation lets developers request storage without knowing the underlying infrastructure.

PersistentVolume (PV)#

A PV is a cluster-level resource that represents a piece of storage. It can be backed by local disks, NFS, cloud block storage (EBS, GCE PD, Azure Disk), or any CSI-compatible driver.

PV specification#

apiVersion: v1
kind: PersistentVolume
metadata:
  name: database-pv
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: fast-ssd
  csi:
    driver: ebs.csi.aws.com
    volumeHandle: vol-0abc123def456

PV lifecycle phases#

• Available — not yet bound to a claim
• Bound — bound to a PVC
• Released — the PVC was deleted, but the PV has not been reclaimed
• Failed — automatic reclamation failed

PersistentVolumeClaim (PVC)#

A PVC is how pods request storage. You specify how much storage you need, what access mode, and optionally which StorageClass to use.

PVC specification#

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: database-storage
  namespace: production
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  storageClassName: fast-ssd

Using a PVC in a pod#

apiVersion: v1
kind: Pod
metadata:
  name: postgres
spec:
  containers:
    - name: postgres
      image: postgres:16
      volumeMounts:
        - mountPath: /var/lib/postgresql/data
          name: db-storage
  volumes:
    - name: db-storage
      persistentVolumeClaim:
        claimName: database-storage

How PVC binding works#

1. You create a PVC with requirements (size, access mode, StorageClass)
2. Kubernetes finds a matching PV (or dynamically creates one)
3. The PV is bound to the PVC (one-to-one relationship)
4. The pod mounts the PVC and reads/writes data
5. If no matching PV exists and no StorageClass can provision one, the PVC stays in Pending state

StorageClass#

A StorageClass defines how to dynamically provision PVs. Instead of pre-creating PVs, Kubernetes creates them on demand when a PVC is submitted.

StorageClass specification#

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iops: "5000"
  throughput: "250"
  encrypted: "true"
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

Common StorageClass parameters by provider#

Volume binding modes#

• Immediate — PV is provisioned as soon as the PVC is created (default)
• WaitForFirstConsumer — PV is provisioned only when a pod using the PVC is scheduled (recommended for topology-aware storage)

WaitForFirstConsumer prevents the situation where a PV is created in availability zone A but the pod is scheduled in zone B.

Access modes#

Access modes define how a volume can be mounted across nodes.

Which mode to use#

• Databases (Postgres, MySQL) — ReadWriteOnce (single writer)
• Shared config or static assets — ReadOnlyMany
• Shared file storage (uploads, media) — ReadWriteMany (requires NFS or similar)
• Strict single-writer guarantee — ReadWriteOncePod

Not all storage backends support all modes. EBS only supports RWO. EFS supports RWX. Check your CSI driver documentation.

Reclaim policies#

When a PVC is deleted, the reclaim policy determines what happens to the PV and its data.

Best practice#

Use Retain for production data. Use Delete for dev/test. Never rely on Recycle.

For Retain volumes, after the PVC is deleted:

1. The PV moves to Released state
2. Back up the data if needed
3. Delete the PV object to clean up
4. Or remove the claimRef to make it available again

CSI drivers#

The Container Storage Interface (CSI) is the standard for connecting Kubernetes to storage backends. Every major cloud and storage vendor provides a CSI driver.

Popular CSI drivers#

Installing a CSI driver#

Most managed Kubernetes services (EKS, GKE, AKS) include CSI drivers as add-ons. For self-managed clusters, install via Helm:

helm repo add aws-ebs-csi-driver https://kubernetes-sigs.github.io/aws-ebs-csi-driver
helm install aws-ebs-csi-driver aws-ebs-csi-driver/aws-ebs-csi-driver \
  --namespace kube-system

Stateful workloads with StatefulSets#

StatefulSets are designed for stateful applications. They provide stable network identities and persistent storage for each replica.

StatefulSet with volumeClaimTemplates#

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:16
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: fast-ssd
        resources:
          requests:
            storage: 50Gi

What StatefulSets guarantee#

• Stable identifiers — postgres-0, postgres-1, postgres-2 (not random suffixes)
• Per-replica storage — each pod gets its own PVC (data-postgres-0, data-postgres-1)
• Ordered deployment — pods start in order (0, then 1, then 2)
• Ordered termination — pods stop in reverse order
• PVCs persist — scaling down does not delete PVCs (data is preserved)

Volume expansion#

Resize volumes without downtime (if the StorageClass allows it).

Requirements#

• StorageClass must have allowVolumeExpansion: true
• The CSI driver must support expansion
• You can only increase size, never decrease

How to expand#

kubectl patch pvc database-storage -p \
  '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'

Some drivers require the pod to be restarted for the filesystem to be resized. Others support online expansion.

Backup strategies#

Persistent data needs backup. Kubernetes does not handle this natively.

Volume snapshots#

Kubernetes supports VolumeSnapshots (CSI feature):

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: db-backup-2026-03-29
spec:
  volumeSnapshotClassName: csi-aws-snapclass
  source:
    persistentVolumeClaimName: database-storage

Restore from a snapshot by referencing it in a new PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: database-restored
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: fast-ssd
  resources:
    requests:
      storage: 100Gi
  dataSource:
    name: db-backup-2026-03-29
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io

Application-level backups#

• Velero — backs up Kubernetes resources and persistent volumes together
• Kasten K10 — enterprise Kubernetes backup and disaster recovery
• Stash — backup for Kubernetes workloads using restic
• pg_dump / mysqldump — application-aware backups run as CronJobs

Backup best practices#

1. Use VolumeSnapshots for block-level backups — fast, consistent, point-in-time
2. Use application-level backups for logical backups — portable, inspectable
3. Test restores regularly — a backup you have never restored is not a backup
4. Automate with CronJobs — schedule snapshots and dumps on a recurring basis
5. Store backups off-cluster — use S3, GCS, or another region for disaster recovery

Common pitfalls#

1. PVC stuck in Pending — no matching PV or StorageClass, check kubectl describe pvc
2. Pod stuck in ContainerCreating — volume not yet attached, check node affinity and zone
3. Data loss on scale-down — StatefulSet PVCs are not deleted, but Deployment PVCs can be
4. Wrong access mode — trying RWX on EBS (only supports RWO)
5. No volume expansion — forgot allowVolumeExpansion: true on StorageClass
6. Zone mismatch — PV in zone A, pod scheduled in zone B (use WaitForFirstConsumer)

Visualize storage architecture on Codelit#

On Codelit, generate any Kubernetes architecture with stateful workloads and see how PVs, PVCs, and StorageClasses connect to your pods. Explore storage topology, replication, and backup flows visually.

---

Article #433 in the Codelit engineering series. Explore our full library of system design, infrastructure, and architecture guides at codelit.io.