Persistent Storage for Your Data
Pods are disposable and disks are not. PersistentVolumes, claims, StorageClasses and access modes — enough to run something stateful without losing it.
Pods are disposable and disks are not. PersistentVolumes, claims, StorageClasses and access modes — enough to run something stateful without losing it.
A pod's filesystem dies with the pod, exactly as a container's did in Docker — and in Kubernetes pods are deleted far more casually. A node is drained, a rollout replaces every replica, a scheduler moves work around. Anything written inside is gone each time.
So a database needs storage that is not part of the pod, and it needs
that storage to follow the pod if it is rescheduled onto a different
machine. By the end of this lesson you will know the three objects that
make that work, why the indirection between them exists, and what
Pending on a claim is telling you.
Not every volume is about durability, and knowing the temporary ones first stops you reaching for a claim when you do not need one.
emptyDir is created when the pod starts and deleted when it goes
away. It exists so containers in one pod can share a directory — the
sidecar in the pods lesson used it — and as scratch space:
volumes:
- name: cache
emptyDir: {}
- name: scratch
emptyDir:
medium: Memory # a tmpfs, capped by the pod's memory limitconfigMap and secret volumes project configuration into the
filesystem, as the last lesson covered. Read-only, and not storage.
All three are tied to the pod's lifetime. For anything that must survive, you need the persistent set.
Persistent storage in Kubernetes is deliberately split in three, and the split is the concept worth learning.
What you write.
A request: this much storage, with this access mode, of this class. Lives in a namespace, and a pod refers to it by name.
"I need 10 GB that survives restarts" — and nothing about where it comes from.
The real storage.
A cloud disk, an NFS export, a path on a node. Cluster-scoped, belonging to no namespace, and in practice created for you.
What the cluster can create on demand.
Fast SSD, cheap spinning disk, replicated — and the driver that provisions it.
The middle object exists so that the same manifests work on your laptop and in production. A pod that named a disk directly could not do that.
Pod ──mounts──> PVC ──bound to──> PV ──is──> real disk
│ ▲
└──── created by ────┘
StorageClass (dynamically)Look at what your cluster can provision:
kubectl get storageclassNAME PROVISIONER DEFAULT AGE
standard (default) rancher.io/local-path true 1hkind installs local-path, which allocates a directory on the node.
Managed clusters give you the provider's disks — gp3 on EKS,
standard-rwo on GKE.
Then the claim:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: postgres-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Gi
# storageClassName: standard # omitted = the default classkubectl apply -f pvc.yaml
kubectl get pvcNAME STATUS VOLUME CAPACITY ACCESS AGE
postgres-data Pending 3sPending here is expected with local-path, which waits until a pod
actually uses the claim before deciding which node to put it on. Other
provisioners bind immediately.
Now mount it:
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgres
spec:
replicas: 1
selector:
matchLabels:
app: postgres
template:
metadata:
labels:
app: postgres
spec:
volumes:
- name: data
persistentVolumeClaim:
claimName: postgres-data
containers:
- name: postgres
image: postgres:16
env:
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: notes-secrets
key: DATABASE_PASSWORD
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
volumeMounts:
- name: data
mountPath: /var/lib/postgresql/datakubectl apply -f postgres.yaml
kubectl get pvcNAME STATUS VOLUME CAPACITY ACCESS AGE
postgres-data Bound pvc-8f3c1a 8Gi RWO 2mBound, with a PV created for it that you never wrote.
Then prove it works, which is the only way to know:
kubectl exec deploy/postgres -- psql -U postgres \
-c "CREATE TABLE notes (id serial, body text);"
kubectl delete pod -l app=postgres # the Deployment replaces it
kubectl exec deploy/postgres -- psql -U postgres -c "\dt"
# the table is still thereAn access mode says how many nodes may mount the volume, and it is a property of the underlying storage rather than a wish.
| Mode | Short | Meaning |
|---|---|---|
ReadWriteOnce | RWO | One node may mount it read-write |
ReadOnlyMany | ROX | Many nodes may mount it read-only |
ReadWriteMany | RWX | Many nodes may mount it read-write |
ReadWriteOncePod | RWOP | Exactly one pod, cluster-wide |
ReadWriteOnce is the one that catches people, because "once" sounds
like one pod and means one node. Two pods on the same node can share
an RWO volume; two pods on different nodes cannot.
And crucially: most block storage only supports RWO. Cloud disks
attach to one machine at a time — that is a property of the hardware
model, not a Kubernetes limitation. ReadWriteMany needs a networked
filesystem: NFS, EFS, Azure Files, CephFS.
That single fact explains the mistake below.
Bad — three replicas, one claim, one RWO volume:
spec:
replicas: 3
template:
spec:
volumes:
- name: data
persistentVolumeClaim:
claimName: postgres-dataGood — a StatefulSet, where each pod gets its own claim:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgres
spec:
replicas: 3
serviceName: postgres
selector:
matchLabels:
app: postgres
template:
metadata:
labels:
app: postgres
spec:
containers:
- name: postgres
image: postgres:16
volumeMounts:
- name: data
mountPath: /var/lib/postgresql/data
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: [ReadWriteOnce]
resources:
requests:
storage: 8GiThe bad version fails in the most confusing way available: which failure you get depends on where the scheduler happened to put the pods.
A visible, stuck failure.
Whichever pod is on the node holding the volume starts
normally. The rest sit in ContainerCreating forever with a
FailedAttachVolume event.
Annoying, and at least it tells you.
All three start, and corrupt the data.
RWO means one node, so three pods on one node can all mount it — and three Postgres processes writing to one data directory destroy it.
Nothing reports a problem until the data is already wrong.
volumeClaimTemplates is the fix and the reason StatefulSets exist:
one claim per pod, created automatically, named after the pod, and
reattached to the same pod when it is rescheduled.
A PVC stuck in Pending has a small set of causes, and describe names
which one:
kubectl describe pvc postgres-dataEvents:
Warning ProvisioningFailed no persistent volumes available for
this claim and no storage class is set| Message | Cause |
|---|---|
no storage class is set | No default class; name one explicitly |
WaitForFirstConsumer | Normal — waiting for a pod to be scheduled |
ProvisioningFailed | The driver could not create it; read the text |
FailedAttachVolume on a pod | RWO volume wanted on a second node |
exceeded quota | A ResourceQuota caps storage in the namespace |
Two more things about the objects themselves.
A PVC cannot usually shrink, and can often grow. If the
StorageClass has allowVolumeExpansion: true, editing the claim's
storage request resizes the volume in place. Shrinking is not
supported by any common driver.
Deleting a PVC may delete your data. The PV's
persistentVolumeReclaimPolicy decides: Delete, which is the default
for dynamically provisioned volumes, destroys the underlying disk;
Retain keeps it for manual recovery.
kubectl get pv # look at the RECLAIM POLICY column
kubectl patch pv pvc-8f3c1a \
-p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'For anything you would be upset to lose, set Retain and take real
backups. A PVC deleted by a careless kubectl delete -f is otherwise
indistinguishable from a deliberate one.
# Ephemeral: dies with the pod
volumes:
- name: cache
emptyDir: {}
- name: scratch
emptyDir: { medium: Memory } # tmpfs
# Persistent: a claim, then mount it
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: postgres-data
spec:
accessModes: [ReadWriteOnce]
resources:
requests:
storage: 8Gi
storageClassName: standard # omit for the default class
---
spec:
volumes:
- name: data
persistentVolumeClaim:
claimName: postgres-data
containers:
- name: postgres
volumeMounts:
- name: data
mountPath: /var/lib/postgresql/data
# More than one replica? One claim each, via a StatefulSet
volumeClaimTemplates:
- metadata: { name: data }
spec:
accessModes: [ReadWriteOnce]
resources: { requests: { storage: 8Gi } }# What this cluster can provision
kubectl get storageclass
# The objects
kubectl get pvc # namespaced: your requests
kubectl get pv # cluster-wide: the real storage
kubectl describe pvc postgres-data # why it is Pending
# Proving durability (the only real test)
kubectl exec deploy/postgres -- psql -U postgres -c "\dt"
kubectl delete pod -l app=postgres
kubectl exec deploy/postgres -- psql -U postgres -c "\dt"
# Access modes — "Once" means one NODE, not one pod
# ReadWriteOnce RWO one node read-write (most block storage)
# ReadOnlyMany ROX many nodes read-only
# ReadWriteMany RWX many nodes read-write (needs NFS/EFS/...)
# ReadWriteOncePod RWOP exactly one pod
# Growing a volume (if the class allows it)
kubectl patch pvc postgres-data \
-p '{"spec":{"resources":{"requests":{"storage":"16Gi"}}}}'
# Keeping the disk when the claim is deleted
kubectl get pv # check RECLAIM POLICY
kubectl patch pv pvc-8f3c1a \
-p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'Your data survives its pod. What the cluster still cannot tell is whether a running pod is actually working — a wedged process looks identical to a healthy one, and traffic keeps arriving at it.
Probes are the answer, and they are next. They also complete the Services story: readiness is what decides whether a pod appears in the endpoints list. Before moving on, run the durability test above yourself. Deleting a database pod and finding your table intact is the only evidence that counts.