beginner30 minby DevFox

Rolling Updates and Rollbacks

Shipping a new version without dropping a request: rollout strategy, maxSurge and maxUnavailable, watching a rollout, and undoing one that went wrong.

Kubernetes

Deploying with Compose meant stopping the old container and starting the new one, and accepting that requests failed in between. That was the fourth thing on the list of problems Kubernetes exists to absorb, and this is the lesson where it gets absorbed.

By the end of it you will have run a rolling update and watched it happen pod by pod, you will know what maxSurge and maxUnavailable control, and you will be able to stop or reverse a bad release in one command.

What triggers a rollout

A rollout starts when the pod template changes — anything under spec.template. A new image, an added environment variable, a modified probe, a new annotation.

Changing something outside the template does not: replicas is a scaling operation, not a rollout, because the existing pods are still correct.

bash

# these start a rollout
kubectl set image deployment/api api=notes-api:0.2.0
kubectl apply -f deployment.yaml        # if the template changed
kubectl rollout restart deployment/api  # a template annotation

# this does not
kubectl scale deployment api --replicas=5

kubectl rollout restart is worth knowing precisely: it stamps a timestamp annotation into the template, which is a change, which starts a normal rolling update. That is how you restart every pod without downtime — and the correct response to a changed ConfigMap that your pods read as environment variables.

Two ReplicaSets, passing each other

The mechanism is the one the Deployments lesson hinted at. A Deployment does not modify pods; it creates a new ReplicaSet for the new template and moves replicas from the old one to the new one, a few at a time.

bash

kubectl set image deployment/api api=notes-api:0.2.0
kubectl get replicasets --watch

text

NAME             DESIRED   CURRENT   READY   AGE
api-6c9f7d4b58   3         3         3       10m    # old
api-7d8b5f9c64   1         1         0       2s     # new
api-6c9f7d4b58   3         3         3       10m
api-7d8b5f9c64   1         1         1       8s     # first one ready
api-6c9f7d4b58   2         2         2       10m    # one old removed
api-7d8b5f9c64   2         2         1       9s
...
api-6c9f7d4b58   0         0         0       11m    # old drained
api-7d8b5f9c64   3         3         3       25s    # new at full

Read the order carefully, because it is the whole guarantee.

A new pod is created
On the new ReplicaSet. The old three are untouched and still serving.
Wait for it to become ready
This is the readiness probe from the last lesson, and it is the entire safety mechanism. Nothing proceeds until it passes.
Only then, remove one old pod
It leaves the endpoints first, then shuts down inside its grace period.

Repeat until the old ReplicaSet is at zero. Capacity never drops below what you asked for.

That guarantee rests entirely on the readiness probe from the last lesson. Without one, a pod counts as ready the moment its process starts, and the rollout cheerfully replaces every working pod with pods that are not yet able to serve — a zero-downtime deployment that drops every request.

Watching and controlling one

bash

kubectl rollout status deployment/api

text

Waiting for deployment "api" rollout to finish: 1 of 3 updated...
Waiting for deployment "api" rollout to finish: 2 of 3 updated...
deployment "api" successfully rolled out

rollout status blocks until the rollout settles and exits non-zero if it fails, which makes it the right last line of a deploy script — the pipeline waits for the release rather than declaring success on apply.

Two commands for a rollout you are unsure about:

bash

kubectl rollout pause deployment/api     # stop where it is
kubectl rollout resume deployment/api    # carry on

Pausing is useful mid-release: you now have some pods on the new version and some on the old, and you can look at logs and metrics before committing. It is also the crude version of a canary — one new pod serving a fraction of traffic while you watch.

maxSurge and maxUnavailable

Two fields decide how aggressive the replacement is:

yaml

spec:
  replicas: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25% # extra pods allowed above replicas
      maxUnavailable: 25% # pods allowed to be missing

Those are the defaults. With ten replicas, 25% each way means the rollout may run up to 12 pods at once and must keep at least 8 available. Both accept a percentage or an absolute number.

Three configurations worth recognising.

maxSurge: 1, maxUnavailable: 0

Full capacity throughout. The safest.

One extra pod exists at a time, and no old pod leaves until a new one is ready.

Also the slowest — and it cannot progress if the cluster has no room for one more pod, which presents as a rollout that simply stops with the new pod Pending.

maxSurge: 50%, maxUnavailable: 50%

Fast, and capacity dips.

Half the fleet is replaced at once. Fine when you have headroom and the release is routine.

maxSurge: 0, maxUnavailable: 1

Never exceeds replicas.

For a cluster with no spare room, or a workload whose licence or connection limit is tied to the replica count. Capacity dips by one throughout.

You are trading speed against capacity. There is no setting that gives you both.

There is also the other strategy, and its one legitimate use:

yaml

strategy:
  type: Recreate

Recreate kills every old pod before creating any new ones — deliberate downtime. It is correct when two versions genuinely cannot coexist: a schema migration that the old code cannot read, or a single writer holding an exclusive lock on a volume.

Rolling back

Kubernetes keeps the old ReplicaSets, scaled to zero, which is what makes rollback cheap:

bash

kubectl rollout history deployment/api

text

REVISION  CHANGE-CAUSE
1         <none>
2         <none>
3         <none>

bash

kubectl rollout undo deployment/api              # to the previous
kubectl rollout undo deployment/api --to-revision=1
kubectl rollout history deployment/api --revision=2   # what it was

A rollback is a rolling update in reverse — the old ReplicaSet scales up, the new one scales down, with the same readiness guarantees. It is the one command worth having in muscle memory before you need it.

Those <none> entries are a missed opportunity. Record what each revision was:

yaml

metadata:
  annotations:
    kubernetes.io/change-cause: 'notes-api 0.2.0: add search endpoint'

bash

kubectl rollout history deployment/api

text

REVISION  CHANGE-CAUSE
1         notes-api 0.1.0: initial release
2         notes-api 0.2.0: add search endpoint

How much history you keep is configurable, and ten is the default:

yaml

spec:
  revisionHistoryLimit: 10

The mistake: a tag that does not change

Bad — the image tag is the same, so nothing rolls out:

bash

docker build -t myname/notes-api:latest .
docker push myname/notes-api:latest
kubectl apply -f deployment.yaml
# deployment.apps/api unchanged

Good — a new tag makes the template different, so a rollout happens:

bash

docker build -t myname/notes-api:0.2.0 .
docker push myname/notes-api:0.2.0
kubectl set image deployment/api api=myname/notes-api:0.2.0

A rollout is triggered by a change to the template, and image: myname/notes-api:latest is character-for-character identical before and after your push. Kubernetes has nothing to react to, reports unchanged, and the old pods keep running the old code — while every artefact you have says the deploy succeeded.

Worse, if a pod is recreated later for an unrelated reason, it pulls whatever latest points at then. Two pods of the same Deployment end up running different code with no record of it.

The fix is a unique tag per build — a version, or the commit SHA — which also means the Deployment manifest states exactly what is deployed.

When a rollout gets stuck

A rollout that never finishes has a short list of causes:

bash

kubectl rollout status deployment/api        # will time out
kubectl get pods -l app=api                  # which are not ready
kubectl describe deployment api              # conditions and events
kubectl describe pod <new-pod>               # the actual reason
kubectl logs <new-pod>                       # what it says

Symptom	Cause
New pod `ImagePullBackOff`	Bad tag, or missing pull secret
New pod `Running`, `READY 0/1`	Failing its readiness probe
New pod `CrashLoopBackOff`	Broken configuration or code
New pod `Pending`	No room for the surge pod
`ProgressDeadlineExceeded`	It has not progressed for 600s

The good news is where you are while this happens: a stuck rollout has made no progress, so the old pods are still serving. maxUnavailable: 0 means capacity was never reduced. You are not in an outage — you are in a paused release, and the correct move is usually:

bash

kubectl rollout undo deployment/api

Diagnose afterwards, with production restored.

progressDeadlineSeconds is what produces that last condition, and 600 is the default:

yaml

spec:
  progressDeadlineSeconds: 600

It marks the Deployment as failed after ten minutes of no progress. Note that it does not roll back automatically — Kubernetes has no automatic rollback. Tools built on top of it do: Argo Rollouts and Flagger run canary and blue-green releases with metric-based automatic reversion.

Cheat sheet

yaml

spec:
  replicas: 10
  revisionHistoryLimit: 10 # how many old ReplicaSets to keep
  progressDeadlineSeconds: 600 # mark failed after no progress
  strategy:
    type: RollingUpdate # or Recreate, for deliberate downtime
    rollingUpdate:
      maxSurge: 1 # extra pods allowed above replicas
      maxUnavailable: 0 # pods allowed to be missing
  template:
    metadata:
      annotations:
        kubernetes.io/change-cause: 'notes-api 0.2.0: search endpoint'

bash

# Starting a rollout (any change to spec.template)
kubectl set image deployment/api api=notes-api:0.2.0
kubectl apply -f deployment.yaml
kubectl rollout restart deployment/api    # restart all pods, no downtime

# Watching and controlling
kubectl rollout status deployment/api     # blocks; use it in a pipeline
kubectl get replicasets --watch           # the two sets passing
kubectl rollout pause deployment/api      # hold, inspect, then decide
kubectl rollout resume deployment/api

# Going back
kubectl rollout history deployment/api
kubectl rollout history deployment/api --revision=2
kubectl rollout undo deployment/api
kubectl rollout undo deployment/api --to-revision=1

# When it is stuck (the old pods are still serving)
kubectl get pods -l app=api               # which new pod is unhealthy
kubectl describe pod <new-pod>            # why
kubectl logs <new-pod>
kubectl rollout undo deployment/api       # restore first, diagnose after

# Strategy choices
#   maxSurge: 1,   maxUnavailable: 0    full capacity, slowest, safest
#   maxSurge: 50%, maxUnavailable: 50%  fast, capacity dips
#   maxSurge: 0,   maxUnavailable: 1    fixed footprint, capacity dips
#   type: Recreate                      all down, then all up

# Two rules
#   a readiness probe is what makes a rolling update safe
#   a unique image tag per build is what makes a rollout happen

Where to go next

You can ship and unship a version safely. What remains is the skill you will use most: working out why a pod is not running, with a method rather than a guess. That is the next lesson, and it collects the symptoms every previous lesson has mentioned into one order of operations.

Before that, run a rollout to an image tag that does not exist and watch what happens: the new pod fails to pull, the rollout stalls, and your application keeps serving from the old pods. Then undo it. Rehearsing a failed release while nothing is at stake is the whole point.