beginner30 minby DevFox

Requests, Limits and Scheduling

Why a pod sits Pending forever, and why another one keeps getting killed. CPU and memory requests versus limits, OOMKilled, and how the scheduler places work.

Performance
Kubernetes

Two symptoms bring people to this lesson. A pod that sits in Pending and never starts, on a cluster with machines that look idle. And a pod that starts fine, runs for twenty minutes, and dies with OOMKilled — repeatedly.

Both are about the same two numbers: how much CPU and memory a container requests, and how much it is limited to. By the end of this lesson you will know what each number actually does, why they are enforced completely differently, and how to choose them without guessing twice.

Two numbers with different jobs

Every container can declare both, and they mean genuinely different things:

yaml

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 500m
    memory: 256Mi

requests

A reservation. Decides placement.

What the scheduler uses to decide whether a pod fits on a node. Once placed, the amount is held for it whether or not it is used.

Get this wrong and the pod sits in Pending.

limits

A ceiling. Enforced at run time.

Enforced by the kernel on the node. The container cannot exceed it, and the scheduler never looks at it.

Get the memory one wrong and the pod is OOMKilled.

Different jobs, different enforcers, different symptoms when you get one wrong

How the scheduler uses requests

The scheduler's arithmetic is simpler than people expect, and it is the key to Pending.

For each node, it adds up the requests of every pod already assigned there. If the node's remaining allocatable capacity is at least this pod's request, the node is a candidate. Otherwise it is rejected.

Actual usage is not part of this calculation at all. A node whose pods request 90% of its CPU but use 5% is 90% full to the scheduler. This is exactly why a cluster with visibly idle machines refuses to place a pod.

Two things to read there. Requests above about 90% mean this node is effectively full. And limits exceeding 100% is normal and expected — that is overcommitment, and it is the point: containers rarely peak together, so allowing the sum of ceilings to exceed the machine gets you much better utilisation.

Note also that Allocatable is less than the machine's capacity. The kubelet reserves some for the operating system and for itself.

Diagnosing Pending

describe on the pod ends with the scheduler's own explanation, node by node:

That message is a list of reasons with counts, and it is precise. Here: one node is the control plane, which normally carries a taint that ordinary pods do not tolerate, and the two workers do not have enough unreserved CPU.

The reasons you will actually see:

Reason	Meaning
`Insufficient cpu` / `memory`	Requests exceed what is unreserved
`untolerated taint`	The node is marked as not for this pod
`node(s) didn't match node selector`	A `nodeSelector` matches nothing
`didn't find available persistent volumes`	A PVC cannot be satisfied
`too many pods`	The node's pod-count cap is reached

The fixes are correspondingly direct: lower the request, add a node, or remove the constraint you did not mean to add.

kubectl top needs the metrics server installed, and it is worth having: the gap between top and describe — real usage versus reserved — is the whole story of a badly tuned cluster.

Why CPU and memory limits behave differently

This is the part that surprises everyone, and it explains OOMKilled entirely.

CPU is compressible

Exceeding the limit throttles you.

The kernel gives the container fewer slices of time. It gets slower.

Nothing is killed, no error appears, and kubectl get pods says everything is healthy. The only symptom is latency — which is why many teams deliberately set no CPU limit at all.

Memory is not

Exceeding the limit kills you.

There is no way to give a process slightly less memory than it just asked for, so the kernel's OOM killer terminates the container and the kubelet restarts it. Exit code 137.

Loud, unambiguous, and never a bug in Kubernetes.

One of these announces itself. The other does not.

OOMKilled is never a bug in Kubernetes and never ambiguous: the container asked for more memory than its limit allowed. Either the limit is too low, or the application has a leak. kubectl top pod over time distinguishes them — a leak climbs steadily and a genuine requirement plateaus.

Quality of service, and who dies first

The relationship between your requests and limits puts the pod into one of three classes, and the class decides who dies when a node runs out of memory.

Guaranteed — evicted last
Every container has requests equal to limits. Given a dedicated allocation. Worth it for the workloads you cannot afford to lose.
Burstable — evicted second
Requests set, and lower than limits. The common case, and the right default for most applications.
BestEffort — evicted first
Nothing set at all. Placed anywhere, because the scheduler thinks it needs nothing, and the first thing thrown off a node under pressure.

The kubelet evicts from the bottom up. The width is roughly how many of your pods should be in each class.

In YAML, the three look like this:

When a node is under memory pressure the kubelet evicts BestEffort pods first, then Burstable pods that are exceeding their requests, and Guaranteed pods last. Setting resources is therefore not just tidiness — it is what keeps your important workloads on the node.

The mistake: nothing set at all

Bad — no resources declared:

Good — a request the scheduler can use, and a memory ceiling:

With nothing declared, the scheduler believes the pod needs nothing, so it will pack any number of them onto one node. The pods then compete for real CPU and memory that was never accounted for, one memory spike takes the node into pressure, and the kubelet starts evicting — beginning with exactly these pods, because BestEffort goes first. A single leaking pod can destabilise every workload on the machine.

Note what the good version does not set: a CPU limit. Requests give the scheduler its arithmetic and provide a proportional share under contention; a CPU limit only adds throttling. A memory limit, by contrast, is worth having — it turns an unbounded leak into one restarted pod instead of a dead node.

Choosing the numbers

You cannot reason your way to these values; you measure them. The process is short:

Deploy with a generous guess — a request at roughly your expectation and a limit at two or three times it.
Run realistic load for long enough to see a steady state.
Read the actual usage:

Set the request near the steady-state average, and the memory limit above the observed peak with headroom.
Revisit when the application changes.

Two extras that automate the loop, once the basics are in place. The VerticalPodAutoscaler observes usage and recommends — or applies — requests. And a LimitRange in a namespace supplies defaults so a pod with no resources declared does not become BestEffort by accident:

A ResourceQuota is the matching cap on a whole namespace, which is how a shared cluster stops one team consuming it:

With a quota in place, a pod with no requests is rejected rather than admitted as BestEffort — which is often the behaviour you wanted all along.

Steering pods to particular nodes

Resources are how the scheduler chooses; three mechanisms let you constrain the choice. Enough to recognise them:

Taints mark a node as unsuitable, and tolerations let specific pods ignore the mark — which is why the control-plane node in your kind cluster runs no workloads of yours.

The one worth reaching for early is topologySpreadConstraints. Three replicas on one node give you no protection from that node failing, and by default the scheduler is free to place them that way.

Cheat sheet

Where to go next

The scheduler now has what it needs, and your pods have bounds. The next lesson is the one all of this has been building towards: shipping a new version without dropping a request — how a rolling update actually works, what maxSurge and maxUnavailable do, and how to undo one.

Before that, run kubectl top pods against something you have deployed and compare it with what you requested. The first time you see how far off a guess was is what turns this lesson from advice into habit.