beginner30 minby DevFox

Requests, Limits and Scheduling

Why a pod sits Pending forever, and why another one keeps getting killed. CPU and memory requests versus limits, OOMKilled, and how the scheduler places work.

Kubernetes
Performance

Two symptoms bring people to this lesson. A pod that sits in Pending and never starts, on a cluster with machines that look idle. And a pod that starts fine, runs for twenty minutes, and dies with OOMKilled — repeatedly.

Both are about the same two numbers: how much CPU and memory a container requests, and how much it is limited to. By the end of this lesson you will know what each number actually does, why they are enforced completely differently, and how to choose them without guessing twice.

Two numbers with different jobs

Every container can declare both, and they mean genuinely different things:

yaml

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 500m
    memory: 256Mi

requests

A reservation. Decides placement.

What the scheduler uses to decide whether a pod fits on a node. Once placed, the amount is held for it whether or not it is used.

Get this wrong and the pod sits in Pending.

limits

A ceiling. Enforced at run time.

Enforced by the kernel on the node. The container cannot exceed it, and the scheduler never looks at it.

Get the memory one wrong and the pod is OOMKilled.

Different jobs, different enforcers, different symptoms when you get one wrong

How the scheduler uses requests

The scheduler's arithmetic is simpler than people expect, and it is the key to Pending.

For each node, it adds up the requests of every pod already assigned there. If the node's remaining allocatable capacity is at least this pod's request, the node is a candidate. Otherwise it is rejected.

Actual usage is not part of this calculation at all. A node whose pods request 90% of its CPU but use 5% is 90% full to the scheduler. This is exactly why a cluster with visibly idle machines refuses to place a pod.

bash

kubectl describe node learning-worker

text

Allocatable:
  cpu:                8
  memory:             16062440Ki

Allocated resources:
  Resource   Requests      Limits
  cpu        7200m (90%)   12 (150%)
  memory     14Gi (89%)    20Gi (127%)

Two things to read there. Requests above about 90% mean this node is effectively full. And limits exceeding 100% is normal and expected — that is overcommitment, and it is the point: containers rarely peak together, so allowing the sum of ceilings to exceed the machine gets you much better utilisation.

Note also that Allocatable is less than the machine's capacity. The kubelet reserves some for the operating system and for itself.

Diagnosing Pending

describe on the pod ends with the scheduler's own explanation, node by node:

bash

kubectl describe pod api-6c9f7d4b58-mn4pq

text

Events:
  Warning  FailedScheduling  0/3 nodes are available: 1 node(s) had
           untolerated taint {node-role.kubernetes.io/control-plane},
           2 Insufficient cpu. preemption: 0/3 nodes are available.

That message is a list of reasons with counts, and it is precise. Here: one node is the control plane, which normally carries a taint that ordinary pods do not tolerate, and the two workers do not have enough unreserved CPU.

The reasons you will actually see:

Reason	Meaning
`Insufficient cpu` / `memory`	Requests exceed what is unreserved
`untolerated taint`	The node is marked as not for this pod
`node(s) didn't match node selector`	A `nodeSelector` matches nothing
`didn't find available persistent volumes`	A PVC cannot be satisfied
`too many pods`	The node's pod-count cap is reached

The fixes are correspondingly direct: lower the request, add a node, or remove the constraint you did not mean to add.

bash

kubectl top nodes                   # actual usage (needs metrics-server)
kubectl top pods
kubectl describe node <name>        # requests vs allocatable

kubectl top needs the metrics server installed, and it is worth having: the gap between top and describe — real usage versus reserved — is the whole story of a badly tuned cluster.

Why CPU and memory limits behave differently

This is the part that surprises everyone, and it explains OOMKilled entirely.

CPU is compressible

Exceeding the limit throttles you.

The kernel gives the container fewer slices of time. It gets slower.

Nothing is killed, no error appears, and kubectl get pods says everything is healthy. The only symptom is latency — which is why many teams deliberately set no CPU limit at all.

Memory is not

Exceeding the limit kills you.

There is no way to give a process slightly less memory than it just asked for, so the kernel's OOM killer terminates the container and the kubelet restarts it. Exit code 137.

Loud, unambiguous, and never a bug in Kubernetes.

One of these announces itself. The other does not.

bash

kubectl get pod api-6c9f7d4b58-mn4pq

text

NAME                   READY   STATUS      RESTARTS      AGE
api-6c9f7d4b58-mn4pq   0/1     OOMKilled   4 (30s ago)   12m

bash

kubectl describe pod api-6c9f7d4b58-mn4pq | grep -A3 'Last State'

text

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137

OOMKilled is never a bug in Kubernetes and never ambiguous: the container asked for more memory than its limit allowed. Either the limit is too low, or the application has a leak. kubectl top pod over time distinguishes them — a leak climbs steadily and a genuine requirement plateaus.

Quality of service, and who dies first

The relationship between your requests and limits puts the pod into one of three classes, and the class decides who dies when a node runs out of memory.

Guaranteed — evicted last
Every container has requests equal to limits. Given a dedicated allocation. Worth it for the workloads you cannot afford to lose.
Burstable — evicted second
Requests set, and lower than limits. The common case, and the right default for most applications.
BestEffort — evicted first
Nothing set at all. Placed anywhere, because the scheduler thinks it needs nothing, and the first thing thrown off a node under pressure.

The kubelet evicts from the bottom up. The width is roughly how many of your pods should be in each class.

In YAML, the three look like this:

yaml

# Guaranteed
resources:
  requests: { cpu: 500m, memory: 512Mi }
  limits: { cpu: 500m, memory: 512Mi }

# Burstable
resources:
  requests: { cpu: 100m, memory: 128Mi }
  limits: { memory: 512Mi }

# BestEffort — no resources block at all

bash

kubectl get pod api-6c9f7d4b58-mn4pq -o \
  jsonpath='{.status.qosClass}'

When a node is under memory pressure the kubelet evicts BestEffort pods first, then Burstable pods that are exceeding their requests, and Guaranteed pods last. Setting resources is therefore not just tidiness — it is what keeps your important workloads on the node.

The mistake: nothing set at all

Bad — no resources declared:

yaml

containers:
  - name: api
    image: notes-api:0.1.0

Good — a request the scheduler can use, and a memory ceiling:

yaml

containers:
  - name: api
    image: notes-api:0.1.0
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
      limits:
        memory: 512Mi

With nothing declared, the scheduler believes the pod needs nothing, so it will pack any number of them onto one node. The pods then compete for real CPU and memory that was never accounted for, one memory spike takes the node into pressure, and the kubelet starts evicting — beginning with exactly these pods, because BestEffort goes first. A single leaking pod can destabilise every workload on the machine.

Note what the good version does not set: a CPU limit. Requests give the scheduler its arithmetic and provide a proportional share under contention; a CPU limit only adds throttling. A memory limit, by contrast, is worth having — it turns an unbounded leak into one restarted pod instead of a dead node.

Choosing the numbers

You cannot reason your way to these values; you measure them. The process is short:

Deploy with a generous guess — a request at roughly your expectation and a limit at two or three times it.
Run realistic load for long enough to see a steady state.
Read the actual usage:

bash

kubectl top pods --containers

Set the request near the steady-state average, and the memory limit above the observed peak with headroom.
Revisit when the application changes.

Two extras that automate the loop, once the basics are in place. The VerticalPodAutoscaler observes usage and recommends — or applies — requests. And a LimitRange in a namespace supplies defaults so a pod with no resources declared does not become BestEffort by accident:

yaml

apiVersion: v1
kind: LimitRange
metadata:
  name: defaults
spec:
  limits:
    - type: Container
      default:
        memory: 512Mi
      defaultRequest:
        cpu: 100m
        memory: 128Mi

A ResourceQuota is the matching cap on a whole namespace, which is how a shared cluster stops one team consuming it:

yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
spec:
  hard:
    requests.cpu: '10'
    requests.memory: 20Gi
    limits.memory: 40Gi

With a quota in place, a pod with no requests is rejected rather than admitted as BestEffort — which is often the behaviour you wanted all along.

Steering pods to particular nodes

Resources are how the scheduler chooses; three mechanisms let you constrain the choice. Enough to recognise them:

yaml

# Simplest: only nodes with this label
nodeSelector:
  disktype: ssd

yaml

# Spread replicas across nodes rather than stacking them
topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: api

yaml

# Tolerate a node that is marked as reserved
tolerations:
  - key: workload
    operator: Equal
    value: gpu
    effect: NoSchedule

Taints mark a node as unsuitable, and tolerations let specific pods ignore the mark — which is why the control-plane node in your kind cluster runs no workloads of yours.

The one worth reaching for early is topologySpreadConstraints. Three replicas on one node give you no protection from that node failing, and by default the scheduler is free to place them that way.

Cheat sheet

yaml

resources:
  requests: # a RESERVATION: the scheduler's arithmetic
    cpu: 100m #   100m = 0.1 core
    memory: 128Mi #   Mi/Gi are binary; M/G are decimal
  limits: # a CEILING: enforced by the kernel
    memory: 512Mi #   exceeded -> OOMKilled (exit 137)
    # cpu: 500m   #   exceeded -> throttled, silently. Often omit.

bash

# Why is this pod Pending?
kubectl describe pod <name>          # FailedScheduling, node by node
kubectl describe node <name>         # requests vs allocatable
#   Insufficient cpu/memory  -> lower the request, or add a node
#   untolerated taint        -> the node is reserved
#   didn't match selector    -> a nodeSelector matches nothing

# Why does this pod keep dying?
kubectl get pod <name>               # STATUS OOMKilled, RESTARTS rising
kubectl describe pod <name> | grep -A3 'Last State'
#   OOMKilled, 137 -> memory limit too low, or a leak

# Actual usage (needs metrics-server)
kubectl top nodes
kubectl top pods --containers
#   the gap between top and describe is reserved-but-unused

# CPU vs memory, the asymmetry that explains everything
#   over the CPU limit    -> throttled: slow, silent, no restart
#   over the memory limit -> killed: OOMKilled, restarted, exit 137

# QoS class, which decides eviction order
kubectl get pod <name> -o jsonpath='{.status.qosClass}'
#   Guaranteed  requests == limits          evicted last
#   Burstable   requests set, limits higher the usual case
#   BestEffort  nothing set                 evicted FIRST

# A defensible default
#   requests: always (CPU and memory)
#   memory limit: always, above the observed peak
#   CPU limit: only with a specific reason

# Namespace-level guardrails
#   LimitRange     defaults for pods that declare nothing
#   ResourceQuota  a cap on the namespace; rejects pods with no requests

# Placement
#   nodeSelector                only nodes with a label
#   taints + tolerations        reserve nodes for specific pods
#   topologySpreadConstraints   spread replicas across nodes

Where to go next

The scheduler now has what it needs, and your pods have bounds. The next lesson is the one all of this has been building towards: shipping a new version without dropping a request — how a rolling update actually works, what maxSurge and maxUnavailable do, and how to undo one.

Before that, run kubectl top pods against something you have deployed and compare it with what you requested. The first time you see how far off a guess was is what turns this lesson from advice into habit.