beginner30 minby DevFox

What a Cluster Is Made Of

Control plane, nodes, kubelet, scheduler and etcd, explained as one loop: you write down what you want, and something keeps reality matching it.

Kubernetes

Kubernetes is often drawn as a single box labelled "the cluster", which is fine until something goes wrong. Then the error mentions the scheduler, or a node reports NotReady, or kubectl hangs with no explanation — and knowing which piece said what is the difference between a diagnosis and a guess.

This is the one theory lesson in the course. By the end of it you will know what a cluster physically consists of, which component makes which decision, and where the reconciliation loop from the last lesson actually runs. It is short, and it pays for itself in every lesson afterwards.

Two kinds of machine

A cluster is a set of machines with two roles.

The control plane

The brain. Does not run your application.

Stores what you want, decides where things should go, and runs the controllers that keep reality in line.

In production it is usually three machines for redundancy — or a managed service that runs it for you.

The nodes

The muscle. This is where containers run.

Each is a machine, physical or virtual. Nodes do as they are told and report back what happened.

These are the ones you supply and pay for.

On a laptop cluster both roles live on one machine. Everywhere else they do not.

text

        ┌──────────────── control plane ────────────────┐
        │  api server   scheduler   controller manager  │
        │                    etcd                       │
        └───────────────────────┬───────────────────────┘
                                │
        ┌───────────────┬───────┴───────┬───────────────┐
        │    node 1     │    node 2     │    node 3     │
        │ kubelet       │ kubelet       │ kubelet       │
        │ kube-proxy    │ kube-proxy    │ kube-proxy    │
        │ [pod] [pod]   │ [pod]         │ [pod] [pod]   │
        └───────────────┴───────────────┴───────────────┘

On a laptop cluster, both roles live on one machine. In production the control plane is usually three machines for redundancy, and a managed service — GKE, EKS, AKS — runs it for you so that the only part you see is the nodes.

The control plane, component by component

Four pieces, each with one job. They are worth learning by name because they appear in error messages and logs constantly.

The API server

Everything goes through the API server. kubectl talks to it. The nodes talk to it. The controllers talk to it. It is the only component that talks to the datastore, and it is the only door into the cluster.

Its job is to accept a change, authenticate whoever asked, authorise the action, validate the object, and store it. Once stored, the change is official — and everything that happens next is some other component noticing.

This is why kubectl hanging or refusing to connect is a categorically different problem from a pod that will not start. One means you cannot reach the door; the other means the cluster is working and disagreeing with you.

etcd

etcd is the database. It is a distributed key-value store, and it holds the entire state of the cluster: every object you have created, plus the status the components report back.

Two consequences. First, etcd is the thing to back up — with it you can rebuild a cluster, and without it there is nothing to rebuild from. Second, nothing except the API server should ever touch it directly.

The scheduler

The scheduler answers exactly one question: this pod has no node yet — which node should it run on?

It filters the nodes that could take the pod, then scores those that remain and picks the best. Filtering considers whether the node has enough unreserved CPU and memory, whether it matches any node selectors or affinity rules the pod asked for, and whether the pod tolerates the node's taints.

Then it writes its answer back through the API server, and its involvement is over. The scheduler never starts a container.

The controller manager

The controller manager runs the controllers — and the controllers are where the reconciliation loop from the last lesson physically lives.

Each controller watches one kind of object and works to make reality match it. The Deployment controller notices a Deployment wants three pods and only two exist, and creates one. The node controller notices a node has stopped reporting and marks it NotReady, then evicts its pods so they can be rescheduled. There are dozens, each running the same loop:

That is the entire engine. Everything Kubernetes appears to "do by itself" is one of these loops noticing a difference.

What runs on every node

Two components plus a container runtime, on each node.

The kubelet

The kubelet is the agent on the node, and it is the component that actually makes containers exist. It asks the API server which pods are assigned to its node, tells the container runtime to start them, watches them, runs their health checks, restarts them when they fail, and reports status back.

The kubelet is also what makes a node appear healthy. It sends a heartbeat; when the heartbeats stop, the node controller concludes the node is gone. A node showing NotReady while the machine is plainly powered on is usually a kubelet problem, not a hardware one.

kube-proxy

kube-proxy implements Services on the node. When a Service gives a group of pods one stable address, something has to make traffic to that address reach one of those pods — and on each node, this is it, programming the kernel's networking rules to do the forwarding. It comes up again in the Services lesson.

The container runtime

The kubelet does not run containers itself; it asks a container runtime to. In practice that is containerd, which is the same component Docker uses underneath — which is precisely why the image you built with Docker runs here unchanged.

One request, end to end

Now put the components in motion. You run this:

kubectl sends the file to the API server
That is the whole of kubectl's involvement.
The API server validates it and writes to etcd
Authenticates you, checks you are allowed, validates the object, stores it. Your command returns here — deployment.apps/api created.
Controllers notice, and create objects
The Deployment controller sees a Deployment wanting three pods and creates a ReplicaSet. The ReplicaSet controller sees three wanted and none existing, and creates three pod objects — with no node assigned.
The scheduler assigns each pod a node
It picks, records the choice through the API server, and its involvement is over. It never starts anything.
The kubelet starts the containers
On each chosen node: notices a pod assigned to it, pulls the image, asks containerd to run it, then reports status back.
kube-proxy updates the network rules
On every node, so a Service address reaches the new pods.

Six components, and not one of them spoke to another directly

Two things are worth extracting from that.

Your command finished at step 2. kubectl apply returning successfully means the desired state was accepted, not that anything is running. This is why kubectl get pods immediately afterwards often shows ContainerCreating, and why a broken image name produces a happy created message followed by a pod that never starts.

No component talked to another directly. Every step was a component watching the API server and reacting. That indirection is what makes the system resilient: a scheduler that restarts picks up where it left off, because the truth is in etcd rather than in anyone's memory.

Seeing it for yourself

Once you have a cluster — the next lesson — these commands show you the pieces:

The kube-system namespace is the interesting one. On most clusters the control plane components run as pods themselves, so you can see kube-apiserver, kube-scheduler, kube-controller-manager, etcd and a kube-proxy per node listed there like any other workload.

Cheat sheet

Where to go next

The next lesson gets a real cluster running on your own machine with kind or minikube, and gets kubectl talking to it — including the kubeconfig file, which is worth understanding well enough that you never apply a manifest to the wrong cluster.

After that the objects begin, starting with the pod. Keep the loop from this lesson in mind as they arrive: each new object is either something that declares desired state, or a controller that reconciles it.