Kubernetes: The 20% That Matters

Kubernetes has the worst reputation of any tool in this course: a thousand-page surface, its own vocabulary, certifications, conference circuits, and a meme economy about how nobody understands it. The reputation is earned, and it is also beside the point.

Here is the bet this lesson makes: for designing systems, which is what this course trains, you need about a fifth of Kubernetes, and that fifth is learnable in one sitting. You need one idea, the reconciliation loop, three objects, the pod, the Deployment, and the Service, and one map of the machinery so its failures do not surprise you. With those you can read an architecture diagram, size a fleet, predict what happens when a node dies at 3 a.m., and hold your own in any design review that says the word 'cluster'. The rest is operational depth you can acquire when a job demands it, and the second-to-last section of this lesson names exactly what was left out so nothing feels hidden.

The previous lesson left you with the problem this tool exists for: hundreds of containers per host, thousands per fleet, starting, dying, and moving. Somebody has to place them, heal them, and roll new versions through them without dropping requests. The surprise is how Kubernetes does it, because the answer is not a smarter deploy script.

Every tool you have used before this one is imperative: you tell it what to do. Deploy this. Restart that. Kubernetes inverts the sentence. You tell it what should be true, 'three replicas of image v2, each needing half a CPU', and you write that wish into the cluster. Then a set of programs called controllers spend their whole lives running one loop: observe what actually exists, compare it with what you declared, act on the difference. Not on a timer, either: controllers hold watches on the state store and are notified the instant something drifts.

Hold that loop in your head and the famous features stop being features. Self-healing? A pod dies; observed says two, desired says three; the controller starts a third. Nobody restarted anything, no alarm fired, no runbook ran: a difference existed and was converged away. Rollouts? You edit desired from v1 to v2; the controller walks the fleet, retiring old pods and starting new ones a few at a time. Rollback? Edit desired back; same loop, opposite direction. Scaling? You changed one integer.

The design consequence is bigger than the convenience. Because everything is an edit to desired state, every actor speaks the same language: a human with a config change, a CI pipeline shipping a version, and, in the next lesson, an autoscaler adjusting the replica count are all just writers to the same declaration, and the loop makes whatever they wrote come true. Operations stops being a sequence of commands that can be half-executed, and becomes a document that is either converged or not yet converged. That inversion is the 20%'s first and largest piece.

Three objects carry nearly every design conversation.

The pod is the atom: one or more containers scheduled together, sharing a network namespace, so they talk over localhost, and optionally a volume. Why is the atom not simply a container? Because you have already met the reason: the sidecar. The mesh pattern from the previous course section needed a proxy beside the application, same host, same network view, and the pod is the unit that gives the sidecar its seat. Pods are cattle, not pets: when one misbehaves it is replaced, never repaired, and everything about the platform assumes a pod can vanish at any moment.

The Deployment is the object you actually edit. It owns a template for pods plus the number wanted, and it manages the versioned pod sets (each called a ReplicaSet) that reconciliation converges. When you change the image, the Deployment performs the rollout, surging a few new pods, waiting for them to become ready, retiring old ones, and when you say rollback, it is the same walk backward. You will almost never create a bare pod, for the same reason you rarely allocate raw memory: the managed version is the point.

The Service will feel familiar, because it is the Service Discovery lesson productized. Registration: pods carry labels, and a Service selects by label, so membership is automatic, no client library, no registration call. Health: only pods passing their readiness probe receive traffic. Lookup: the Service owns a stable virtual IP and DNS name that outlives every pod behind it. Every duty that lesson made you build by hand, the platform now performs as a built-in, which is the quiet pattern of this whole section: the previous course section's patterns keep reappearing as product features.

North-south traffic gets one more object, the Ingress (and its successor, the Gateway API): a standard socket where the API Gateway & BFF layer plugs into the cluster. Config gets ConfigMaps and Secrets, the externalized-configuration pattern with a native mount point. The ladder ends there for design purposes.

You can design with the ladder alone, but failures make sense only if you know the six moving parts.

The API server is the single front door: every read and write, from every human, controller, and node agent, flows through it. Behind it sits etcd, the store where desired and observed state live, and etcd should ring a bell: it is the exact tool you drove in Codelab: Leader Election with etcd, a consensus store running Raft, every write committed only when a majority agrees, Raft, Visually's machinery holding up the world's clusters. The scheduler does one job: watch for pods that have no node yet, and pick one, packing by the resources each pod requests. The controller manager runs the reconciliation loops. And on every worker node, a kubelet starts the pods assigned to it, runs their probes, and reports observed state back, while kube-proxy makes Service addresses actually route on that node.

Now the question this course has trained you to ask three times already: what happens when the brain dies? You already know the answer, because it is the law you learned at the registry, again at the mesh, and again at the config store: a dead control plane stops change, never traffic. Running pods keep running, kubelets keep them alive locally, kube-proxy keeps routing on its last rules. What stops is convergence: no new pods, no healing, no rollouts, no edits. Kubernetes is the fourth system in this course built on that law, at the biggest scale yet, and recognizing it should now feel like meeting an old friend in a new uniform.

Three small mechanisms complete the design kit, and two of them hide famous traps.

Probes are how the platform judges health, and there are two verdicts with very different consequences. A failing liveness probe means 'this container is wedged, restart it', and the kubelet kills and restarts it. A failing readiness probe means 'do not route to me right now', and the Service quietly drops the pod from its set until it recovers. The trap is wiring liveness to a dependency: a liveness probe that pings the database turns a thirty-second database blip into the kubelet restarting every pod in the fleet, a restart storm you inflicted on yourself. You met this disease at the discovery lesson as the too-deep health check; the rule here is the same. Liveness checks only the process itself. Readiness may check dependencies, because its failure mode is polite: traffic sheds, nothing dies.

Requests and limits are the pod's appetite paperwork, and they are not new machinery: they are the cgroup rations from the previous lesson with a scheduling role. The request is what the scheduler reserves when placing the pod, the bin-packing input. The limit is the cgroup ceiling at runtime. Set requests too low and nodes overpack, and the noisy-neighbor fights return; set limits too low and your own pod gets throttled or killed at its ceiling. For design purposes: requests decide how many nodes you need, limits decide how gracefully you degrade.

Placement is the scheduler packing pods onto nodes by requests, about a hundred pods per node by default, clusters into the thousands of nodes. The details, affinity rules, taints, spread constraints, are 80%-territory; the design-level fact is that placement is automatic, request-driven, and re-done from scratch every time a pod is replaced.

Below is a small cluster: two nodes, a Deployment, a Service routing live requests to whatever is ready. You hold the desired state: type a replica count, pick an image version. Then be the chaos. Kill a pod and watch the loop notice the drift and heal it without you. Kill a whole node and watch the survivors inherit its work. Ship the broken v3 and see why readiness gates save the fleet: the rollout stalls at the first sick pod while the old version keeps serving. And take the control plane down mid-mess to prove the law for yourself: traffic keeps flowing, and nothing heals until the brain returns.

The sentence to carry out: you never once told the cluster what to do. You edited what should be true, broke reality, and watched the loop repair the difference, except when the brain was down, when reality politely froze.

Three failure walks cover most of what production will show you.

A node dies. The kubelet's heartbeats stop; the node controller marks the node gone; every pod it carried is now drift, and the loop re-schedules them onto surviving nodes. Total time: tens of seconds to minutes, tunable. Notice what did not happen: no pager fired for the pods themselves, because pods were designed to be lost. Capacity is the real question, which is why requests, and the headroom your node count leaves, decide whether a node death is a non-event or a cascade.

A bad version ships. The broken image starts, and its readiness probe never passes. The rollout surges one new pod, waits for ready, and simply never proceeds: the old ReplicaSet keeps serving every request while the new one sits stuck at one sick pod. This is the readiness gate doing its real job, and it is why 'a bad deploy took us down' is, on this platform, usually a story about a probe that lied.

The control plane goes away. Managed control planes fail too. Traffic flows on last-known rules; running pods live; but a pod that dies during the outage stays dead, a node loss goes unrepaired, and every write, from kubectl (the cluster CLI) or CI, is refused. The design consequence: control-plane outages convert your self-healing fleet into a static one, so the exposure is your drift rate, how much dies per hour, times the outage length. Fleets with headroom ride it out; fleets running hot discover that the brain was load-bearing after all.

A single-point-of-reference course owes you the list of what it skipped and the moment you would need each item, so here it is.

StatefulSets exist for pods that need stable identity and their own storage, databases, brokers; you reach for them the day you run stateful things in-cluster, and many teams deliberately never do. Operators and CRDs are the reconciliation loop opened to your own objects: you define a resource, write a controller, and the platform converges it, which is how databases-as-objects and the whole ecosystem work; you need the concept in design reviews, the authoring only if you build platforms. Helm and kustomize are packaging for manifests. RBAC, namespaces, network policies are the multi-team fences. Jobs and CronJobs run batch work. Each is a bounded topic you can learn in an afternoon when a real need names it, and none of them changes the mental model you now hold.

And the judgment call, because this course refuses to hand you a hammer without the 'not every thing is a nail' clause: Kubernetes earns its keep when fleets are large, teams are many, and churn is constant, the exact conditions of the previous section's patterns. A three-service system run by one team does not need a cluster; a VM, containers, and a process manager is honest engineering there, the same innovation-token arithmetic this course has used since the styles section. Whether you run Kubernetes yourself or rent it managed, and whether you should be on it at all versus simpler compute, is a real decision with real prices, and it gets its own lesson shortly: EC2 vs EKS vs Lambda: The Decision.

• One idea carries the platform: declare, don't command. Desired state in a store, controllers watching, observe-compare-act converging the difference. Self-healing, rollouts, rollback, and scaling are that one loop in four costumes.

• Three objects carry the design talk. The pod, the atom with a seat for the sidecar; the Deployment, the rollout machine you actually edit; the Service, the discovery lesson productized: labels register, readiness gates, DNS answers.

• Six parts, one law. API server in front, etcd underneath running Raft, scheduler placing, controllers converging, kubelet and kube-proxy on every node. And for the fourth time in this course: a dead control plane stops change, never traffic.

• Probes have two verdicts; respect the difference. Liveness restarts, so it checks only the process; readiness sheds routing, so it may check dependencies. Wiring liveness to a database is how teams turn a blip into a fleet-wide restart storm.

• Requests place, limits cap. The cgroup rations from last lesson, wearing scheduling clothes: requests size your fleet, limits shape your degradation.

• The 80% is acquirable on demand. StatefulSets, operators, Helm, RBAC: bounded topics with named triggers, none of which changes the model. And a small system still deserves a small answer.

One thread is left deliberately hanging. The loop converges on whatever replica count is declared, and today a human declared it. But traffic is not constant, and the platform can edit its own declaration: something can watch load and write a new number into the same desired state everyone else edits. Choosing what signal that watcher reads, and how fast it reacts, is a design problem with sharp edges, and it is next: Autoscaling Policies.