Service Discovery

Inside a monolith, one part of the system calls another with a function call. The address question never comes up, because there is no address. The callee is in the same process, at a location the compiler worked out before the program ever ran.

Split that system apart, as Microservices — and When to Split argues you sometimes should, and every one of those function calls becomes a network call. Now there is an address, and the address is the problem. The instance you want is on some host, on some port, and both of those were handed out minutes ago by a scheduler that will hand out different ones tomorrow.

The first instinct is to write the addresses down. A config file, a list of hosts, an environment variable. That works exactly until the first deploy replaces the instances, or the autoscaler adds four, or a host dies at three in the morning and comes back somewhere else. Anything you write down about location starts going stale the moment you write it.

So this lesson answers one question, and it is a smaller question than it looks: who is alive, and where? Getting an answer is the easy half. The hard half, the half this lesson really cares about, is that whoever answers is guessing, and the ways they guess wrong will take your system down in two completely different directions.

The mechanism is unglamorous. You run a service registry: a database of services, their instances, and their locations. An instance registers itself when it starts and deregisters when it shuts down. Callers ask the registry instead of asking a config file.

There are two ways an instance gets into that database.

Self-registration. The instance calls the registry on boot, says "I am checkout, I am at 10.0.9.2:8080", then sends a heartbeat every few seconds to say it is still there. On a clean shutdown it calls back to remove itself. Simple, and the service owns its own truth.

A registrar. A separate component watches instances start and stop and does the registering on their behalf. Your service code stays clean, with no registry client inside it, which matters a great deal when your services are written in four languages. The container platform usually plays this role without being asked.

Now the question that decides the design. What happens when an instance does not shut down cleanly, but simply dies? A process that is killed does not get to run its deregistration code. The entry stays in the registry, pointing at nothing, and every caller that reads it gets sent to a corpse.

This is why a registration is not a row you insert. A registration is a lease. It is granted for a short time and it expires unless the instance keeps renewing it. The machinery is the machinery you already met in Distributed Locks & Leases, and the primitives are the ones from ZooKeeper & etcd in Practice: an ephemeral entry tied to a session, plus a watch so interested parties hear about the change instead of polling for it. Death by silence is the only kind of death a registry can actually detect, which is the same uncomfortable fact Heartbeats & Failure Detection made you sit with.

One question hangs over all of this, and it deserves a straight answer before we move on: if callers find services through the registry, how do they find the registry? Is that not the same problem again? It is, and the answer is deliberately boring. The registry runs as a small, fixed fleet at stable, well-known addresses, handed to every service through configuration or a plain DNS name, and those addresses almost never change. You have not eliminated the moving-address problem. You have cornered it into one place small enough, and stable enough, that writing an address down is finally safe.

With a registry in place, there are two places the lookup can happen, and the choice shapes everything downstream.

Client-side discovery. The caller asks the registry which instances of checkout exist, gets a list back, and picks one itself. Because it holds the whole list, it is also doing the load balancing. The classic pairing is a registry with a client library beside it, so a call to a logical name like http://checkout/orders is resolved at the moment of the call. You get fewer moving parts and one less network hop. You pay by coupling every caller to the registry, and by writing that discovery client again for every language and framework you run.

Server-side discovery. The caller does something much dumber: it sends the request to one stable address and forgets about the problem. A router sitting at that address consults the registry and forwards the request to a real instance. Caller code gets simpler, and it stays simple in every language, because there is nothing to implement. You pay with an extra hop, and with a router that must speak your protocol and must never be down.

If that router sounds familiar, it should. Server-side discovery is Load Balancers: The Traffic Directors with one upgrade: the backend list is no longer typed in by a human, it is fed from the registry and changes by itself. Most platforms hand you this shape by default, which is why many teams use service discovery for years without ever naming it.

Every registry stores the same field, and the whole system leans on it: healthy. It is worth asking who fills that field in, and how they could be wrong, because the answer is not one failure mode. It is two, and they push in opposite directions.

Health Checks & Failover introduced the probe. Production sorts probes into three tiers, and the tiers are not just degrees of thoroughness, they have different blast radii.

Liveness checks test basic connectivity and the presence of a server process. They are usually run by the load balancer or an external agent and know nothing about how your application works. They are nearly free and nearly blind.

Local health checks go further, and verify the application is likely to be able to function. The defining property is what they are allowed to touch: resources this server owns alone, like its own disk or its own worker process. Because nothing is shared, they are unlikely to fail on many servers at the same time.

Dependency health checks are a thorough inspection of whether the application can talk to its adjacent systems, and this is where the trouble lives. They catch problems that are genuinely local, such as this one server holding expired credentials. They also produce false positives whenever the dependency itself is having a moment.

Now put the two failure directions side by side.

Check too shallow and you get a zombie: a process that answers the probe, sits happily in the registry, and cannot serve a single real request. Every caller routed to it fails.

Check too deep and you get something far worse. A shared database gets slow. Every instance runs its dependency check, every instance fails it at the same instant, every instance is pulled from the registry, and the registry now says that zero instances of your service exist. The database was merely slow. You made it an outage.

This is the part that is hard to believe until you watch it, so drive it yourself.

Choose how deeply the checks probe. Then break something: wedge one instance, expire one instance's credentials, or wobble the database that every instance depends on. Watch what reaches the registry, and watch what the callers get back.

The pattern to look for is the one that feels wrong: the most thorough setting is the one that turns a partial problem into a total one. Then arm the fail-open policy and run the same break again.

The fix is a single instinct, and once you have it you will see it in three different layers of your stack.

The rule, as the teams who learned it the expensive way state it: automation should stop directing traffic to a single bad server, but keep allowing traffic if the entire fleet appears to be having trouble.

Read that again, because it is deliberately asymmetric. One server says it is sick: believe it, evict it, you have plenty more. Every server says it is sick at the same instant: disbelieve the evidence. Six servers do not fail simultaneously for six independent reasons. Something they share broke, or something watching them broke, and in both cases pulling them all out of service helps nobody.

At the router this is called failing open. When one server fails its check the load balancer stops sending it traffic, but when all of them fail at once the balancer keeps sending traffic to all of them, on the theory that a fleet that is degraded is still worth more than a fleet that is unreachable.

At the registry the same instinct has a different name. Netflix's Eureka registry ships it as self-preservation mode: when too many instances stop heartbeating too quickly, Eureka refuses to expire any of them, holding the last known list instead of emptying itself. It looks like stubbornness. It is the same judgment: a sudden mass disappearance is far more likely to be a network problem than an extinction event.

Fail-open and that refusal to expire are the same idea wearing two uniforms. They are both a system declining to act on evidence that is too catastrophic to trust. That is the most portable thing in this lesson, and it is worth carrying into every piece of automation you build that is allowed to remove things from service.

It also comes with an honest bill. A registry that refuses to expire entries is a registry that hands out addresses of the dead. Which is only survivable because of what the caller already owns.

Here is the mental model to leave with, and it is the one most treatments skip. The registry is not a source of truth. It is a cache of a fact that is already changing.

There is always a window. An instance dies; the registry does not know yet. An instance starts; callers have not refreshed yet. You can shrink the window with shorter leases and faster propagation, and you can never close it, because the information has to travel and the world does not pause while it does.

The window is bigger than intuition says. Add up the defaults in a classic Eureka stack: a heartbeat every 30 seconds, eviction only after 90 seconds of silence, and a client that refreshes its cached copy of the registry every 30 seconds. A caller can keep dialling an instance that died two minutes ago, with every component working exactly as configured. Nothing is broken. That is simply what the defaults add up to.

Accepting that changes what you optimise for. It is tempting to want a strongly consistent registry, one that never lies. But consider what a strongly consistent registry does during a network partition: the callers stuck on the minority side cannot reach a quorum, so they lose their discovery mechanism entirely. They go blind, while the instances they wanted are sitting right beside them, perfectly healthy.

The teams who ran into this concluded something that sounds like heresy and is not: for service discovery, it is better to have information that may contain falsehoods than to have no information at all. Knowing which servers existed five minutes ago beats having no idea what the world looks like because a link flapped. Discovery is one of the few places in this course where availability genuinely should win over consistency.

That trade only works because a wrong address is a survivable event and an absent address is not. A caller handed a dead instance has a timeout, a retry with backoff, and somewhere to send the traffic instead, exactly as Calling Services Safely: Timeouts, Retries & Backoff set out, and it has a breaker to stop hammering the corpse, as Circuit Breakers insisted. A caller handed nothing at all has no move to make.

One consequence worth stating plainly, because it shows up in reviews: the registry belongs to the control plane, the machinery that decides where traffic should go, while the requests themselves ride the data plane, the path traffic actually flows through. If your registry going down stops traffic that is already flowing, you have wired those two planes together. Callers should keep serving from their last known good list, and a registry outage should be a period where the system stops learning about change, not a period where it stops working. That last-known-good list is also the third home of the distrust-mass-death instinct: a caller refusing to forget every address it knows just because the registry went silent is making exactly the same bet as the balancer that fails open.

You will rarely build any of this. You will inherit it, so it pays to recognise the shapes.

The container platform. You declare a service with a stable name; a controller watches which instances currently match and are ready, and keeps a list of their addresses; a proxy on every host turns that list into routing rules, and a cluster DNS server answers the name. That is server-side discovery with the registry hidden inside the platform. There is also an opt-out worth knowing, the headless service: ask for the service without a stable virtual address, and DNS returns one record per instance instead of one for the service. The platform has just handed you back client-side discovery, which is what you want when you need to address a specific instance rather than any instance. The object model behind this belongs to Kubernetes: The 20% That Matters.

The agent-based registries. Consul is the canonical one: an agent on every node gossips with the others to track which nodes are alive, exactly the membership machinery from Gossip Protocols, while a separate consensus-backed catalog holds the authoritative service list. It is a deliberate split: gossip is fast and survives partitions, so it decides liveness; consensus is careful, so it holds the catalog. Discovery does not have to choose between availability and consistency once for everything it does; Consul chooses availability for the fast-changing fact and consistency for the slow-changing one.

Plain DNS, the oldest answer and the one that traps people. It is genuinely a directory, and DNS: The Internet's Phonebook covered how resolution works. As a discovery mechanism it has two gaps. It carries no health information, so a name resolves whether or not anything behind it can serve. And the freshness knob is not yours: every resolver and every runtime in the path is free to cache. The notorious case is the JVM, whose default policy is to cache a successful lookup forever, so an instance moves and the process keeps dialling the old address until someone restarts it. It has a second sting: the setting that controls it is a security property, not an ordinary system property, so the obvious command-line override quietly does nothing. Teams ship the fix, confirm the flag is present, and stay broken.

The fourth shape, where a sidecar beside each instance receives the list and does the routing, is the subject of Sidecar & Service Mesh.

• The address is not a fact, it is a rumour with a timestamp. Anything you write down about location begins going stale immediately.
• A registration is a lease, not a row. Crashed instances never deregister themselves, so entries must expire on their own or a registrar must remove them.
• Two places to look up. Client-side gives you fewer hops and one discovery client per language. Server-side gives you trivial callers and a router you must keep alive.
• "Healthy" is a judgment with two failure directions. Too shallow routes traffic to a zombie. Too deep lets one shared wobble condemn the entire fleet at once.
• Distrust mass death. Evict one sick server; refuse to evict all of them. Failing open and refusing to expire leases are the same instinct, and it belongs in any automation allowed to remove things from service.
• Stale beats empty. A wrong address is survivable because the caller owns timeouts, retries and a breaker. No address at all leaves it with no move.
• Keep the planes apart. A registry outage should stop the system learning about change, not stop it serving traffic.

You can now find a service. The next question is what sits at the edge of all this and takes the request from the outside world in the first place, which is where API Gateway & BFF picks up.