Service Discovery Patterns#

In a monolith, calling another module is a function call. In microservices, it is a network call to a moving target. Services scale up, scale down, crash, and redeploy — their IP addresses and ports change constantly. Service discovery solves the question: where is the instance I need to talk to right now?

The Core Problem#

Without discovery, you hardcode addresses:

ORDER_SERVICE=http://10.0.1.5:3000
USER_SERVICE=http://10.0.1.6:3001

This breaks when:

• A service restarts on a different port
• Auto-scaling adds new instances
• A deployment rolls out to new IPs
• A node fails and workloads migrate

Service discovery automates the mapping from logical name to network location.

Client-Side Discovery#

The client queries a service registry directly and picks an instance.

1. Order Service asks registry: "Where is User Service?"
2. Registry returns: [10.0.1.6:3001, 10.0.2.8:3001, 10.0.3.2:3001]
3. Order Service picks one (round-robin, least connections, etc.)
4. Order Service calls 10.0.2.8:3001 directly

Advantages:

• No extra network hop (client talks directly to the target)
• Client can implement smart load balancing
• Lower latency

Disadvantages:

• Every client needs discovery logic
• Language-specific client libraries required
• Tighter coupling between client and registry

Examples: Netflix Eureka + Ribbon, gRPC client-side load balancing.

Server-Side Discovery#

The client calls a load balancer or router that handles discovery internally.

1. Order Service calls http://user-service/api/users
2. Load balancer queries registry for User Service instances
3. Load balancer routes to 10.0.2.8:3001
4. Response flows back through load balancer

Advantages:

• Clients stay simple (just call a DNS name)
• Language-agnostic — any HTTP client works
• Centralized load balancing logic

Disadvantages:

• Extra network hop through the load balancer
• Load balancer becomes a potential bottleneck
• Higher latency than client-side

Examples: AWS ALB, Kubernetes Services, NGINX with Consul.

Service Registry#

The registry is the source of truth for which instances are alive and where they live.

How Registration Works#

Self-registration: Services register themselves on startup and send heartbeats.

// On startup
await registry.register({
  name: "user-service",
  address: "10.0.2.8",
  port: 3001,
  healthCheck: "/health"
});

// Every 10 seconds
await registry.heartbeat("user-service", instanceId);

// On shutdown
await registry.deregister("user-service", instanceId);

Third-party registration: A separate process (registrar) watches for new instances and registers them.

Platform (K8s, ECS) starts container
  -> Registrar detects new container
  -> Registrar adds entry to registry
  -> Registrar removes entry when container stops

Popular Registries#

Consul Example#

service {
  name = "user-service"
  port = 3001
  tags = ["v2", "primary"]

  check {
    http     = "http://localhost:3001/health"
    interval = "10s"
    timeout  = "2s"
  }
}

Query healthy instances:

GET /v1/health/service/user-service?passing=true

etcd with Leases#

# Create a lease (TTL 30s)
etcdctl lease grant 30
# lease 694d7550e3b3d101 granted with TTL(30s)

# Register with lease
etcdctl put /services/user-service/instance-1 \
  '{"address":"10.0.2.8","port":3001}' \
  --lease=694d7550e3b3d101

# Keep alive (renew before TTL expires)
etcdctl lease keep-alive 694d7550e3b3d101

If the service crashes, the lease expires and the key is automatically deleted.

DNS-Based Discovery#

The simplest form: use DNS to resolve service names to IPs.

dig user-service.internal A
;; ANSWER SECTION:
user-service.internal. 30 IN A 10.0.2.8
user-service.internal. 30 IN A 10.0.3.2

SRV records add port information:

_user-service._tcp.internal. 30 IN SRV 0 50 3001 10.0.2.8.
_user-service._tcp.internal. 30 IN SRV 0 50 3001 10.0.3.2.

Advantages: Universal — every language has a DNS client.

Limitations:

• DNS caching causes stale entries (TTL trade-off)
• No built-in health checking
• Limited metadata (no tags, versions, weights)

Tools: CoreDNS, AWS Route 53, Consul DNS interface.

Health Checking#

Discovery without health checking sends traffic to dead instances.

Levels of Health Checks#

Implementation Pattern#

app.get("/health", (req, res) => {
  res.status(200).json({ status: "ok" });
});

app.get("/ready", async (req, res) => {
  const dbOk = await db.ping().catch(() => false);
  const cacheOk = await redis.ping().catch(() => false);

  if (dbOk && cacheOk) {
    res.status(200).json({ status: "ready", db: true, cache: true });
  } else {
    res.status(503).json({ status: "not ready", db: dbOk, cache: cacheOk });
  }
});

Important: Liveness checks should be lightweight. Heavy liveness checks can cause cascading restarts.

Service Mesh Discovery#

A service mesh (Istio, Linkerd, Consul Connect) moves discovery into the infrastructure layer.

[Order Service] -> [Sidecar Proxy] -> [Sidecar Proxy] -> [User Service]

The sidecar proxy handles:

• Service discovery (where to route)
• Load balancing (which instance)
• Health checking (remove unhealthy)
• mTLS (encrypt in transit)
• Retries and circuit breaking

Advantage: Application code has zero discovery logic. Just call http://user-service:3001.

Cost: Extra memory and CPU per pod, operational complexity of the mesh.

Kubernetes DNS#

Kubernetes has built-in service discovery via DNS.

ClusterIP Services#

apiVersion: v1
kind: Service
metadata:
  name: user-service
  namespace: production
spec:
  selector:
    app: user-service
  ports:
    - port: 3001

Every pod can now resolve:

user-service.production.svc.cluster.local -> 10.96.0.15 (ClusterIP)

Kubernetes handles load balancing across healthy pods via iptables/IPVS rules.

Headless Services (Direct Pod Discovery)#

spec:
  clusterIP: None  # headless
  selector:
    app: user-service

DNS returns individual pod IPs — useful for stateful workloads (databases, caches) where clients need to reach specific pods.

External Services#

kind: Service
spec:
  type: ExternalName
  externalName: db.example.com

Map external dependencies into the Kubernetes DNS namespace for uniform discovery.

Choosing the Right Pattern#

Key Takeaways#

• Service discovery maps logical names to live network addresses
• Client-side discovery is faster; server-side discovery is simpler
• Health checking is non-negotiable — never route to dead instances
• Kubernetes DNS solves discovery for most containerized workloads
• Service meshes push discovery into infrastructure, keeping app code clean
• Start with the simplest option that meets your needs — do not over-engineer

---

Article #248 in the System Design series. Keep building: codelit.io/blog.