Network Protocols for System Design#

Every system design decision sits on top of network protocols. Choosing the wrong protocol means unnecessary latency, wasted bandwidth, or broken real-time features. Here's what matters.

TCP vs UDP#

The foundational choice:

TCP (Transmission Control Protocol):
  ✓ Reliable delivery (retransmits lost packets)
  ✓ Ordered delivery (packets arrive in sequence)
  ✓ Flow control and congestion control
  ✗ Higher latency (3-way handshake, retransmission delays)
  ✗ Head-of-line blocking

UDP (User Datagram Protocol):
  ✓ Low latency (no handshake, no retransmission)
  ✓ No head-of-line blocking
  ✓ Supports multicast and broadcast
  ✗ No delivery guarantee
  ✗ No ordering guarantee
  ✗ Application must handle reliability if needed

When to Use Each#

The TCP Handshake Cost#

Client              Server
  │── SYN ──────────▶│     t=0ms
  │◀── SYN-ACK ─────│     t=RTT/2
  │── ACK ──────────▶│     t=RTT
  │── Data ─────────▶│     t=RTT (piggybacked)

Minimum: 1 RTT before data flows
With TLS: add 1-2 more RTTs

For a 100ms RTT, TCP+TLS setup costs 200-300ms before the first byte of application data. This is why connection reuse matters enormously.

HTTP/1.1 vs HTTP/2 vs HTTP/3#

HTTP/1.1#

Connection 1: GET /index.html ──▶ response ──▶ GET /style.css ──▶ response
Connection 2: GET /app.js ──▶ response ──▶ GET /image.png ──▶ response
Connection 3: GET /api/data ──▶ response

Problems:

• One request per connection at a time (head-of-line blocking)
• Browsers open 6 connections per domain as a workaround
• Headers sent uncompressed, repeated on every request
• No server push

HTTP/2#

Single Connection (multiplexed):
  Stream 1: GET /index.html ──▶ response
  Stream 2: GET /style.css ──▶ response      (concurrent)
  Stream 3: GET /app.js ──▶ response          (concurrent)
  Stream 4: GET /image.png ──▶ response       (concurrent)

Improvements:

• Multiplexing — multiple requests/responses over one connection
• Header compression (HPACK) — reduces redundant header bytes
• Stream prioritization — critical resources first
• Server push — proactively send resources the client will need
• Binary framing — more efficient parsing than text-based HTTP/1.1

Remaining problem: TCP head-of-line blocking. If one TCP packet is lost, all streams on that connection stall until retransmission.

HTTP/3 (QUIC)#

QUIC Connection (over UDP):
  Stream 1: ──▶ response    (independent)
  Stream 2: ──▶ response    (independent)
  Stream 3: ──▶ response    (independent)

  Packet loss in Stream 2 does NOT block Streams 1 and 3

QUIC advantages:

• No head-of-line blocking — streams are independent at the transport layer
• 0-RTT connection setup — for resumed connections, data flows immediately
• 1-RTT initial setup — TLS 1.3 baked in, handshake and encryption in one step
• Connection migration — survives IP changes (mobile switching WiFi to cellular)
• Built on UDP — avoids middlebox ossification of TCP

Connection Setup Comparison:
  HTTP/1.1 + TLS 1.2:  3 RTTs (TCP + TLS + request)
  HTTP/2 + TLS 1.3:    2 RTTs (TCP + TLS/request)
  HTTP/3 (QUIC):       1 RTT  (QUIC+TLS combined)
  HTTP/3 (0-RTT):      0 RTT  (resumed connections)

WebSocket Protocol#

For bidirectional real-time communication:

HTTP Upgrade:
  Client: GET /chat HTTP/1.1
          Upgrade: websocket
          Connection: Upgrade

  Server: HTTP/1.1 101 Switching Protocols
          Upgrade: websocket

After handshake:
  Client ◀──────▶ Server
  (full-duplex, persistent connection)

WebSocket vs Alternatives#

When to Use WebSocket#

• Chat applications — bidirectional, low-latency messaging
• Live dashboards — real-time metric updates
• Collaborative editing — cursor positions, typing indicators
• Gaming — player state synchronization
• Financial tickers — real-time price updates

When NOT to use WebSocket: If data only flows server-to-client, use Server-Sent Events (SSE) — simpler, auto-reconnects, works through HTTP infrastructure.

gRPC over HTTP/2#

gRPC uses HTTP/2 for transport and Protocol Buffers for serialization:

┌────────────┐    HTTP/2 + Protobuf    ┌────────────┐
│  Service A  │ ◀═══════════════════▶  │  Service B  │
└────────────┘    Multiplexed streams   └────────────┘

gRPC Communication Patterns#

Unary:            Client ──request──▶ Server ──response──▶ Client
Server Streaming: Client ──request──▶ Server ══responses═══▶ Client
Client Streaming: Client ══requests══▶ Server ──response──▶ Client
Bidirectional:    Client ══requests══▶ Server
                  Client ◀══responses══ Server

Why gRPC for Microservices#

• Protobuf — 3-10x smaller than JSON, strongly typed
• HTTP/2 multiplexing — many RPCs over one connection
• Code generation — client/server stubs from .proto files
• Streaming — native support for all four patterns
• Deadlines — built-in timeout propagation across service calls

gRPC Limitations#

• Browser support requires grpc-web proxy
• Not human-readable — binary protocol, harder to debug
• Load balancing — L7 load balancers must understand HTTP/2 frames
• Firewall traversal — some corporate proxies block HTTP/2

DNS Resolution#

Every network call starts with DNS:

Browser                Resolver             Root NS    TLD NS     Auth NS
  │── api.example.com? ──▶│                    │          │          │
  │                        │── .com NS? ──────▶│          │          │
  │                        │◀── ns.tld.com ────│          │          │
  │                        │── example.com NS? ─────────▶│          │
  │                        │◀── ns.example.com ──────────│          │
  │                        │── api.example.com? ───────────────────▶│
  │                        │◀── 93.184.216.34 ─────────────────────│
  │◀── 93.184.216.34 ─────│                                        │

DNS in System Design#

• TTL tuning — low TTL (60s) for failover flexibility, high TTL (3600s) for reduced lookup latency
• DNS load balancing — return different IPs (round-robin or weighted)
• GeoDNS — return the IP of the nearest data center
• DNS failover — health checks remove unhealthy IPs from responses

TLS Handshake#

Every HTTPS connection requires TLS negotiation:

TLS 1.2 (2 RTTs):
  Client ── ClientHello ──────────────▶ Server
  Client ◀── ServerHello + Cert ────── Server
  Client ── KeyExchange + Finished ──▶ Server
  Client ◀── Finished ────────────── Server
  Client ══ Encrypted Data ══════════▶ Server

TLS 1.3 (1 RTT):
  Client ── ClientHello + KeyShare ──▶ Server
  Client ◀── ServerHello + Cert ────── Server
  Client ══ Encrypted Data ══════════▶ Server

TLS 1.3 saves a full RTT. On a 100ms RTT connection, that's 100ms less time-to-first-byte.

Connection Pooling#

Opening a new connection per request is wasteful:

Without pooling:
  Request 1: TCP handshake + TLS + request + response + close
  Request 2: TCP handshake + TLS + request + response + close
  Request 3: TCP handshake + TLS + request + response + close

With pooling:
  Setup: TCP handshake + TLS (once)
  Request 1: request + response
  Request 2: request + response (reuse connection)
  Request 3: request + response (reuse connection)

Pool Configuration#

Pool Settings:
  max_connections: 100        # total connections in pool
  max_per_host: 10            # connections per upstream host
  idle_timeout: 90s           # close idle connections after 90s
  connection_lifetime: 300s   # max age before forced close
  health_check_interval: 30s  # verify connections are alive

Keep-Alive#

HTTP keep-alive reuses TCP connections across multiple requests:

Without Keep-Alive (HTTP/1.0 default):
  [TCP+TLS] → Request 1 → Response 1 → [Close]
  [TCP+TLS] → Request 2 → Response 2 → [Close]

With Keep-Alive (HTTP/1.1 default):
  [TCP+TLS] → Request 1 → Response 1
             → Request 2 → Response 2
             → Request 3 → Response 3
             → [Idle timeout] → [Close]

Keep-Alive Tuning#

• Timeout — how long to hold idle connections (default 5-15s for servers)
• Max requests — close after N requests to prevent resource leaks
• Behind load balancers — keep-alive between LB and backends reduces connection churn
• Between microservices — always use keep-alive for service-to-service calls

Key Takeaways#

1. TCP for reliability, UDP for speed — most web services use TCP; real-time and streaming often benefit from UDP
2. HTTP/2 multiplexes but still suffers TCP head-of-line blocking — HTTP/3 (QUIC) solves this
3. WebSocket for bidirectional real-time communication; use SSE if data only flows server-to-client
4. gRPC over HTTP/2 is the default for microservice communication — smaller payloads, streaming, code generation
5. Connection pooling and keep-alive eliminate repeated handshake costs — configure them in every service
6. TLS 1.3 and QUIC 0-RTT dramatically reduce connection setup time — upgrade where possible

---

286 articles on system design at codelit.io/blog.