API Gateway vs Service Mesh: When to Use Each and How They Work Together

As organizations adopt microservices, two infrastructure components inevitably enter the conversation: API gateways and service meshes. They share surface-level features — traffic routing, TLS termination, observability — yet they solve fundamentally different problems. Understanding when to use an API gateway vs service mesh (and when to use both) prevents architectural redundancy and operational headaches.

The Core Distinction#

The simplest mental model is traffic direction:

• API Gateway — Manages north-south traffic: requests entering the cluster from external clients (browsers, mobile apps, third-party integrations).
• Service Mesh — Manages east-west traffic: requests flowing between services inside the cluster.

External clients
       │
       ▼  (north-south)
┌──────────────┐
│  API Gateway  │
└──────┬───────┘
       ▼
┌──────────────────────────────────────┐
│            Service Mesh               │
│                                      │
│  Service A ◄──▶ Service B            │
│       │            │    (east-west)   │
│       ▼            ▼                  │
│  Service C ◄──▶ Service D            │
└──────────────────────────────────────┘

API Gateway Deep Dive#

An API gateway is the single entry point for external traffic. It sits at the edge of your infrastructure and handles concerns that external consumers care about.

Core Responsibilities#

1. Request routing — Map external URLs to internal services. /api/users routes to the user service, /api/orders routes to the order service.
2. Authentication and authorization — Validate JWTs, API keys, or OAuth tokens before requests reach backend services.
3. Rate limiting and throttling — Protect services from abuse and enforce usage quotas per API key or tenant.
4. Request/response transformation — Translate between external and internal schemas, aggregate responses from multiple services (BFF pattern).
5. API versioning — Route /v1/users and /v2/users to different service versions.
6. Developer portal and documentation — Expose OpenAPI specs, usage analytics, and key management.

Popular API Gateways#

Service Mesh Deep Dive#

A service mesh manages communication between services. It deploys a sidecar proxy alongside every service instance, forming a data plane controlled by a centralized control plane.

Core Responsibilities#

1. Mutual TLS (mTLS) — Encrypt all inter-service traffic and verify service identity automatically. No application code changes needed.
2. Traffic management — Canary deployments, traffic mirroring, circuit breaking, retries, and timeouts — all configured declaratively.
3. Observability — Distributed tracing, request-level metrics (latency, error rate, throughput), and access logging without instrumenting application code.
4. Service discovery — Sidecar proxies resolve service endpoints dynamically, replacing manual configuration.
5. Authorization policies — Define which services can talk to which other services (e.g., the payment service can call the fraud service, but the notification service cannot).

Popular Service Meshes#

Where They Overlap#

The confusion between API gateways and service meshes stems from genuine feature overlap:

The overlap is real, but the context differs. An API gateway rate-limits an external API consumer; a service mesh rate-limits an internal service to protect a downstream dependency. Same mechanism, different trust boundary.

When to Use Each#

Use an API Gateway When#

• You expose APIs to external consumers (public APIs, mobile apps, third parties).
• You need API key management, usage plans, and developer portals.
• You want to aggregate or transform requests at the edge (BFF pattern).
• You need protocol translation (REST to gRPC, GraphQL to REST).

Use a Service Mesh When#

• You have many services communicating internally and need consistent mTLS, retries, and observability without changing application code.
• You want fine-grained authorization policies between services.
• You need advanced traffic management for canary releases or traffic mirroring.
• You are operating at a scale where manual sidecar configuration is impractical.

Use Both When#

• You have external APIs and complex internal service-to-service communication.
• You want edge concerns (auth, rate limiting, transformation) handled at the gateway and internal concerns (mTLS, circuit breaking, traffic shaping) handled by the mesh.
• Most production microservice architectures at scale end up here.

Combined Architecture#

When using both, the API gateway handles ingress and the service mesh handles everything after:

Internet
   │
   ▼
┌──────────────────┐
│   API Gateway     │  ← AuthN, rate limiting, API versioning
│   (Kong / APISIX) │
└────────┬─────────┘
         ▼
┌──────────────────────────────────────────┐
│          Service Mesh (Istio)             │
│                                          │
│  ┌─────────┐    ┌─────────┐             │
│  │ Service A│◄──▶│ Service B│  ← mTLS,   │
│  │ + Envoy  │    │ + Envoy  │    retries, │
│  └────┬────┘    └────┬────┘    tracing   │
│       ▼              ▼                    │
│  ┌─────────┐    ┌─────────┐             │
│  │ Service C│◄──▶│ Service D│             │
│  │ + Envoy  │    │ + Envoy  │             │
│  └─────────┘    └─────────┘             │
└──────────────────────────────────────────┘

Key principle: Avoid duplicating logic. If the mesh handles retries and circuit breaking, do not configure the same policies in the gateway. If the gateway handles JWT validation, do not re-validate in the mesh.

Ingress Gateway Pattern#

Istio and Linkerd both offer ingress gateway components that blur the line further. Istio's ingress gateway is an Envoy instance that can replace a standalone API gateway for simpler use cases. However, it lacks features like developer portals, API key management, and advanced request transformation that dedicated API gateways provide.

Rule of thumb: If your ingress needs are simple (routing + TLS), the mesh's ingress gateway may suffice. If you need API management features, deploy a dedicated API gateway in front of the mesh.

Migration Path#

Teams rarely adopt both components at once. A pragmatic migration path:

Phase 1 — API Gateway only Start with an API gateway for external traffic. Internal services communicate directly. This works well for fewer than ten services.

Phase 2 — Add observability As services multiply, inject sidecar proxies for distributed tracing and metrics. You do not need to enable mTLS or traffic policies yet — observability alone justifies the mesh.

Phase 3 — Enable mTLS Turn on mutual TLS in permissive mode first (accept both plaintext and TLS). Monitor for issues. Then switch to strict mode.

Phase 4 — Traffic policies Add retry budgets, circuit breakers, and canary routing. Migrate these configurations out of application code and into mesh policy.

Phase 5 — Authorization policies Define service-to-service authorization. Lock down communication so only expected paths are allowed.

Each phase delivers value independently. You can pause at any phase and resume when complexity warrants the next step.

Key Takeaways#

• An API gateway manages north-south traffic (external to internal); a service mesh manages east-west traffic (internal to internal).
• They overlap in features like load balancing, TLS, and observability, but operate at different trust boundaries.
• Most production microservice architectures benefit from both, with clear separation of responsibilities.
• Avoid duplicating policies across both layers — assign each concern to exactly one component.
• Adopt incrementally: gateway first, then mesh observability, then mTLS, then traffic and auth policies.

---

Build and explore system design concepts hands-on at codelit.io.

220 articles on system design at codelit.io/blog.