Component Architecture · Phase 2: Core Infrastructure
API Gateways & Edge Routing
Master edge gateway responsibilities: JWT/OAuth2 authentication, request transformation, dynamic route lookup, rate limiting, and distributed tracing propagation.
Primary specification: Pattern: API Gateway (Chris Richardson)
Lesson 9 of 33
1d streakThe Architectural Question
Every downstream microservice redundantly implements authentication, CORS, rate limiting, and logging, leading to inconsistent security postures and maintenance overhead.
Core Engineering Objectives
- Edge offloading: centralized TLS termination, auth validation, and header normalization
- Cross-cutting distributed tracing: W3C TraceContext headers, correlation IDs, and Span context
- BFF (Backend for Frontend) architectural pattern vs universal monolithic gateways
Put one policy boundary in front of many services
An API gateway is the public request boundary for a group of services. It accepts a client request, applies policies that should be consistent across the product, selects the owning backend, and returns or streams the response. The gateway should centralize edge concerns without absorbing business rules that belong to services.
| Concern | Gateway owns | Backend service owns |
|---|---|---|
| Identity | Verify credentials, reject malformed or expired tokens | Authorize the action against domain data |
| Routing | Map host, path, method, and version to a service | Choose the domain operation and workflow |
| Protection | Request-size limits, quotas, coarse rate limits | Domain-specific limits and invariants |
| Observability | Create or propagate trace context and request IDs | Add spans and metrics for domain work |
A gateway is not a distributed monolith
Trace a request through the gateway in a fixed order
- 01Terminate TLS or accept a trusted upstream connection, then reject requests that exceed header or body limits.
- 02Match the host, path, HTTP method, and API version to a route before doing expensive work.
- 03Verify the token signature locally using a cached issuer key; do not call the identity provider for every request.
- 04Apply coarse tenant or client quotas and attach a retry-safe request identifier when the contract supports it.
- 05Create or propagate W3C trace context, remove untrusted forwarding headers, and send the request to a healthy backend with a deadline.
- 06Return the backend response while preserving useful status and retry metadata; record latency, status, route, and saturation metrics.
Architectural Boundary · Do Not Confuse
Universal gateway: one stable public API boundary with shared policy and routing.
Backend for Frontend: a gateway tailored to the data shape and interaction needs of web, mobile, or another client.
Begin with a universal gateway. Add a BFF when clients repeatedly need different aggregation, payload, or release behavior; do not create one merely to rename fields.
Treat authentication, authorization, and identity propagation as different jobs
The gateway authenticates the caller by validating the token format, signature, issuer, audience, and time-based claims. The owning service still authorizes the operation because only it knows facts such as who owns order O-1842, whether that order is refundable, or whether a tenant may access a particular record. Passing an authenticated identity downstream does not grant permission to every resource.
JWKS cache
A bounded cache of an issuer’s public JSON Web Keys. Refresh on an unknown key ID, respect cache policy, retain the previous valid key during rotation, and fail closed when a token cannot be verified.
Principal context
A minimal internal representation of the authenticated subject, tenant, scopes, and assurance level. Services must receive it only from a trusted gateway or workload identity.
Revocation window
The interval during which a correctly signed but revoked self-contained token may remain usable. Short lifetimes reduce the window; high-risk operations may require stateful checks.
Never trust forwarded identity headers from the public request
x-user-id, tenant, scheme, or source-address data that a backend mistakes for gateway-authenticated context.Give the gateway its own latency and capacity budget
Suppose the public API peaks at 40,000 requests per second and gateway work takes 5 ms before proxying. Little’s Law estimates 40,000 × 0.005 = 200 requests concurrently inside gateway policy code. That is only the application-level work: open client connections, response streaming, TLS state, retries, logs, and health-check traffic require separate limits.
| Budget item | Worked estimate | Design implication |
|---|---|---|
| Policy concurrency | 40,000 RPS × 5 ms = 200 in flight | Avoid synchronous remote checks in the common path |
| Request-header ingress | 40,000 × 2 KB × 8 ≈ 640 Mbps | Cap headers and measure request bytes, not only request count |
| Three-zone target | About 13,334 RPS per zone before headroom | Size surviving zones for a zone loss, then add deployment headroom |
| Retry allowance | For example, at most 2% extra attempts | A retry budget prevents a sick backend from receiving multiplied traffic |
Bandwidth needs request and response distributions
Separate the configuration control plane from the request data plane
The data plane accepts user traffic; the control plane distributes routes, certificates, issuer keys, quota policy, and rollout configuration. A control-plane outage should delay configuration changes, not immediately stop requests that can still be handled safely with the last validated snapshot.
- 01Build an immutable, versioned configuration snapshot and validate duplicate routes, missing clusters, invalid certificates, and unsafe timeouts before distribution.
- 02Canary the snapshot to a small gateway cohort and compare error rate, route misses, authentication failures, and latency with the previous version.
- 03Promote only after health checks pass; retain the previous snapshot for a fast rollback.
- 04Persist the last known-good snapshot locally so a restart during a brief control-plane interruption does not create an empty gateway.
- 05Expire security-sensitive material according to policy. “Last known good” must not mean using an expired certificate or verification key forever.
Active Recall Checkpoint
Why should gateways keep a validated configuration snapshot instead of querying a route database for every request?
Hint: Separate request availability from configuration availability.
Show answer & explanation
A local immutable snapshot makes route lookup fast and keeps the request path operating during a short control-plane or database outage. Versioned distribution and canaries make changes observable and reversible. The snapshot still needs explicit expiry rules for security-sensitive material.
Measure the gateway as an overload boundary, not merely a proxy
| Signal | Question it answers | Useful dimensions |
|---|---|---|
| Gateway latency | How much time did edge policy add? | route, region, status class |
| Upstream latency | Which backend consumed the deadline? | cluster, endpoint, outcome |
| Rejections | Are protection policies activating? | reason: auth, size, quota, overload |
| Saturation | Which finite resource is near its bound? | connections, queues, CPU, memory |
| Retry volume | Is the gateway amplifying dependency load? | backend, attempt, final outcome |
requestDeadline = min(clientDeadline, routeMaximum)
remaining = requestDeadline - elapsedTime
if remaining <= minimumUsefulBackendTime:
reject(504, "deadline exhausted before proxying")
attemptBudget = operation.isIdempotent ? route.retryBudget : 0
proxy(request, timeout = remaining, retries = attemptBudget)Trace context is untrusted input at an Internet boundary
traceparent, bound tracestate, apply sampling limits, and never place personal data in either header. A gateway may restart a trace at a security boundary while preserving a separate request ID for support correlation.Design the gateway so its failure does not become every failure
Run multiple stateless gateway instances across failure domains and keep route and key caches usable during short control-plane outages. Every backend call needs a deadline. Retries require a small budget, backoff, jitter, and an idempotent operation; otherwise the gateway multiplies load exactly when a dependency is struggling.
Use it when
Use a gateway for a public or multi-service API that needs consistent authentication, routing, quotas, protocol handling, and trace propagation.
Reconsider when
Do not insert a new gateway between every pair of internal services. Direct calls or a service-mesh data plane are usually clearer for internal service-to-service traffic.
Active Recall Checkpoint
Why should a gateway verify a signed access token locally instead of asking the identity service about every request?
Hint: Consider latency, availability, and load on the identity service.
Show answer & explanation
Local signature verification removes a synchronous network dependency from every request. The gateway can cache public verification keys with bounded refresh behavior, reducing latency and preventing an identity-service slowdown from becoming a product-wide outage. Revocation requirements may still need short token lifetimes or a separate mechanism.
Production Postmortem & Verification
Production Incident Postmortem
- Trigger / Setup: Microservices independently call a centralized OAuth identity server on every single incoming API request.
- Production Symptom: The identity server collapses under 50,000 auth check RPS, causing an outage across the entire company product suite.
- Root Cause & Fix: Validate asymmetric JWTs cryptographically at the edge gateway using cached public verification keys without remote lookups.
Acceptance Criteria & Outcomes
- Specify the division of responsibility between the edge gateway and internal service meshes.
- Demonstrate W3C traceparent header propagation across downstream asynchronous boundaries.
- Evaluate the trade-offs of the Backend for Frontend (BFF) pattern across web, iOS, and Android clients.
Architectural & Technical Interview Review
Core FAANG Interview Questions
Q01How do you justify introducing API Gateways & Edge Routing into a high-scale production architecture?Click to reveal model answer →Hide answer ↑Requirements & Design Brief
Introduce API Gateways & Edge Routing only when measured scale, data size, or failure boundaries exceed single-node capacity. State the exact bottleneck, evaluate operational overhead, and verify that the trade-offs justify the added moving parts.
Q02What is the primary trade-off and hidden operational cost of API Gateways & Edge Routing?Click to reveal model answer →Hide answer ↑Architecture Trade-offs
It improves scalability and availability, but introduces network hops, serialization overhead, and eventual consistency or coordination complexity.
Q03How do you handle catastrophic failure or dependency degradation in API Gateways & Edge Routing?Click to reveal model answer →Hide answer ↑Failure Review & Recovery
Implement connect/read timeouts, exponential backoff with jitter, and fallback defaults so downstream degradations do not cause cascading outages.
Production Pitfalls & Anti-Patterns
Critical Failure Anti-Patterns
Premature Distributed Complexity
Scattering distributed components before identifying concrete write, read, or storage limits creates accidental points of failure and operational debt.
Production Mitigation: Start with the simplest modular design that satisfies requirements, evolving it only as measured constraints dictate.
Unbounded Retries and Cascading Storms
When a dependency degrades, immediate synchronized retries multiply downstream traffic, turning a minor slowdown into a total fleet outage.
Production Mitigation: Enforce strict retry budgets, exponential backoff, and full randomized jitter on all client calls.
Neglecting Observability and Metrics
Deploying architecture without distributed tracing or p99 tail latency alerts leaves engineers blind when partial partitions occur.
Production Mitigation: Export metrics for request rate, error rate, p95/p99 duration, and saturation at every service boundary.