Component Architecture · Phase 2: Core Infrastructure
Load Balancers & Reverse Proxies
Deep dive into Layer 4 (TCP) vs Layer 7 (HTTP) traffic balancing, routing algorithms (Round Robin, Least Connections, P2C), and active vs passive health checks.
Primary specification: Maglev: A Fast and Reliable Software Network Load Balancer (Google)
Lesson 8 of 33
1d streakThe Architectural Question
Application instances experience uneven load distribution and degraded backends continue receiving customer traffic, multiplying response latency.
Core Engineering Objectives
- Layer 4 transport forwarding (Maglev/IPVS) vs Layer 7 application proxying (Envoy/NGINX)
- Balancing algorithms: Round Robin, Weighted Least Connections, Power of Two Random Choices (P2C)
- Readiness probes vs liveness probes and zombie server eviction
Interactive system trace
Watch the architecture make its key decision
Switch scenarios, predict the next active component, and step through the request or failure path.
Normal flow
Load Balancing
GET /products arrives while all three servers are healthy.
Healthy backends
3 / 3
Chosen backend
Pending
Requests in this trace
No dispatch yet
The client calls one public address instead of knowing every backend.
Why it matters: Backends can be added or replaced without changing the client.
1. What is it & Why it Matters
A load balancer is a reverse proxy that accepts incoming client traffic and distributes it across a pool of healthy backend servers. It acts as the single stable front door for an application.
Why it matters in System Design
2. Layer 4 (Transport) vs. Layer 7 (Application) Balancing
Load balancers operate at different layers of the OSI network model with massive implications for throughput and routing intelligence:
| Dimension | Layer 4 (L4) Balancer (AWS NLB, Katran, IPVS) | Layer 7 (L7) Balancer (Envoy, NGINX, AWS ALB) |
|---|---|---|
| OSI Layer | Layer 4 (TCP / UDP) | Layer 7 (HTTP / HTTPS / gRPC) |
| Routing Basis | IP address + TCP/UDP port (5-tuple hash) | HTTP path, headers, cookies, query parameters |
| TLS Handling | Passthrough (passes encrypted packets to backends) | Terminates TLS, decrypts payload, can re-encrypt |
| Throughput Ceiling | Massive: > 10,000,000 packets/sec per host | Moderate: 50,000 – 150,000 requests/sec per host |
| Direct Server Return (DSR) | Supported: backend replies directly to client, bypassing LB | Not possible: proxy must mediate bidirectional HTTP stream |
| Hardware Cost | Minimal CPU / memory overhead | High CPU cost for TLS decryption and header parsing |
LAYER 4 (NLB / IPVS) - PACKET-LEVEL TCP BALANCING:
Client ----[TCP SYN/ACK + Encrypted Packets]----> L4 Balancer ----[Direct Forward]----> Backend Server
(No TLS decryption, 5-tuple hash [Src/Dst IP & Port], >10M pkts/sec, Direct Server Return supported)
LAYER 7 (ALB / Envoy / NGINX) - APPLICATION-LEVEL PROXY:
Client ----[TLS Handshake + HTTP GET /checkout]----> L7 Balancer (Terminates TLS, inspects headers)
|
+----(Route /checkout)----> Payment Service
+----(Route /catalog)-----> Catalog Service
(Decrypts payload, evaluates cookies/JWTs/paths, 50k–150k req/sec)3. Routing Algorithms & The Fast-Failure Trap
Round Robin
Routes requests sequentially across nodes. Simple and state-free, but slow requests accumulate on unlucky nodes.
Weighted Round Robin
Assigns traffic proportional to static hardware capacity (e.g., an 8-core server receives 2x the requests of a 4-core server).
Least Connections
Routes to the server with the fewest active open sockets. Effective for long-lived sessions, but vulnerable to fast-failure cascades.
Power of Two Choices (P2C)
Picks two healthy nodes at random and routes to the one with lower active connections or latency. Achieves near-optimal load distribution in O(1) time without centralized locks.
Architectural Boundary · Do Not Confuse
Least Connections (Dynamic Socket Counting)
Power of Two Choices (P2C) with EWMA Latency
Under Least Connections, if Node A crashes and begins instantly returning HTTP 500 in 1ms (instead of normal 100ms processing), its connection count drops to zero. The balancer interprets this as high capacity and diverts 100% of traffic into the dying instance (Fast-Failure Sinkhole). P2C with latency-based weighting prevents this trap.
4. Health Checks, Draining & High Availability
- 011. Active Health Checking: The balancer periodically probes backends (e.g., GET /healthz every 5 seconds). If a node fails 3 consecutive probes, it is marked unhealthy and removed from the active pool.
- 022. Passive Outlier Detection: The balancer monitors real user traffic in real time. If a node returns 5 consecutive HTTP 5xx errors to live clients, it is ejected immediately for 30 seconds without waiting for the next active probe.
- 033. Connection Draining (Graceful Shutdown): When a server is deregistered for deployment, the load balancer stops routing new requests to it while allowing in-flight connections to finish (e.g., 30-second drain window).
- 044. Slow Start (Warming Up): When a new node joins, the balancer gradually ramps traffic from 5% to 100% over 60 seconds to allow JIT compilation and local caches to warm up without getting overwhelmed.
Use it when
Deploy an L4 balancer (e.g., AWS NLB) at the outer edge to handle raw high-packet TCP ingestion, and place an L7 balancer (e.g., Envoy) behind it for path-based microservice routing.
Reconsider when
For simple applications under 20,000 QPS, a single managed L7 cloud load balancer (AWS ALB or Cloudflare) is sufficient and eliminates operational complexity.
5. Routing Algorithm Cheat Sheet
| Algorithm | Algorithmic Complexity | State Overhead | Major Weakness / Failure Mode |
|---|---|---|---|
| Round Robin | O(1) | None (single counter) | Slow requests accumulate on unlucky nodes |
| Weighted Round Robin | O(1) | Low (static weight map) | Does not adapt to runtime dynamic latency spikes |
| Least Connections | O(N) | Tracks active sockets | Fast-Failure Sinkhole: floods crashing nodes |
| Consistent Hashing (IP) | O(log N) | Ring token structure | Non-uniform traffic from corporate NAT gateways / proxy IPs |
| Power of Two Choices (P2C) | O(1) | EWMA latency history | Slight variance in load, but avoids herd oscillation |
6. Active Recall Checkpoint
Active Recall Checkpoint
What is the "Fast-Failure Sinkhole" pathology in Least Connections load balancing, and how do modern balancers prevent it?
Hint: Think about what happens to the active socket count when a server crashes and returns fast HTTP 500 errors.
Show answer & explanation
When a backend node experiences internal failure (e.g., database connection pool exhaustion) that returns errors faster than healthy nodes, its active connection count can look artificially low. A least-connections balancer may then send it disproportionate traffic Modern balancers prevent this using passive outlier detection (ejecting nodes that emit consecutive 5xx errors) and P2C with response latency tracking.
Topic sizing snapshot
The four numbers that materially affect this architecture. Use them as bounds, not as a generic calculator embedded in every lesson.
Peak QPS
120,000 RPS
5-Yr Storage
N/A
Bandwidth
15 Gbps
Working Set RAM
32 GB connection state table
Production Postmortem & Verification
Production Incident Postmortem
- Trigger / Setup: A cluster uses basic Round Robin routing when one instance suffers an uncaught thread deadlock.
- Production Symptom: Every Nth request routed to the deadlocked instance hangs until client timeout, elevating system error rate to 20%.
- Root Cause & Fix: Health checks must evaluate deep application readiness, and routing algorithms should track active in-flight connections.
Acceptance Criteria & Outcomes
- Contrast L4 packet forwarding speed with L7 content-aware routing capabilities.
- Derive the mathematical advantage of Power of Two Random Choices (P2C) over round robin under heterogeneous workloads.
- Define timeout, retry, and connection draining policies during rolling deployments.
Architectural & Technical Interview Review
Core FAANG Interview Questions
Q01When would you choose a Layer 4 (Transport) balancer over a Layer 7 (Application) balancer in a large-scale architecture?Click to reveal model answer →Hide answer ↑Requirements & Traffic Distribution
Choose Layer 4 (e.g. AWS NLB, IPVS) when you require ultra-high packet throughput (>millions of packets/s), sub-millisecond wire latency, or protocol transparency (raw TCP/UDP/WebSockets) without TLS termination overhead. Choose Layer 7 (e.g. Nginx, Envoy, AWS ALB) when routing decisions depend on HTTP host headers, URL paths, cookie-based session pinning, gRPC method paths, or request body payload inspection.
Q02Why should backend health checks verify deep dependency readiness rather than just shallow process liveliness?Click to reveal model answer →Hide answer ↑Request & Failure Flows
A web process can be alive in memory (returning HTTP 200 on /ping) while its database connection pool is completely exhausted, its local disk is 100% full, or essential cache clusters are unreachable. Routing traffic to such an instance causes cascading 500 errors. Readiness checks verify that the instance can actually fulfill end-to-end work right now.
Q03How do you prevent a single slow backend server from triggering a thundering herd failure across the entire fleet?Click to reveal model answer →Hide answer ↑Architecture Trade-offs
Implement Least Connections or Peak EWMA routing algorithms rather than naive Round Robin so loaded nodes receive fewer incoming requests. Couple this with aggressive connect/read timeouts, circuit breakers that eject degraded instances after N consecutive timeouts, and bounded request queues with load shedding.
Production Pitfalls & Anti-Patterns
Critical Failure Anti-Patterns
Sticky Sessions Memory Leaks and Imbalance
Pinning clients to specific servers via cookies prevents even load distribution, creates single-server memory hotspots, and breaks user sessions whenever that backend node restarts or autoscales.
Production Mitigation: Store session state externally in an in-memory Redis cluster or use signed stateless JWT tokens.
Health Check DDoS Amplification
Aggressive health checks configured to hit expensive database queries every 1 second from 10 redundant load balancers can generate thousands of queries per second, saturating the database during outages.
Production Mitigation: Keep health check endpoints lightweight and cache the internal health status for 2-3 seconds.
Single Load Balancer Single Point of Failure (SPOF)
Placing all application traffic behind a single balancer instance makes any network or hardware glitch a full site outage.
Production Mitigation: Deploy redundant balancers in an active-passive setup with floating Virtual IP (VRRP / Keepalived) or active-active via Anycast BGP routing.