Complex Systems · Phase 4: High-Level Design (HLD)
Ride-Sharing Proximity Dispatch (Uber & Lyft)
Architect a ride-sharing proximity system using a 1M-active-driver and 10M-rides/day worked example: location ingestion, H3 candidate search, in-memory geospatial state, and dispatch.
Primary specification: H3: Uber’s Hexagonal Hierarchical Spatial Index
Lesson 30 of 33
1d streakThe Architectural Question
Persisting every one of the exercise's 250k GPS updates/sec into one indexed relational write path creates avoidable contention and I/O.
Core Engineering Objectives
- Geospatial indexing: Uber H3 hexagonal hierarchical spatial cells vs Google S2 geometry vs Geohash
- High-frequency location ingestion: 1M active drivers emitting GPS updates every 4 seconds (250k writes/sec)
- Efficient K-ring neighbor lookups: finding available drivers within radius R with minimal CPU calculation
Step 1: Problem Statement & Operational Context
1. Driver app
Streams fresh location updates.
2. Location service
Validates and writes the latest position.
3. Spatial index
Finds candidates in nearby cells.
4. Dispatch
Filters availability and ranks by travel time.
5. Rider
Receives a matched driver and live ETA.
Grid or H3 cells narrow the search; road-network travel time makes the final decision.
A ride-sharing geospatial proximity dispatch system coordinates supply and demand across metropolitan markets. The platform tracks 1,000,000 active drivers continuously emitting high-frequency GPS coordinates every 4 seconds (250,000 writes/sec), processes 10 million completed rides per day, executes proximity spatial queries to find optimal nearby drivers within sub-second latencies, and dynamically adjusts surge pricing multipliers across micro-regions based on real-time supply-demand imbalances.
The Database Lock Collapse of High-Frequency GPS Pings
Uber H3 Hexagonal Hierarchical Spatial Index
A hierarchical discrete global grid that represents most cells as hexagons and includes twelve pentagons at every resolution. Hexagons generally provide more uniform neighbor relationships than square cells, but projection and icosahedron boundaries still introduce distortion.
K-Ring Neighbor Search
A grid traversal that returns cells within k graph steps of an origin. It is useful for candidate generation, but grid distance is not road distance and pentagon distortion requires library-supported traversal behavior.
Bipartite Matching (Batch Dispatch)
An optimization algorithm (Hungarian / Kuhn-Munkres algorithm) that batches ride requests and available drivers over a 5-to-10 second window, finding the globally optimal assignment that minimizes total aggregate rider waiting time across the entire city.
Step 2: Requirements (Functional & Non-Functional)
The dispatch system coordinates mobile telemetry ingestion, spatial candidate discovery, and atomic ride assignment under strict real-time contracts.
| Dimension | Requirement Specification | Architectural Implementation |
|---|---|---|
| Functional: Location Ingestion | Accept GPS updates (lat, lon, bearing, status) from 1M drivers every 4s | Stateless ingestion gateway -> Kafka topic -> in-memory Redis H3 grid |
| Functional: Proximity Search | Find top K available drivers within radius R (e.g., 3km) of passenger | H3 K-Ring lookup + routing engine road-network ETA calculation |
| Functional: Ride Lifecycle | Request ride, reserve driver lock, offer trip, accept/reject, complete | Distributed state machine with atomic Redis locks (SET NX EX 15) |
| Functional: Dynamic Surge Pricing | Compute surge multiplier per spatial zone based on supply vs demand | Stream aggregation in Flink/Spark computing H3 cell multipliers every 30s |
| Non-Functional: Ingestion Scale | 250,000 GPS pings/sec sustained (Peak: 500,000 pings/sec) | Decoupled in-memory spatial index; zero synchronous disk writes |
| Non-Functional: Query Latency | Proximity search & ETA calculation < 100ms at p99 | In-memory H3 spatial indexing + pre-calculated road network graph |
| Non-Functional: Double-Dispatch Prevention | A driver must NEVER be assigned to two passengers simultaneously | Strict atomic distributed lock reservation with automatic lease timeout |
| Non-Functional: Availability | 99.99% availability across all metropolitan regions | Multi-AZ active-active deployment with localized regional failover |
Use it when
Adopt Batch Bipartite Matching (dispatching in 5-second windows) in high-density urban markets to minimize city-wide passenger waiting times and reduce driver deadhead travel.
Reconsider when
Use Greedy Immediate Dispatch (matching the single closest driver instantly) only in sparse rural areas where driver density is low (<1 driver per 10 square kilometers) and batching yields zero candidate overlap.
Step 3: Detailed Back-of-the-Envelope Capacity Estimations
Back-of-the-envelope calculations size the GPS telemetry throughput, memory buffers, and persistent trip history storage.
| Metric Dimension | Mathematical Derivation | Baseline Number | Peak Load Headroom |
|---|---|---|---|
| Active Driver Fleet | Given operational scope | 1,000,000 active drivers | 1,500,000 drivers |
| GPS Ingestion Frequency | One ping per driver every 4 seconds | 0.25 pings/sec per driver | Every 2 seconds during active trips |
| Ingestion Throughput | 1,000,000 drivers / 4 seconds | 250,000 GPS pings/sec | 500,000 GPS pings/sec |
| GPS Ping Payload Size | Driver ID (8B) + Lat (8B) + Lon (8B) + Bearing (2B) + Status (1B) + Time (8B) | ~35 bytes raw (~128B over wire) | ~200 bytes with TLS headers |
| Network Ingestion Bandwidth | 250,000 pings/s * 128 bytes | ~32 Megabytes/sec | ~64 MB/sec (~512 Mbps) |
| Daily Completed Trips | Given operational scope | 10,000,000 trips/day | 15,000,000 trips/day |
| Trip Booking Rate | 10,000,000 trips / 86,400s | ~115 bookings/sec | ~1,200 bookings/sec peak |
| In-Memory Driver Registry RAM | 1M drivers * (coords + H3 cell + status + trajectory ring) (~250B) | ~250 Megabytes RAM | ~4.0 GB provisioned cluster RAM |
| 5-Year Trip History Storage | 10M trips/day * 365 days * 5 yrs * 2 KB per record | ~36.5 Terabytes | ~110 Terabytes (with 3x replication) |
Why Active Driver State Takes Less Than 4 GB of RAM
Step 4: High-Level Architecture & Component Responsibilities
The architecture decouples high-frequency driver location telemetry from rider dispatch matching, routing, and pricing.
- 01Driver Telemetry Gateway: WebSocket gateway clusters terminating TLS and ingesting 4-second GPS pings from driver mobile applications.
- 02Location Ingestion Pipeline (Kafka): Partitioned Kafka topic
driver-locationsbuffers high-frequency pings, smoothing spikes before processing. - 03Location Ingestion Workers: Stream consumers that convert raw (lat, lon) coordinates into Uber H3 hexagonal cell indices (Resolution 8: ~460m edge length).
- 04Geospatial In-Memory Store (Redis Cluster): Maintains sets of active drivers indexed by H3 cell ID (
h3:cell:{cell_id}-> Set ofdriver_id). - 05Rider Dispatch Service: Receives ride requests from passengers, identifies the passenger's pickup H3 cell, and invokes the Dispatch Matching Engine.
- 06Dispatch Matching Engine: Performs H3 K-Ring neighbor expansion, filters available drivers, queries the Routing Engine for road-network ETAs, and runs bipartite matching.
- 07Driver Lock & Offer Manager: Issues atomic distributed locks in Redis (
SET driver:lock:{id} NX EX 15) and pushes dispatch offers to the selected driver via WebSockets. - 08Surge Pricing Engine (Flink / Spark Streaming): Continuously calculates supply vs demand ratios per H3 cell, updating surge multipliers in Redis every 30 seconds.
- 09Trip Management Service & DB: Authoritative PostgreSQL/CockroachDB storing completed trip records, billing, and rider/driver ratings.
Never Calculate Distance with Euclidean Geometry
Step 5: Detailed Data Models & Storage Schema
The data layer pairs transient in-memory spatial indexes with persistent transactional trip databases.
-- PostgreSQL Authoritative Trip Storage
CREATE TABLE trips (
trip_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
rider_id UUID NOT NULL,
driver_id UUID NULL, -- Assigned after dispatch acceptance
status VARCHAR(20) NOT NULL, -- 'REQUESTED', 'OFFERED', 'ACCEPTED', 'COMPLETED'
pickup_h3_index VARCHAR(15) NOT NULL, -- H3 Resolution 8 index (e.g., '8828308281fffff')
dropoff_h3_index VARCHAR(15) NOT NULL,
pickup_latitude DOUBLE PRECISION NOT NULL,
pickup_longitude DOUBLE PRECISION NOT NULL,
dropoff_latitude DOUBLE PRECISION NOT NULL,
dropoff_longitude DOUBLE PRECISION NOT NULL,
surge_multiplier NUMERIC(3, 2) NOT NULL DEFAULT 1.00,
fare_amount NUMERIC(10, 2) NULL,
requested_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
completed_at TIMESTAMP WITH TIME ZONE NULL
);
CREATE INDEX idx_trips_rider ON trips (rider_id, requested_at DESC);
CREATE INDEX idx_trips_driver ON trips (driver_id, requested_at DESC) WHERE driver_id IS NOT NULL;
-- Redis Data Structures for Geospatial Tracking:
-- 1. Cell Member Set:
-- Key: h3:cell:8828308281fffff -> Set of driver_id { "drv_101", "drv_402" }
-- 2. Driver Location Hash:
-- Key: driver:drv_101 -> Hash { "lat": 37.7749, "lon": -122.4194, "cell": "8828308281fffff", "status": "AVAILABLE", "updated_at": 1718000000 }
-- 3. Surge Pricing State:
-- Key: surge:cell:8828308281fffff -> Hash { "multiplier": 1.8, "demand": 45, "supply": 12 }Architectural Boundary · Do Not Confuse
Geohash / Square Grid
Uber H3 Hexagonal Grid
Geohashes divide the earth into rectangles with variable neighbor distances: orthogonal neighbors are distance d, but diagonal neighbors are distance 1.414d away. Furthermore, Geohash boundaries suffer from discontinuity flaws where two points 1 meter apart can share completely different prefixes. Hexagons have identical distance to all 6 neighbors and minimize edge effects, making radius searches uniform.
Step 6: Core Algorithms & Step-by-Step Request Flows
The dispatch pipeline utilizes H3 K-Ring neighbor expansion, road-network routing, and atomic distributed locking.
// Production Geospatial Dispatch & Atomic Lock Reservation
import { latLngToCell, gridDisk } from 'h3-js';
export interface DriverCandidate {
driverId: string;
latitude: number;
longitude: number;
estimatedEtaSeconds: number;
}
export class ProximityDispatchEngine {
constructor(
private redisClient: RedisClient,
private routingEngine: RoutingServiceClient
) {}
async findAndLockNearestDriver(
riderLat: number,
riderLon: number,
tripId: string
): Promise<DriverCandidate | null> {
// 1. Convert rider GPS to H3 Resolution 8 Cell (~460m edge length)
const centerCell = latLngToCell(riderLat, riderLon, 8);
// 2. Expand K-Rings: Ring 0 (origin) + Ring 1 (6 cells) + Ring 2 (12 cells) = 19 cells
const searchCells = gridDisk(centerCell, 2);
// 3. Multi-get all available driver IDs residing in these 19 H3 cells
const pipeline = this.redisClient.pipeline();
for (const cell of searchCells) {
pipeline.smembers(`h3:cell:${cell}`);
}
const results = await pipeline.exec();
const candidateDriverIds: string[] = [];
for (const [, drivers] of results ?? []) {
if (Array.isArray(drivers)) candidateDriverIds.push(...drivers);
}
if (candidateDriverIds.length === 0) return null;
// 4. Batch query Routing Engine for true road-network driving ETAs
const candidateDetails = await this.fetchDriverCoords(candidateDriverIds);
const rankedDrivers = await this.routingEngine.calculateEtas(
{ lat: riderLat, lon: riderLon },
candidateDetails
);
rankedDrivers.sort((a, b) => a.estimatedEtaSeconds - b.estimatedEtaSeconds);
// 5. Atomic Lock Reservation: Attempt to acquire lock on top candidate
for (const driver of rankedDrivers) {
// Redis SET NX with 15-second TTL ensures driver is only offered one ride at a time
const acquired = await this.redisClient.set(
`driver:lock:${driver.driverId}`,
tripId,
'NX',
'EX',
15
);
if (acquired === 'OK') {
return driver; // Successfully locked driver for dispatch offer!
}
// If lock failed, driver was already offered another trip; try next candidate
}
return null;
}
}- 01Step 1: Telemetry Ingestion: Driver app emits GPS ping every 4 seconds over WebSocket. Ingestion worker computes H3 cell and updates Redis in <2ms.
- 02Step 2: Passenger Ride Request: Rider requests trip. Dispatch service queries Surge Engine for pickup H3 cell multiplier (e.g. 1.5x) and locks quote.
- 03Step 3: K-Ring Spatial Search: Dispatch engine retrieves available drivers in H3 center cell and concentric K-rings (19 cells total).
- 04Step 4: Road Network ETA Ranking: Routing engine calculates traffic-aware driving duration for top candidates.
- 05Step 5: Atomic Lock Reservation: Dispatcher executes
SET driver:lock:{driver_id} trip_id NX EX 15in Redis to prevent concurrent double-booking. - 06Step 6: Push Offer Dispatch: Gateway pushes trip offer to the selected driver's phone with a 15-second countdown timer. If accepted, trip transitions to
ACCEPTED. If rejected or timed out, lock releases and next candidate is offered.
Step 7: Deep Dives into Critical Bottlenecks
Spatial Boundary Arbitrage and Surge Cliffs
Dead Reckoning Under Intermittent GPS Disconnection
Step 8: Real-World Production War Stories
Uber: The DISCO Dispatch Architecture Evolution
Failure Exercise: A Dispatch Retry Storm
Step 9: Interviewer Deep-Dive Gauntlet & Active Recall
Deep-Dive Q1: Why did Uber develop the H3 hexagonal hierarchical spatial index instead of using existing Geohash or Google S2 systems?
Deep-Dive Q2: Why does Batch Bipartite Matching produce superior business outcomes compared to Greedy Closest Driver dispatch?
Deep-Dive Q3: How do you prevent two nearby riders from being dispatched the exact same driver at the same millisecond?
SET driver:lock:{driver_id} {trip_id} NX EX 15. The NX flag guarantees that only the first worker successfully acquires the lock. The second worker receives a nil response, immediately skips that driver, and locks the second-ranked candidate.Deep-Dive Q4: How do you handle driver location ingestion during a complete network partition between the mobile driver fleet and the dispatch cluster?
Active Recall Checkpoint
Calculate the total number of GPS pings per second generated by 1,000,000 active drivers reporting their location every 4 seconds.
Hint: Divide total active drivers by the reporting interval in seconds.
Show answer & explanation
1,000,000 drivers / 4 seconds = 250,000 GPS pings per second. This demonstrates why raw pings must be ingested via streaming buffers (Kafka) and stored in memory (Redis), rather than writing directly to disk databases.
Active Recall Checkpoint
Why are H3 cells useful for proximity candidate generation, and why are they not an exact distance calculation?
Hint: Examine the distance from the center of a cell to all of its immediate neighboring cells.
Show answer & explanation
H3 provides a hierarchical neighbourhood that is convenient for expanding a candidate search. Its projected cells are not perfectly uniform, and every resolution includes pentagons, so the final ranking should use measured distance or travel time.
Topic sizing snapshot
The four numbers that materially affect this architecture. Use them as bounds, not as a generic calculator embedded in every lesson.
Peak QPS
250,000 average driver GPS pings/sec; size peak separately
5-Yr Storage
N/A for raw pings; 50 TB completed ride history over 5 years
Bandwidth
~0.128 Gbps payload floor before protocol overhead
Working Set RAM
40 GB in-memory Redis cluster for active global driver location registry
Architectural Trade-Off Analysis
Compare latency, throughput, consistency, and operational complexity across core industry alternatives.
Architecture Decision Matrix
Multi-axis engineering evaluation and FAANG interview rules of thumb.
SQL (Postgres / MySQL)
Relational ACID Store
Structured schema, relational joins, ACID transactions, and strict foreign keys.
NoSQL (Cassandra / DynamoDB)
Distributed Non-Relational Store
Horizontal partition keys, denormalized documents/wide-columns, and tunable consistency.
| Evaluation Axis | SQL (Postgres / MySQL) | NoSQL (Cassandra / DynamoDB) | Details |
|---|---|---|---|
Latency | Moderate | Strong | |
Throughput | Moderate | Strong | |
Scalability | Moderate | Strong | |
Consistency | Strong | Moderate | |
Complexity | Strong | Moderate | |
Cost | Moderate | Strong |
When to Choose in an Architecture Interview
Choose SQL when ACID guarantees are non-negotiable (financial ledgers, payment processing, inventory checkout) or when query access patterns are unpredictable and require ad-hoc analytical joins.
Choose NoSQL when write volume exceeds single-node capacity (>20K writes/sec), data is unstructured/hierarchical (logs, telemetry, user feeds), or multi-region active-active replication is mandatory.
Never say "NoSQL is faster than SQL" without qualification. SQL with proper B-Tree indexes on indexed primary keys is lightning fast. NoSQL wins on horizontal write scaling, not inherent single-row read speed.
Production Postmortem & Verification
Production Incident Postmortem
- Trigger / Setup: A ride-sharing app updates driver coordinates in a PostgreSQL PostGIS table with spatial indexes on every GPS ping.
- Production Symptom: Index re-indexing lock contention causes database CPU to hit 100% at only 15,000 updates/sec, freezing dispatch matching.
- Root Cause & Fix: Buffer transient driver locations in memory (Redis geospatial / in-memory H3 grid); never write raw 4-second GPS pings to disk.
Acceptance Criteria & Outcomes
- Calculate ingestion throughput: 1M active drivers / 4-second interval = 250,000 GPS updates per second.
- Explain the candidate-search benefits and known distortions of an H3 grid.
- Design the dispatch matching pipeline: Passenger request -> H3 cell lookup -> K-ring driver search -> ETA ranking -> Lock offering.
End-of-Chapter Synthesis & Key Takeaway
Ride-Sharing Proximity Dispatch (Uber & Lyft) is built around one fundamental rule: Establish clear responsibility and explicit trade-offs.. Before adding components, establish explicit quantitative SLAs for latency, throughput, and consistency. Choose the simplest architecture that meets stated constraints.
Architectural & Technical Interview Review
Core FAANG Interview Questions
Q01How do you justify introducing Ride-Sharing Proximity Dispatch (Uber & Lyft) into a high-scale production architecture?Click to reveal model answer →Hide answer ↑Requirements & Design Brief
Introduce Ride-Sharing Proximity Dispatch (Uber & Lyft) only when measured scale, data size, or failure boundaries exceed single-node capacity. State the exact bottleneck, evaluate operational overhead, and verify that the trade-offs justify the added moving parts.
Q02What is the primary trade-off and hidden operational cost of Ride-Sharing Proximity Dispatch (Uber & Lyft)?Click to reveal model answer →Hide answer ↑Architecture Trade-offs
It improves scalability and availability, but introduces network hops, serialization overhead, and eventual consistency or coordination complexity.
Q03How do you handle catastrophic failure or dependency degradation in Ride-Sharing Proximity Dispatch (Uber & Lyft)?Click to reveal model answer →Hide answer ↑Failure Review & Recovery
Implement connect/read timeouts, exponential backoff with jitter, and fallback defaults so downstream degradations do not cause cascading outages.
Production Pitfalls & Anti-Patterns
Critical Failure Anti-Patterns
Premature Distributed Complexity
Scattering distributed components before identifying concrete write, read, or storage limits creates accidental points of failure and operational debt.
Production Mitigation: Start with the simplest modular design that satisfies requirements, evolving it only as measured constraints dictate.
Unbounded Retries and Cascading Storms
When a dependency degrades, immediate synchronized retries multiply downstream traffic, turning a minor slowdown into a total fleet outage.
Production Mitigation: Enforce strict retry budgets, exponential backoff, and full randomized jitter on all client calls.
Neglecting Observability and Metrics
Deploying architecture without distributed tracing or p99 tail latency alerts leaves engineers blind when partial partitions occur.
Production Mitigation: Export metrics for request rate, error rate, p95/p99 duration, and saturation at every service boundary.