Complex Systems · Phase 4: High-Level Design (HLD)
Flash Sale & High-Concurrency Inventory Reservation
Design an e-commerce flash sale system handling 500k QPS surges: multi-tier traffic shedding, Redis Lua atomic stock decrement, inventory sharding across keys, async order creation queues, and anti-overselling DB constraints.
Primary specification: Redis scripting documentation
Lesson 32 of 33
1d streakThe Architectural Question
Half a million users attempt to purchase 1,000 discounted items in the first 2 seconds; concurrent database writes cause massive row-lock contention and oversell inventory.
Core Engineering Objectives
- Atomic inventory reservation: Redis Lua script executing `tonumber(stock) >= qty` in memory with single-threaded atomicity
- Inventory key sharding: splitting a 1,000-unit hot SKU into 10 sub-keys (`sku_1`, ..., `sku_10`) to eliminate Redis single-core saturation
- Multi-tier traffic shedding: Edge CAPTCHA rate limiting -> Queue token admission -> Async order confirmation
1. Frame the product and the failure
A flash sale turns one SKU into a hot key: many shoppers arrive together, but only a bounded number of units can be sold. The system must reject excess demand quickly, prevent overselling, and make a retry safe without creating duplicate orders.
Start with a bounded contract
2. Requirements and invariants
Inventory invariant
Confirmed orders plus active reservations never exceed sellable stock.
Backpressure
Admission control protects the reservation service when demand exceeds safe capacity.
Recovery
A retry or expiry returns to one consistent stock state.
Active Recall Checkpoint
Why is a queue alone insufficient to prevent overselling?
Show answer & explanation
A queue orders work but does not make the stock check and decrement atomic. The reservation transition must be atomic, then the queue carries the accepted intent.
3. The request path
1. Edge admission
Bot checks, rate limits, and queue tokens bound arrivals.
2. Reservation API
Validates SKU, customer, and idempotency key.
3. Atomic stock script
Decrements one shard only when sufficient stock exists.
4. Order queue
Durably records accepted reservations.
5. Order database
Commits one order and reconciles expiry.
The database constraint remains the final defense against overselling during reconciliation.
Keep the synchronous path small: validate at the edge, perform one atomic reservation, and move order creation to a durable queue. Expose reservation outcome, queue age, and expiry rate as first-class metrics.
4. Capacity model from explicit assumptions
Use these as illustrative interview assumptions, not production measurements. Write units beside every number and apply the peak factor once.
| Input | Worked assumption | What it sizes |
|---|---|---|
| Traffic | 500,000 requests/sec peak for one campaign | Admission and gateway capacity |
| Stock | 1,000 sellable units for the hot SKU | Reservation state |
| Lease | 10 inventory shards and a 60-second reservation lease | Hot-key distribution and expiry work |
Validate before scaling
5. Data model and consistency boundary
Use the atomic reservation record as the source of truth for available stock. Cache views and order status may lag, but the customer-facing state must distinguish reserved, confirmed, expired, and rejected.
Use it when
Use an atomic in-memory reservation plus a durable order queue when demand is bursty and users can tolerate asynchronous confirmation.
Reconsider when
Use a directly transactional database path when inventory is not hot and measured write capacity is sufficient.
6. Failure modes and recovery
A queue acknowledgement is not payment success. Expired or abandoned reservations must return stock exactly once: guard the expiry transition with the reservation ID and version so a sweeper cannot restore stock after confirmation. If a worker crashes after payment but before order confirmation, reconciliation must query the payment provider and close the business state rather than blindly retrying.
- 01Bound retries with jitter and a caller budget.
- 02Use idempotency keys for reservation and order creation.
- 03Reconcile reservations, payments, and orders from durable records.
- 04Alert on oversell attempts, queue age, expiry backlog, and mismatch counts.
7. The decision you must defend
Architectural Boundary · Do Not Confuse
Short reservation lease
Long reservation lease
A short lease improves fairness but increases expiry churn. A long lease reduces churn but strands stock when shoppers abandon carts.
State the sacrificed property explicitly: the reservation path is strongly consistent, while order confirmation is intentionally asynchronous.
8. Interview walkthrough
- 01Clarify whether overselling is forbidden and whether checkout may be asynchronous.
- 02Estimate peak arrivals, hot-key skew, and available stock.
- 03Draw admission, atomic reservation, queue, and reconciliation.
- 04Explain expiry, duplicate retries, and payment ambiguity.
- 05Close with a load test that proves the hot SKU remains safe.
Active Recall Checkpoint
What would you measure first before shipping this design?
Hint: Choose a signal that can disprove the central assumption.
Show answer & explanation
Measure hot-key skew and reservation correctness under a replayed burst; aggregate QPS can hide one overloaded SKU.
9. Synthesis and primary references
The durable lesson is that admission control protects availability, while one atomic stock transition protects correctness. A queue helps absorb work but cannot replace either contract.
Redis scripting
[Redis scripting documentation](https://redis.io/docs/latest/develop/programmability/eval-intro/)
AWS Builders Library
[Avoiding fallback in distributed systems](https://aws.amazon.com/builders-library/avoiding-fallback-in-distributed-systems/)
Topic sizing snapshot
The four numbers that materially affect this architecture. Use them as bounds, not as a generic calculator embedded in every lesson.
Peak QPS
500,000 Reservation RPS surge
5-Yr Storage
5 TB order history
Bandwidth
10 Gbps ingress traffic
Working Set RAM
16 GB Redis cluster for hot sale SKUs and user reservation locks
Architectural Trade-Off Analysis
Compare latency, throughput, consistency, and operational complexity across core industry alternatives.
Architecture Decision Matrix
Multi-axis engineering evaluation and FAANG interview rules of thumb.
SQL (Postgres / MySQL)
Relational ACID Store
Structured schema, relational joins, ACID transactions, and strict foreign keys.
NoSQL (Cassandra / DynamoDB)
Distributed Non-Relational Store
Horizontal partition keys, denormalized documents/wide-columns, and tunable consistency.
| Evaluation Axis | SQL (Postgres / MySQL) | NoSQL (Cassandra / DynamoDB) | Details |
|---|---|---|---|
Latency | Moderate | Strong | |
Throughput | Moderate | Strong | |
Scalability | Moderate | Strong | |
Consistency | Strong | Moderate | |
Complexity | Strong | Moderate | |
Cost | Moderate | Strong |
When to Choose in an Architecture Interview
Choose SQL when ACID guarantees are non-negotiable (financial ledgers, payment processing, inventory checkout) or when query access patterns are unpredictable and require ad-hoc analytical joins.
Choose NoSQL when write volume exceeds single-node capacity (>20K writes/sec), data is unstructured/hierarchical (logs, telemetry, user feeds), or multi-region active-active replication is mandatory.
Never say "NoSQL is faster than SQL" without qualification. SQL with proper B-Tree indexes on indexed primary keys is lightning fast. NoSQL wins on horizontal write scaling, not inherent single-row read speed.
Production Postmortem & Verification
Production Incident Postmortem
- Trigger / Setup: A flash sale backend updates inventory directly in PostgreSQL using `UPDATE items SET stock = stock - 1 WHERE id = 123`.
- Production Symptom: Database connection pool is exhausted instantly as 100,000 threads queue for the single row lock; database crashes and 1,050 items are sold for 1,000 units.
- Root Cause & Fix: Decouple inventory reservation from relational persistence; reserve stock in atomic Redis Lua counters and enforce DB constraints (`WHERE stock >= qty`).
Acceptance Criteria & Outcomes
- Prove why database row locks collapse under 100,000 concurrent updates to a single row.
- Implement atomic stock decrement in Redis Lua with rollback and lease expiration mechanics.
- Structure asynchronous order creation pipeline: Redis reservation -> Kafka order event -> Worker DB commit -> Payment settlement.
Architectural & Technical Interview Review
Core FAANG Interview Questions
Q01How do you justify introducing Flash Sale & High-Concurrency Inventory Reservation into a high-scale production architecture?Click to reveal model answer →Hide answer ↑Requirements & Design Brief
Introduce Flash Sale & High-Concurrency Inventory Reservation only when measured scale, data size, or failure boundaries exceed single-node capacity. State the exact bottleneck, evaluate operational overhead, and verify that the trade-offs justify the added moving parts.
Q02What is the primary trade-off and hidden operational cost of Flash Sale & High-Concurrency Inventory Reservation?Click to reveal model answer →Hide answer ↑Architecture Trade-offs
It improves scalability and availability, but introduces network hops, serialization overhead, and eventual consistency or coordination complexity.
Q03How do you handle catastrophic failure or dependency degradation in Flash Sale & High-Concurrency Inventory Reservation?Click to reveal model answer →Hide answer ↑Failure Review & Recovery
Implement connect/read timeouts, exponential backoff with jitter, and fallback defaults so downstream degradations do not cause cascading outages.
Production Pitfalls & Anti-Patterns
Critical Failure Anti-Patterns
Premature Distributed Complexity
Scattering distributed components before identifying concrete write, read, or storage limits creates accidental points of failure and operational debt.
Production Mitigation: Start with the simplest modular design that satisfies requirements, evolving it only as measured constraints dictate.
Unbounded Retries and Cascading Storms
When a dependency degrades, immediate synchronized retries multiply downstream traffic, turning a minor slowdown into a total fleet outage.
Production Mitigation: Enforce strict retry budgets, exponential backoff, and full randomized jitter on all client calls.
Neglecting Observability and Metrics
Deploying architecture without distributed tracing or p99 tail latency alerts leaves engineers blind when partial partitions occur.
Production Mitigation: Export metrics for request rate, error rate, p95/p99 duration, and saturation at every service boundary.