Complex Systems · Phase 4: High-Level Design (HLD)
Pastebin & Ephemeral Blob Storage
Design a paste service for 50M creates/day using scoped direct uploads, separate metadata and blob storage, logical expiry, lifecycle cleanup, and safe caching.
Primary specification: Amazon S3 Pre-Signed URLs Architecture
Lesson 26 of 33
1d streakThe Architectural Question
Proxying every large paste body through application instances couples API capacity to blob bandwidth and increases memory pressure.
Core Engineering Objectives
- Direct client-to-object storage uploads via pre-signed URLs to bypass backend memory saturation
- Metadata vs Blob storage separation: Relational/NoSQL metadata index + S3/MinIO blob store
- Automatic TTL cleanup using object storage lifecycle rules and database TTL indexes
1. Start with the Data Lifecycle, Not the Upload API
1. Client
Requests an upload slot with size and policy.
2. Metadata API
Creates a pending record and scoped object key.
3. Object store
Accepts the body through a short-lived signed request.
4. Finalize
Verifies object facts before publishing metadata.
5. Reader
Checks visibility and expiry, then fetches the blob.
The read path treats metadata expiry as authoritative even when physical object deletion is delayed.
A paste service stores a text or code blob, returns a compact identifier, and later serves the blob until it expires or is deleted. The beginner-friendly mental model is two records with different jobs: a small metadata record answers who owns the paste, whether it is public, and when it expires; an object holds the larger immutable body. Separating them keeps list, permission, and cleanup queries away from blob transfer.
Metadata
Small queryable fields such as paste ID, owner, object key, size, content type, checksum, visibility, and expiration time.
Blob
The paste body stored as an immutable object. The application refers to it by an opaque object key.
Presigned URL
A time-limited signed request that delegates one specific object-storage operation without revealing cloud credentials.
Lifecycle rule
An object-store policy that queues eligible objects for expiration or transition; it is asynchronous cleanup, not an exact-time user-visible timer.
Use it when
Separate metadata and blob storage when bodies are large, immutable, independently cacheable, or expensive for application servers to proxy.
Reconsider when
Store small pastes directly in a relational row when the size is tightly bounded, traffic is modest, transactional simplicity matters more than independent scaling, and measurements show the database handles the payload comfortably.
2. Define the Product and Security Contract
| Requirement | Worked-example decision | Why it matters |
|---|---|---|
| Create | Public or unlisted paste, maximum 1 MiB | A hard limit bounds storage, bandwidth, scanning, and abuse cost |
| Read | Fetch by opaque ID; public reads may be cached | Authorization and cache policy depend on visibility |
| Expiry | 1 hour, 1 day, 30 days, or never | The read path enforces expiry even if physical deletion is delayed |
| Integrity | Record size and SHA-256 checksum | Finalization can verify the uploaded object before publication |
| Availability | Reads remain available during metadata-primary failover | Hot immutable objects can be cached safely |
| Abuse controls | Per-account quotas, content scanning, reports, takedown | Presigned upload is delegated authority, not trusted input |
A Presigned URL Does Not Enforce Your Whole Product Policy
3. Derive Capacity from Explicit Assumptions
Given 50 million creates/day, a 50 KiB average body, five reads per create, 30-day average retention, and a 3x peak factor, derive each dimension independently. These are course assumptions, not measurements from Pastebin.
| Dimension | Arithmetic | Result |
|---|---|---|
| Average creates | 50,000,000 / 86,400 | ≈579 writes/s |
| Peak creates | 579 × 3 | ≈1,737 writes/s |
| Average reads | 579 × 5 | ≈2,895 reads/s |
| Peak reads | 2,895 × 3 | ≈8,685 reads/s |
| Active blob bytes | 50M × 30 × 50 KiB | ≈75 TB decimal payload |
| Average read payload | 2,895 × 50 KiB | ≈148 MB/s before cache |
| Metadata | 1.5B active rows × 300 B | ≈450 GB logical data before indexes/replicas |
Retention Dominates Storage
4. Separate the Control Plane from the Blob Data Plane
- 01Create API: authenticates, checks quota, allocates paste ID and object key, then creates a PENDING metadata row.
- 02Object storage: accepts the body through a narrowly scoped presigned PUT; application instances do not buffer it.
- 03Finalize API: verifies ownership, object size, checksum, and existence before changing PENDING to ACTIVE.
- 04Read API: checks metadata, visibility, status, and logical expiry, then returns the body or a short-lived signed GET.
- 05CDN: caches only public immutable pastes with a cache key that cannot cross tenants or visibility modes.
- 06Cleanup worker: removes abandoned PENDING rows and reconciles metadata with asynchronous lifecycle deletion.
Architectural Boundary · Do Not Confuse
Proxy upload through the application
Direct upload with presigned URL
Proxying provides one enforcement point but consumes application bandwidth and memory. Direct upload removes the application from the byte path, but requires scoped credentials, finalization, orphan cleanup, and storage-level policy guardrails.
5. Model State Transitions and Ownership
PASTE_METADATA
paste_id opaque primary key
owner_id nullable tenant/user identifier
object_key unique, server-generated storage key
status PENDING | ACTIVE | DELETING | DELETED
visibility PUBLIC | UNLISTED | PRIVATE
size_bytes expected and verified size
sha256 expected and verified checksum
created_at
expires_at nullable logical expiry
version optimistic-concurrency token
INVARIANTS
ACTIVE implies the verified object exists
now >= expires_at implies reads return 410/404 even if bytes still exist
PRIVATE content is never placed in a shared public cache
object_key is never supplied directly by the clientDeletion Is a Workflow
6. Walk Through a Safe Direct Upload
function beginPaste(user, request):
require request.size <= 1 MiB
require quotaAllows(user, request.size)
pasteId = randomOpaqueId()
objectKey = tenantPrefix(user) + "/" + pasteId + "/" + randomNonce()
insert metadata(status=PENDING, expectedSize, expectedChecksum)
upload = signPut(objectKey, expiresIn=5 minutes,
contentLength=request.size, checksum=request.sha256)
return pasteId, upload
function finalizePaste(user, pasteId):
row = loadForUpdate(pasteId)
require row.owner == user and row.status == PENDING
object = objectStore.head(row.objectKey)
require object.size == row.expectedSize
require object.checksum == row.expectedChecksum
compareAndSet(row.status, PENDING, ACTIVE)
return publicOrPrivateReadUrl(row)Retries are safe because the paste ID identifies one workflow and the PENDING-to-ACTIVE transition is conditional. If finalization times out after committing, the client can retry and receive the already active record. If upload never completes, an expiry index lets the sweeper remove the abandoned row and object.
7. Design for Expiry, Orphans, and Hot Pastes
| Failure | Visible symptom | Recovery |
|---|---|---|
| Upload succeeds, finalize fails | Object exists but paste remains PENDING | Retry finalize; later reconcile HEAD results |
| Metadata commits, object is missing | Read receives storage 404 | Return controlled error, alert on invariant breach, repair/delete metadata |
| Lifecycle deletion is delayed | Expired bytes remain physically stored | Read path enforces logical expiry; lifecycle/reconciler handles cost cleanup |
| One paste becomes viral | Origin and metadata hot key overload | Immutable CDN caching, request coalescing, and bounded origin concurrency |
| Private response is cached publicly | Cross-user data disclosure | Default private/no-store policy and visibility-aware cache keys |
Use it when
Return 410 Gone when the product wants clients to distinguish an expired paste from an unknown identifier.
Reconsider when
Return the same 404 for missing, private, and expired resources when reducing identifier enumeration is more important than expressive status.
8. Operate the Lifecycle as Two Clocks
Logical expiry is an application decision evaluated on every read. Physical expiry is asynchronous storage reclamation. Monitor them separately: expired-read rate, PENDING age, orphan bytes, lifecycle backlog, finalize failures, CDN hit ratio, and bytes served from origin. S3 documentation states that eligible objects are queued for removal, so the course must not promise deletion at the exact expiration timestamp.
Failure Exercise: Lifecycle Configuration Is Disabled
9. Active Recall
Active Recall Checkpoint
Why is a successful object upload not enough to publish a paste?
Hint: Consider ownership, size, checksum, and an upload that completes after the metadata request times out.
Show answer & explanation
The service must bind the object to a server-created metadata workflow and verify its properties. A conditional finalize transition prevents an unverified, oversized, corrupted, or cross-tenant object from becoming ACTIVE and makes retries idempotent.
Active Recall Checkpoint
At 50M creates/day, 50 KiB average size, and 30-day retention, what is the active blob payload?
Show answer & explanation
50,000,000 × 30 × 50 KiB is about 75 TB using decimal planning units. Replication, versions, incomplete uploads, metadata, and object-store billing units require additional headroom.
Topic sizing snapshot
The four numbers that materially affect this architecture. Use them as bounds, not as a generic calculator embedded in every lesson.
Peak QPS
~8,685 Read RPS / ~1,737 Write RPS at an illustrative 3x peak
5-Yr Storage
~75 TB active blob payload at 30-day average retention
Bandwidth
~1.2 Gbps average read payload before cache and protocol overhead
Working Set RAM
Measure distinct hot metadata keys; request count alone is insufficient
Architectural Trade-Off Analysis
Compare latency, throughput, consistency, and operational complexity across core industry alternatives.
Architecture Decision Matrix
Multi-axis engineering evaluation and FAANG interview rules of thumb.
SQL (Postgres / MySQL)
Relational ACID Store
Structured schema, relational joins, ACID transactions, and strict foreign keys.
NoSQL (Cassandra / DynamoDB)
Distributed Non-Relational Store
Horizontal partition keys, denormalized documents/wide-columns, and tunable consistency.
| Evaluation Axis | SQL (Postgres / MySQL) | NoSQL (Cassandra / DynamoDB) | Details |
|---|---|---|---|
Latency | Moderate | Strong | |
Throughput | Moderate | Strong | |
Scalability | Moderate | Strong | |
Consistency | Strong | Moderate | |
Complexity | Strong | Moderate | |
Cost | Moderate | Strong |
When to Choose in an Architecture Interview
Choose SQL when ACID guarantees are non-negotiable (financial ledgers, payment processing, inventory checkout) or when query access patterns are unpredictable and require ad-hoc analytical joins.
Choose NoSQL when write volume exceeds single-node capacity (>20K writes/sec), data is unstructured/hierarchical (logs, telemetry, user feeds), or multi-region active-active replication is mandatory.
Never say "NoSQL is faster than SQL" without qualification. SQL with proper B-Tree indexes on indexed primary keys is lightning fast. NoSQL wins on horizontal write scaling, not inherent single-row read speed.
Production Postmortem & Verification
Production Incident Postmortem
- Trigger / Setup: Application servers buffer 5MB text uploads in heap memory before persisting to disk.
- Production Symptom: A spike of concurrent buffered uploads exhausts heap or container memory and causes retries that amplify load.
- Root Cause & Fix: Use tightly scoped direct uploads when byte-path scale requires them, followed by an idempotent verification and finalization step.
Acceptance Criteria & Outcomes
- Calculate storage requirements for 50M pastes/day with 30-day average retention.
- Design end-to-end upload sequence: client request -> pre-signed URL -> S3 PUT -> metadata confirmation.
- Implement cache strategy for hot pastes using distributed CDN and Redis.
Architectural & Technical Interview Review
Core FAANG Interview Questions
Q01How do you justify introducing Pastebin & Ephemeral Blob Storage into a high-scale production architecture?Click to reveal model answer →Hide answer ↑Requirements & Design Brief
Introduce Pastebin & Ephemeral Blob Storage only when measured scale, data size, or failure boundaries exceed single-node capacity. State the exact bottleneck, evaluate operational overhead, and verify that the trade-offs justify the added moving parts.
Q02What is the primary trade-off and hidden operational cost of Pastebin & Ephemeral Blob Storage?Click to reveal model answer →Hide answer ↑Architecture Trade-offs
It improves scalability and availability, but introduces network hops, serialization overhead, and eventual consistency or coordination complexity.
Q03How do you handle catastrophic failure or dependency degradation in Pastebin & Ephemeral Blob Storage?Click to reveal model answer →Hide answer ↑Failure Review & Recovery
Implement connect/read timeouts, exponential backoff with jitter, and fallback defaults so downstream degradations do not cause cascading outages.
Production Pitfalls & Anti-Patterns
Critical Failure Anti-Patterns
Premature Distributed Complexity
Scattering distributed components before identifying concrete write, read, or storage limits creates accidental points of failure and operational debt.
Production Mitigation: Start with the simplest modular design that satisfies requirements, evolving it only as measured constraints dictate.
Unbounded Retries and Cascading Storms
When a dependency degrades, immediate synchronized retries multiply downstream traffic, turning a minor slowdown into a total fleet outage.
Production Mitigation: Enforce strict retry budgets, exponential backoff, and full randomized jitter on all client calls.
Neglecting Observability and Metrics
Deploying architecture without distributed tracing or p99 tail latency alerts leaves engineers blind when partial partitions occur.
Production Mitigation: Export metrics for request rate, error rate, p95/p99 duration, and saturation at every service boundary.