Scalable URL Shortener (TinyURL & Bitly)

A URL shortener maps arbitrarily long web addresses (which frequently exceed 500 to 2,000 characters due to UTM tracking parameters, session tokens, and deep-link paths) into compact, alphanumeric aliases such as https://tiny.url/7bX9k2. These shortened URLs conserve character limits on SMS and social platforms, improve readability in print and QR codes, and provide a centralized interception point for link telemetry, geo-routing, and security inspection.

A production URL shortener operates under strict contracts across both public redirection endpoints and administrative creation APIs.

Capacity sizing derives the infrastructure bounds required to sustain 100 million new URLs created per month and 10 billion redirects per month.

The architecture isolates the low-volume, write-heavy URL creation path from the latency-critical, high-volume redirection path.

1. 01DNS & Edge Routing: DNS and routing policy steer traffic to a healthy edge point of presence (POP), which terminates TLS and checks its edge cache. The selected POP is not guaranteed to be geographically closest.
2. 02API Gateway Fleet: NGINX / Envoy proxy fleet enforces rate limiting, authenticates API tokens, and routes traffic between URL Creation Service and URL Redirection Service.
3. 03URL Redirection Service (Read Path): Ultra-lightweight, stateless Go or Rust services that query the local Redis cluster. On cache hit, they return an immediate HTTP 302 Found. On cache miss, they query PostgreSQL/CockroachDB, backfill Redis, and emit a redirect.
4. 04URL Creation Service (Write Path): Orchestrates validation, malware scanning, range-based sequence ID acquisition, Base62 encoding, and persistence.
5. 05Distributed ID Generator (Ticket Server): Centralized sequence allocator reserving ranges of 1,000,000 contiguous IDs per worker, eliminating lock contention.
6. 06Authoritative Storage Tier: Sharded PostgreSQL or distributed SQL (CockroachDB / YugabyteDB) maintaining persistent URL mappings with multi-AZ replication.
7. 07Asynchronous Telemetry Pipeline: Redirection services emit lightweight click events to Apache Kafka, which streams data into ClickHouse for sub-second analytical queries.

The system separates transactional mapping tables from append-only analytics event streams.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

-- PostgreSQL / CockroachDB Authoritative URL Schema
CREATE TABLE url_mappings (
    id BIGINT PRIMARY KEY,                     -- 64-bit globally unique sequence ID
    short_key VARCHAR(7) NOT NULL,             -- Base62 representation (7 chars)
    original_url TEXT NOT NULL,                -- Target URL (up to 2048 chars)
    user_id UUID NULL,                         -- Creator ID for authenticated users
    custom_alias BOOLEAN NOT NULL DEFAULT FALSE,
    is_active BOOLEAN NOT NULL DEFAULT TRUE,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
    expires_at TIMESTAMP WITH TIME ZONE NULL   -- NULL represents perpetual link
);

-- Unique index enables logarithmic B-tree lookup with a small tree height
CREATE UNIQUE INDEX idx_url_mappings_short_key ON url_mappings (short_key);
CREATE INDEX idx_url_mappings_user_created ON url_mappings (user_id, created_at DESC) WHERE user_id IS NOT NULL;

-- ClickHouse Analytics Schema (Optimized for append-only analytical aggregations)
CREATE TABLE url_click_events (
    short_key LowCardinality(String),
    clicked_at DateTime64(3, 'UTC'),
    ip_pseudonym FixedString(64),              -- Rotating keyed HMAC; still personal data
    country_code LowCardinality(FixedString(2)),
    referrer String,
    user_agent String,
    device_type LowCardinality(String)
) ENGINE = MergeTree()
PARTITION BY toYYYYMM(clicked_at)
ORDER BY (short_key, clicked_at);

The core conversion maps an allocated integer from the finite range [0, 62^7) into a fixed-width Base62 key. If sequential keys are too easy to enumerate, apply a reviewed keyed permutation over that same finite domain or generate random keys with a uniqueness check.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

ALPHABET = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
KEY_SPACE = 62^7

function createShortKey():
  rawId = idAllocator.nextUniqueId()
  require 0 <= rawId < KEY_SPACE

  // Optional: use a reviewed, keyed, reversible permutation whose domain
  // is exactly [0, KEY_SPACE). Do not apply an unrestricted 64-bit transform:
  // its output may not fit in seven Base62 characters.
  publicId = finiteDomainPermutation(rawId, secretKey, KEY_SPACE)
  return encodeBase62(publicId).leftPad("0", 7)

function resolveShortKey(key):
  require key.length == 7 and every character is in ALPHABET
  publicId = decodeBase62(key)
  rawId = inverseFiniteDomainPermutation(publicId, secretKey, KEY_SPACE)
  return mappingStore.get(rawId)

1. 01Step 1 (Write Execution): Client issues POST /api/v1/urls with { "original_url": "https://..." }. API Gateway validates URL syntax against an SSRF blocklist.
2. 02Step 2 (ID Generation): Write Service requests an ID from its local in-memory counter. If the allocated range is exhausted, it reserves a new 1,000,000-block range from the central Ticket Server via UPDATE id_allocator SET next_id = next_id + 1000000 RETURNING next_id.
3. 03Step 3 (Encoding & Storage): The sequence ID is bit-shuffled and Base62-encoded to 7 characters (7bX9k2). The record is written to the primary DB with INSERT INTO url_mappings (...).
4. 04Step 4 (Pre-Warming): The newly created mapping is immediately written to Redis with a 24-hour TTL to ensure that immediate clicks avoid database hits.
5. 05Step 5 (Read Execution): User navigates to https://tiny.url/7bX9k2. Routing policy selects a healthy edge proxy. Redis evaluates GET url:7bX9k2. On hit, the server returns HTTP 302 Found with Location: https://original.url. The service publishes an analytics event according to its chosen durability and overload policy.