Pastebin System Design: Scalable Paste Service Guide

Introduction#

Pastebin system design is a classic interview question that covers URL generation, storage trade-offs, expiration policies, and scaling to millions of users. Services like Pastebin, GitHub Gist, and Hastebin all follow a similar architecture.

This guide walks through every component you need to design a production-grade paste service.

Requirements#

Functional#

• Create a paste and receive a unique short URL
• Read a paste by its short URL
• Set expiration (time-to-live)
• Access control: public, unlisted, or private
• Syntax highlighting for code snippets
• View count and basic analytics

Non-Functional#

• High read-to-write ratio (roughly 10:1)
• Low read latency (under 200ms p99)
• Paste size limit of 10 MB
• System should handle 5 million new pastes per day at scale

Scale Estimation#

Short URL length: with a base62 alphabet and 7 characters you get 62^7 (about 3.5 trillion) unique keys — more than enough.

High-Level Architecture#

Client --> API Gateway --> Paste Service --> Metadata DB (MySQL/Postgres)
                                  |
                                  +--> Object Store (S3)
                                  |
                          Cache (Redis) <-- Read path

1. API Gateway — rate limiting, authentication, routing.
2. Paste Service — core business logic.
3. Metadata DB — stores paste metadata (short URL, owner, expiration, access level).
4. Object Store (S3) — stores the actual paste content.
5. Cache (Redis) — caches hot pastes for fast reads.

Paste Creation Flow#

1. Client sends POST /api/pastes with content, language, expiration, and access level.
2. Paste Service generates a unique short key.
3. Content is uploaded to S3 at pastes/{short_key}.
4. Metadata is written to the database.
5. The short URL is returned: https://paste.example.com/{short_key}.

Short URL Generation#

Option A — Pre-generated keys. A Key Generation Service (KGS) pre-computes random 7-character base62 strings and stores them in a keys table. When a paste is created, one key is popped atomically. This avoids collisions entirely.

Option B — Hash-based. Compute MD5 or SHA-256 of the content, take the first 7 base62 characters, and check for collisions. Simpler but requires retry logic.

Pre-generated keys are preferred at scale because they guarantee uniqueness without retries.

Storage#

Metadata (Relational DB)#

CREATE TABLE pastes (
    short_key   VARCHAR(7) PRIMARY KEY,
    owner_id    BIGINT,
    language    VARCHAR(30),
    access      ENUM('public', 'unlisted', 'private'),
    created_at  TIMESTAMP,
    expires_at  TIMESTAMP NULL,
    view_count  BIGINT DEFAULT 0
);

Content (S3)#

Storing content in S3 keeps the database lean and leverages S3's built-in durability (eleven nines). For pastes under 1 KB you could inline them in the database, but a uniform S3 path simplifies the architecture.

Expiration (TTL)#

Pastes can have a TTL: 10 minutes, 1 hour, 1 day, 1 week, 1 month, or never.

Active cleanup: A cron job (or distributed scheduler) runs periodically:

DELETE FROM pastes WHERE expires_at IS NOT NULL AND expires_at < NOW();

A matching S3 lifecycle rule deletes objects whose metadata tag expires_at has passed.

Lazy cleanup: On read, check expires_at. If expired, return 404 and enqueue a delete task. This catches anything the cron missed.

Access Control#

Private pastes require an auth token. The API Gateway validates the token and the Paste Service checks ownership before returning content.

Syntax Highlighting#

Highlighting is done client-side using a library like Prism.js or highlight.js. The server stores the language tag; the client applies the grammar. This keeps the server stateless and avoids CPU-heavy rendering.

For API consumers who need highlighted HTML, offer an optional query parameter ?highlight=true that runs server-side highlighting via a worker pool.

Rate Limiting#

Apply rate limits at the API Gateway level:

• Writes: 60 pastes per minute per IP (anonymous) or per user (authenticated).
• Reads: 300 requests per minute per IP.

Use a sliding window counter backed by Redis. Return 429 Too Many Requests when limits are exceeded.

Analytics (View Count)#

Incrementing view_count on every read would bottleneck the database. Instead:

1. On each read, publish a paste.viewed event to Kafka.
2. A consumer aggregates counts in a buffer and flushes batch updates to the database every few seconds.

This gives near-real-time counts without per-request writes to MySQL.

Spam Detection#

Pastes are a common vector for phishing links and malware distribution. Defenses include:

• Content scanning — check against a blocklist of known malicious URLs and patterns.
• CAPTCHA on creation — for anonymous users or after a threshold of pastes.
• Abuse reporting — allow users to flag pastes; a moderation queue reviews flagged items.
• Machine learning classifier — trained on historical spam pastes to auto-flag new ones.

Caching Strategy#

Use Redis with an LRU eviction policy. Cache the paste content keyed by short_key. On a cache miss, fetch from S3, populate the cache, and set a TTL matching the paste expiration.

For extremely popular pastes, use a CDN (CloudFront) in front of the read path to absorb traffic spikes.

Database Scaling#

• Read replicas handle the 10:1 read-to-write ratio.
• Sharding by short_key hash distributes writes evenly.
• The KGS can run on its own small database to avoid contention with the main metadata store.

API Design#

POST   /api/pastes          — create a paste
GET    /api/pastes/{key}    — read a paste
DELETE /api/pastes/{key}    — delete a paste (owner only)
GET    /api/pastes/{key}/stats — view count and metadata

All endpoints return JSON. The GET paste endpoint also supports Accept: text/plain for raw content.

Summary#

Pastebin is a deceptively simple service that touches on many distributed systems fundamentals — URL generation, storage tiering, caching, and abuse prevention.

---

Build and practice system design problems interactively on codelit.io.

This is article #208 in the Codelit engineering blog series.