Design a File Upload System — Chunked, Resumable, and Scalable

Uploads are deceptively complex#

"Let users upload files." Four words that hide enormous complexity. Large files fail over flaky connections. Virus-infected files endanger your users. Unprocessed images crash your CDN.

Here's how to build file uploads that actually work.

The naive approach (and why it fails)#

Client → POST /upload (entire file in body) → Server → Save to disk

Fails because:

• 500MB file over 3G? Request times out
• Connection drops at 80%? Start over from scratch
• Server runs out of memory buffering the file
• No virus scanning, no processing, no CDN

Chunked uploads#

Split the file into small pieces (5-10MB each) and upload them individually.

Flow:

1. Client splits file into chunks
2. Client uploads each chunk with chunk index + upload ID
3. Server stores chunks temporarily
4. When all chunks arrive, server reassembles the file

Why it works: Each chunk is a small, fast request. If one fails, retry just that chunk, not the entire file.

Resumable uploads#

The gold standard. If the upload is interrupted, resume from where it stopped.

Protocol (tus.io or similar):

1. Client: POST /uploads → server returns upload URL + upload ID
2. Client: PATCH /uploads/{id} with Upload-Offset: 0 + first chunk
3. Client: PATCH /uploads/{id} with Upload-Offset: 5242880 + next chunk
4. Connection drops at chunk 3
5. Client reconnects: HEAD /uploads/{id} → server returns Upload-Offset: 10485760
6. Client resumes from byte 10485760

Libraries: tus-js-client (frontend), tusd (server), or presigned URLs to S3 with multipart upload.

Direct-to-cloud uploads#

Don't route files through your server. Upload directly to cloud storage.

Presigned URL flow:

1. Client requests an upload URL from your API
2. Server generates a presigned S3/GCS URL (valid for 15 minutes)
3. Client uploads directly to cloud storage
4. Cloud triggers a webhook/event when upload completes
5. Your server processes the file asynchronously

Why this is better: Your server never handles file bytes. No memory pressure, no bandwidth bottleneck. The cloud provider handles the heavy lifting.

Processing pipeline#

After upload, files need processing:

Upload Complete → Queue → Virus Scan → Process → Store → Notify

For images:

1. Virus scan (ClamAV)
2. EXIF data extraction
3. Thumbnail generation (multiple sizes)
4. Format conversion (WebP for web)
5. Store originals + thumbnails in cloud storage
6. Update database with file metadata

For videos:

1. Virus scan
2. Transcoding (multiple resolutions + HLS)
3. Thumbnail extraction
4. Store in cloud storage
5. Serve via CDN with adaptive bitrate

For documents:

1. Virus scan
2. Text extraction (for search indexing)
3. Preview generation (PDF thumbnail)
4. Store with access controls

Storage strategy#

Security considerations#

• Virus scanning on every upload before making files accessible
• File type validation — check magic bytes, not just extension
• Size limits — per-file and per-user quotas
• Access control — signed URLs with expiration for private files
• Content-Type enforcement — prevent XSS via uploaded HTML files

See file storage architectures#

On Codelit, search "file storage" or "Dropbox" in ⌘K to see complete file management architectures — chunked sync, deduplication, CDN delivery, and processing pipelines.

---

Design your upload system: search "file storage" on Codelit.io and explore the architecture.