File Upload Pipeline

The Problem

File uploads seem simple until you handle large files (multi-GB videos), unreliable networks (mobile users losing connection mid-upload), concurrent uploads from thousands of users, and post-upload processing (virus scanning, thumbnail generation, format conversion). A production upload pipeline needs resumable uploads (don’t restart a 2GB upload from scratch), direct-to-storage uploads (bypass your API server), progress tracking, file validation, and an async processing pipeline that transforms uploads into the formats your application needs.

Architectural Approaches

Architecture

Loading diagram...

Key Components

Data Model

Loading diagram...

S3 Multipart Upload Flow

For files larger than 100MB, S3 multipart upload splits the file into parts uploaded in parallel.

Flow:

1. Initiate: API calls CreateMultipartUpload → returns an upload ID
2. Upload Parts: Client uploads each part (5MB-5GB each) with the upload ID and part number. Each part returns an ETag.
3. Complete: API calls CompleteMultipartUpload with the list of part numbers and ETags. S3 assembles the final object.
4. Abort: If upload fails, call AbortMultipartUpload to clean up parts.

Presigned approach: Generate a presigned URL for each part. Client uploads parts directly to S3. Your server only orchestrates the initiate/complete calls.

Resume: If a part upload fails, only that part needs to be re-uploaded. List uploaded parts with ListParts and resume from the last successful part.

Parallel uploads: Upload multiple parts simultaneously (3-5 concurrent parts is typical). Total upload time = file_size / (part_count × bandwidth_per_connection). A 1GB file with 5 parallel 100MB parts completes 5x faster than sequential upload.

Cleanup: Set an S3 lifecycle rule to abort incomplete multipart uploads after 7 days. Otherwise, orphaned parts accumulate and incur storage costs.

File Validation and Security

Uploaded files are untrusted user input — treat them with extreme caution.

Content type validation: Don’t trust the client’s Content-Type header. Validate by reading the file’s magic bytes (file signature). A .jpg file should start with FF D8 FF. Libraries: file-type (Node.js), python-magic.

Size limits: Enforce per-file and per-user limits. Check BEFORE the upload completes — presigned URLs can include a content-length-range condition. S3 presigned POSTs support conditions like ["content-length-range", 0, 10485760] (10MB max).

Filename sanitization: Never use the original filename for storage. Generate a random key (UUID or hash). Store the original filename in the database for display only. This prevents path traversal attacks and filename collisions.

Image-specific: Reprocess all uploaded images (strip EXIF metadata for privacy, re-encode to prevent polyglot files). Use a library like sharp to decode and re-encode. This neutralizes steganography and embedded exploits.

Virus scanning: Scan every file before it’s accessible. Use ClamAV (open source) or a cloud API (VirusTotal, AWS GuardDuty). Quarantine suspicious files and notify the uploader. Never serve a file that hasn’t been scanned.

Storage isolation: Store uploads in a separate bucket/container from your application code. Never store uploads in a web-accessible directory. Serve only through your CDN or via presigned download URLs.

Progress Tracking and UX

File uploads are one of the few user-facing operations that can take minutes. Good UX is critical.

Client-side progress: XMLHttpRequest and fetch both support upload progress events. Show a progress bar with percentage, upload speed, and estimated time remaining. For chunked uploads, track per-chunk and overall progress.

Server-side status: Store upload status in the database: pending → uploading → processing → complete. Clients can poll a status endpoint or subscribe via SSE for real-time updates. Show processing status after upload completes (e.g., “Generating thumbnails…”).

Retry UX: On failure, show “Upload failed. Retry?” button — don’t silently restart. For resumable uploads, show “Resuming from 67%…” to reassure the user that progress wasn’t lost.

Drag and drop: Support drag-and-drop file selection alongside the traditional file picker. Show a drop zone overlay when files are dragged over the page. Uppy provides this out of the box.

Batch uploads: For multiple files, show individual progress per file with an overall progress summary. Allow canceling individual files without affecting others. Process files in parallel (2-3 concurrent uploads) for faster batch completion.

Mobile considerations: Mobile networks are unreliable. Use smaller chunk sizes (1-2MB vs 5-10MB on desktop). Implement aggressive retry logic. Consider pausing uploads when the app is backgrounded and resuming when it returns to the foreground.

Real-World Examples

Architectural Tradeoffs

# Learn the System Design of: File Upload Pipeline

You are an expert software architect and teacher. I want to understand file upload pipeline as a practical system design capability. (Chunked and resumable uploads, server-side processing, and CDN delivery at scale).

Use the following architectural guide as a learning reference. Avoid one-size-fits-all recommendations. Keep assumptions, costs, and tradeoffs explicit.

---

import { Mermaid, ComponentList, ComponentCard, ApproachList, Approach, Example, Tradeoff } from '@/components/mdx';

## The Problem

File uploads seem simple until you handle large files (multi-GB videos), unreliable networks (mobile users losing connection mid-upload), concurrent uploads from thousands of users, and post-upload processing (virus scanning, thumbnail generation, format conversion). A production upload pipeline needs resumable uploads (don't restart a 2GB upload from scratch), direct-to-storage uploads (bypass your API server), progress tracking, file validation, and an async processing pipeline that transforms uploads into the formats your application needs.

## Architectural Approaches

<ApproachList>
<Approach
  name="Presigned URL Direct Upload"
  pros={[
    'API server handles zero file bytes — massively reduces bandwidth and CPU',
    'Scales naturally — cloud storage handles concurrent uploads',
    'Client can upload directly from browser/mobile with progress tracking',
    'Built-in multipart upload support in S3/GCS for large files',
  ]}
  cons={[
    'Two-step process: get presigned URL, then upload',
    'Harder to validate file content before it reaches storage',
    'CORS configuration required for browser uploads',
    'Presigned URLs have expiry — must handle edge cases',
  ]}
>
Generate a presigned URL from your API server that allows the client to upload directly to cloud storage (S3, GCS, R2). Your server never handles the file bytes — it only generates the upload URL and is notified via event/webhook when th...

[Full prompt copied to clipboard]