Presigned Multipart Upload Lab (Interactive)
Stream a gigabyte video over a lossy link directly to S3 in parallel chunks versus proxying it through the API tier. Compare presigned direct multipart uploads to web-server-proxied PUTs on wall-clock time, retried bytes and worker-thread burn.
Presigned Multipart vs Web-Server Proxy Uploads
A 1.6 GB video over a lossy 40 Mbps mobile link — chunk it or choke the API tier.
Wall-clock upload
4.4 min
Chunk retries
17 / 160 parts
Bytes actually sent
1,770 MB (+11%)
API worker-seconds burned
0
S3 Multipart part ledger (InitiateMultipartUpload → Complete with ETags)
Only parts 2, 6, 14, 17, 25… re-sent (170 MB wasted). Uploaded bytes persist server-side — a laptop lid close resumes the missing chunks, not byte zero.
Flow: client authenticates → API SigV4-signs a 15-min PUT-only ticket (Content-Length capped, MIME whitelisted) → client uploads parts straight to the S3 regional endpoint. The app server never touches a media byte. s3:ObjectCreated then fans the work to an SQS-backed FFmpeg fleet for HLS renditions.
How It Works Under the Hood
Streaming user media through application servers is a classic failure: a multi-gigabyte upload pins a worker thread and socket for minutes, doubles egress cost crossing the cloud edge twice, and OOMs the heap under load. The fix is presigned URLs: the API server verifies the JWT, then SigV4-signs a short-lived, PUT-only, size-and-type-constrained ticket so the client uploads straight to S3 and the app never touches a byte. For large files, S3 Multipart Upload splits them into 5-25 MB parts sent across parallel connections; a dropped part retries alone instead of restarting from byte zero, making uploads resumable, then an S3 event fires the async transcode.
Core Architectural Principles
- A SigV4 presigned URL grants temporary, method- and size-restricted direct writes to S3.
- Multipart splits into parts so only the failed chunk retries, and the upload survives disconnects.
- Parallel chunk PUTs saturate client bandwidth that a single stream cannot.
Designing YouTube/Instagram uploads, immediately propose direct-to-S3 presigned URLs plus multipart uploads and justify with the failure mode: proxying through web servers starves threads, doubles bandwidth and OOMs. Explain that a SigV4 ticket is method-, size- and expiry-bound, so the app server authenticates but never touches media bytes. Finish with the event-driven post-processing chain: S3 ObjectCreated to SQS to an autoscaled FFmpeg fleet producing HLS for the CDN.
Presigned multipart uploads maximize throughput and resilience while keeping the API tier stateless, at the cost of client-side chunk orchestration.