Chunked Dedup Sync Lab (Interactive)
Chunk a file, delta-sync only changed blocks, and dedup identical uploads to one copy. Visualize fixed-size chunk hashing, upload savings from delta-sync on an edited file, and cross-user dedup of identical chunks.
Dropbox-style Sync: 4MB Chunks, SHA-256 CAS & Delta Upload
Edit a file, re-upload only the chunks whose content hash changed, and dedupe identical chunks across every copier.
amber = modified chunk uploaded to block server · emerald = block server replies "already have this hash" (zero transfer)
The sync loop: watch local FS → chunk at 4 MB → SHA-256 each block → ask block server which hashes it lacks → upload only the diff → commit the new manifest (ordered chunk hashes + version vector) to the metadata service → notify devices over long-polling/WebSocket. Two offline edits collide on the same manifest parent, so the client gets a conflict copy rather than silent loss. Because a re-uploaded movie ISO has byte-identical chunks, content addressing means the 10,000th uploader transfers 0 bytes and the fleet stores one copy.
How It Works Under the Hood
File sync never re-uploads a whole file. It chops content into chunks — fixed blocks around 4 MB, or content-defined to survive insertions — hashes each, and asks the server which hashes it already has. For a lightly edited file only the chunks that actually changed need to upload, a delta sync, so a 1 MB tweak to a 100 MB file moves megabytes, not a hundred. Globally, identical chunks dedup to a single stored copy, so 10,000 users uploading the same installer consume one copy of storage and bandwidth. Set file size, edited volume, and copiers to see upload savings and dedup capacity shrink.
Core Architectural Principles
- Chunk count = ceil(size / chunk_bytes); only changed chunks upload under delta sync.
- Upload bytes drop from full size to edited-MB when the server already holds the rest.
- Cross-user dedup: 10,000 identical uploads collapse to 1 stored copy of each unique chunk.
Center on chunking plus hashing for dedup and delta sync, so only new chunks travel the wire. Discuss fixed versus content-defined chunking, referencing rsync, and why content-defined resists the shift-boundary problem on inserts. Cover block-level servers where manifests map hashes to blocks, and security via convergent encryption. Mention resumable upload and versioned block retention for history.
Small chunks maximize dedup and delta savings but inflate hash and manifest overhead; large chunks store efficiently yet re-upload more on edits.