Discord: Trillions of Messages on ScyllaDB & Elixir WebSockets
Scaling real-time communications to billions of daily messages: Elixir process multiplexing, migrating from Cassandra to ScyllaDB C++, and WebRTC audio mesh routing.
Discord moved trillions of messages from Apache Cassandra to ScyllaDB. The migration eliminated Java Garbage Collection pauses that previously caused tail latency spikes, reducing P99 read latencies from seconds to under 5ms.
Cassandra to ScyllaDB: C++ & Seastar Framework
Trillions of messages, millions of concurrent voice & chat connectionsJVM Garbage Collection (GC) pauses causing severe P99 and P99.9 latency spikes across a 177-node Cassandra cluster storing trillions of messages.
Migrated to ScyllaDB, a C++ rewrite of Cassandra utilizing the Seastar thread-per-core asynchronous reactor model with no GC pauses.
Engineered complex zero-downtime dual-write and asynchronous reconciliation pipelines vs 90%+ drop in cluster hardware footprint.
Explain how JVM GC pauses can trigger false-positive node failure detections in gossip protocols, leading to cascading cluster instability.
TCP vs UDP Tradeoffs
Compare reliable byte streams against lightweight datagrams: flow control, congestion avoidance, head-of-line blocking, and when to pick UDP for real-time scale.
Write-Through Caching
Achieve tight cache-database consistency: Synchronous inline writes, cache as system-of-record, and read-latency optimization.
Discord: Scaling Elixir, Rust, & ScyllaDB for Millions of Concurrent Users
Power millions of simultaneous voice/text channels: Elixir BEAM processes, Rust NIF memory safety, migrating from Cassandra to ScyllaDB, and WebSocket gateway scaling.
Bo Ingram, Discord Engineering • 2023
Discord Engineering Blog • 2017
Ready to Practice Discord-Style Systems?
Start with foundational networking, compute, and storage, and build up to complex distributed consensus.