TAO Graph Cache Tier Lab (Interactive)

Sweep Follower hit rates and Wormhole lag across a graph store serving a billion queries. Model TAO, Meta's cache layer over sharded MySQL: Followers absorb reads regionally, Leaders write-through in the home datacenter, and invalidation lag sets how long stale edges fly around the graph.

Meta TAO Two-Tier Cache Flow

Drive a billion `assoc_get` queries per second through Follower RAM caches, the Leader serialization tier, and Wormhole invalidation back to sharded MySQL.

Graph read queries700M / sec

Follower cache hit rate99.8%

Writes (as % of reads)0.2%

Wormhole invalidation lag80 ms

Follower tier QPS700M / s

Leader tier load (misses + writes)2800K / s

MySQL sharded store QPS2100.0K / s

Read load absorbed by RAM tiers99.700%

Avg read latency0.8 ms

Stale Follower window (in-flight invalidations)112,000 keys

MySQL sees 0.300% of graph traffic; 112,000 invalidation messages in flight explain why TAO adds read-after-write session consistency for the writer.

Objects and Associations (id1, atype, id2) live as nodes and typed edges; symmetric friendship becomes two inverse associations so either side is an O(1) `assoc_get`.

How It Works Under the Hood

TAO extends Memcached with graph awareness to serve the social graph behind Facebook, Messenger, and Threads at over a billion queries per second with a six-to-one read-write ratio. Tier one is the Follower cache in every region, absorbing roughly ninety percent of reads; tier two is the write-through Leader cache in the home datacenter sitting in front of shardable MySQL. Cross-datacenter changes propagate through Wormhole, so a profile edit in one region can race against stale Followers elsewhere — eventual consistency priced in milliseconds of invalidation lag.

Core Architectural Principles

Follower cache hit rate determines how few reads descend to Leader and MySQL, driving both QPS and p99 latency.
Write-through Leaders keep the home-region cache and database consistent atomically, making writes the rare, expensive operation.
Wormhole invalidation lag bounds staleness: keys in flight between regions stay stale for the length of the replication gap.

Interview Round Script

For social-graph questions, name TAO and its tiering: regional Followers for reads, write-through Leaders, MySQL as the durable tail. Then quantify: "At 90% Follower hits, a billion reads per second leaves a hundred million to the core." Finish with how invalidation via Wormhole bounds cross-region staleness — that shows you understand geo-consistency, not just caching.

Key Trade-Offs

Aggressive multi-region read caching collapses latency and database load but accepts bounded staleness for any key touched across regions.

Related Curriculum Chapter