Estimating QPS, Storage, & Bandwidth: End-to-End Walkthrough

Let us design the capacity and infrastructure requirements for a global social platform (e.g., a Twitter/X or Instagram hybrid) with the following product scale specifications:

• Daily Active Users (DAU): 500 Million
• User Activity (Writes): 100 Million new posts/day (0.2 posts/user/day).
  • 100\% contain text & metadata (500 bytes).
  • 10\% of posts include an image (2 MB average size).
• User Activity (Reads): 10 Billion feed requests/day (Each active user opens the feed 20 times/day).
  • Each feed request returns a page of 20 post summaries ≈ 10 KB response payload.
• Read-to-Write Ratio: 10 Billion reads : 100 Million writes = 100:1 (Heavily Read-Heavy).

Step 1: Traffic Estimation (QPS Modeling)

Using the 100,000 seconds/day mental approximation:

Average Write QPS = \frac{100,000,000 writes}{100,000 sec} = 1,000 Write QPS

Peak Write QPS = 1,000 × 2 = 2,000 Peak Write QPS

Average Read QPS = \frac{10,000,000,000 reads}{100,000 sec} = 100,000 Read QPS

Peak Read QPS = 100,000 × 2 = 200,000 Peak Read QPS

Step 2: Storage Sizing (Daily & 5-Year Projections)

1. Metadata & Text Database Storage:
   • Daily volume: 100M posts × 500 bytes = 50 GB/day.
   • 5-Year persistent volume: 50 GB/day × 365 days × 5 years = 91,250 GB ≈ 91.25 TB.
   • With 3× cross-AZ replication + 30\% index overhead:

91.25 TB × 3 × 1.30 = 355.8 TB Raw Disk Capacity

1. Blob / Object Storage (Photos & Media):
   • 10\% of 100M posts = 10M images/day.
   • Daily media ingestion: 10M images × 2 MB = 20,000,000 MB = 20 TB/day.
   • 5-Year media volume: 20 TB/day × 365 × 5 ≈ 36.5 Petabytes (PB).
   • Architecture decision: Store images in Amazon S3 / Google Cloud Storage with automated lifecycle policies transitioning objects older than 90 days to S3 Glacier Deep Archive (95\% cost savings).

Step 3: Network Bandwidth Estimation

1. Ingress Bandwidth (Writes):

   • Media Ingress: 10M images/day × 2 MB = 20 TB/day = \frac{20 × 10^{12} bytes}{10^5 sec} = 200 MB/s.
   • Text Ingress: 1,000 QPS × 500 B = 0.5 MB/s.
   • Total Ingress Line Rate: 200.5 MB/s × 8 bits/byte = 1.6 Gbps.

2. Egress Bandwidth (Reads):

   • API Feed Egress: 100,000 Read QPS × 10 KB = 1,000,000 KB/s = 1 GB/s.
   • In Line Rate: 1 GB/s × 8 bits/byte = 8 Gbps Egress.
   • With a Global CDN (Cloudflare/CloudFront) caching 90\% of feed assets and media at the edge, origin datacenter egress drops from 8 Gbps to 800 Mbps, protecting backend API infrastructure.

Step 4: Memory Cache Sizing (80/20 Pareto Working Set)

• Daily API read volume: 10 Billion views × 10 KB = 100 TB read data/day.
• Applying the 80/20 Pareto Rule, 20\% of the unique daily content generates 80\% of read traffic.
• Required in-memory cache size:

RAM Cache Size = 100 TB × 0.20 = 20 TB RAM

A senior system design interview requires connecting mathematical storage and bandwidth numbers to concrete physical node fleet counts:

1. API Application Fleet Sizing:

   • A single optimized stateless Go/Java/Node.js API container handles ~ 2,000 QPS under normal CPU load.
   • Peak Read Traffic = 200,000 QPS.
   • Required API instances = \frac{200,000 Peak QPS}{2,000 QPS/instance} = 100 API Pods (Provision 130 pods for 30\% headroom and redundancy).

2. Redis In-Memory Caching Cluster:

   • Total RAM required = 20 TB.
   • Standard AWS memory-optimized node (r6g.4xlarge): 128 GB RAM.
   • Usable RAM per node (leaving 25\% for Redis copy-on-write BGSAVE overhead) ≈ 96 GB.
   • Cluster Node Count = \frac{20,000 GB}{96 GB/node} ≈ 208 Redis Primary Nodes (+208 replicas across AZs for high availability).

3. Database Sharding Fleet:

   • 5-Year Metadata Storage = 91.25 TB.
   • To keep PostgreSQL/MySQL B-Tree index scans fast and backup restore times under 30 minutes, limit each database shard size to ≤ 2 TB SSD.
   • Shard Count = \frac{91.25 TB}{2 TB/shard} ≈ 46 Shards (Round to 64 shards for clean power-of-2 consistent hashing partition rings).