Notification Channel Fan-out Lab (Interactive)
Fan an audience across push, SMS, and email; rate caps turn sends into drain minutes. Fan a large send across channels with real throughput caps and cost-per-message to surface backlog drain time and spend.
Notification Fan-Out: Per-Channel Queues, Quotas & Bill Shock
Route one event to Push (FCM/APNs), SMS (Twilio), and Email (SendGrid) and watch each isolated queue drain at its own rate.
The Notification Ingestion Service validates templates, user preference matrices, DND quiet hours, and rate limits before publishing to one SQS/Kafka topic per channel — a slow Twilio adapter can never backpressure the push fleet. Delivery workers claim messages with visibility timeouts, deduplicate with idempotency keys so network retries don't double-alert, and after exponential-backoff retries route poison messages to a Dead-Letter Queue for operator inspection.
How It Works Under the Hood
A global notification system is a rate-limiting and cost problem disguised as a broadcast. One logical send-to-N-users explodes per channel with very different ceilings: push can do about 25k/sec, but SMS is capped near 2k/sec and costs real money at roughly $0.008 per message, and email throttles around 5k/sec. So the same audience that clears push in seconds becomes tens of minutes of SMS backlog and a four-figure bill. Prioritizing a breaking-news push versus a marketing email keeps the critical queue from stalling behind the cheap one, and a dead-letter queue catches the two percent that fail to deliver.
Core Architectural Principles
- Per-channel drain minutes = audience x channel% / channel rate cap (push 25k/s, SMS 2k/s, email 5k/s).
- SMS cost = messages x $0.008 — marketing SMS to millions is instantly thousands of dollars.
- Failed deliveries route to a dead-letter queue (about 2%) instead of blocking the main pipeline.
Emphasize channel heterogeneity: push and websocket are cheap and fast, SMS and email are rate-capped and costly, so per-channel queues with independent workers prevent a slow channel from stalling everything. Discuss priority tiers, dedup and frequency capping to avoid user spam, opt-out state, and read receipts. Quantifying SMS spend and drain time shows production thinking.
Fan-out per channel isolates slow or expensive paths but demands separate queues, retries, and dead-letter handling for each.