Time-Bucket Scheduler Lab (Interactive)
Scan only the due time bucket instead of the whole table, then advance the minute to fire. Compare bucketed due-task scans against a full-table scan and track rows scanned per second as pending load grows.
Scheduler Partitions by Time Bucket, Not by Hope
100M pending alarms. Compare a due-minute index range scan against the naive full-table WHERE run_at <= NOW().
› scheduler-1 holds the lease; it scans bucket (00 min slot) and pushes task_ids onto the priority queue.
› Workers claim via SQS visibility timeout; a crashed worker's task re-appears and runs again → handlers must be idempotent (at-least-once).
› Heartbeats re-lock the next buckets; a dead leader lets another scheduler take the lease within the session timeout (≈ 15–30s gap tolerated).
› Temporal/Cadence style durable execution instead checkpoints each step, so recovery replays workflow state rather than re-firing timers blindly.
The naive query is a slow-motion outage: scanning 100M rows every second just to find this second's 600 due tasks. Bucketing run_at into 1-minute partitions turns the hot path into a point lookup of one bucket (~69,444 rows at 1min width), and the dispatcher stays leader-elected and thin: its only job is moving due IDs onto queues that stateless workers consume at their own pace.
How It Works Under the Hood
A scheduler that runs "execute every task that is due" with a WHERE run_at <= now full-table scan degenerates as rows pile up — every tick reads history it already fired. Time-bucketing stores tasks keyed to a coarse slot, such as one minute, and only scans the buckets at or before now, so per-tick work tracks due tasks rather than total history. Spread pending tasks across 1,440 daily buckets to compute rows/sec; a bucketed scan stays flat while the naive scan reads the entire backlog every time. Advance the clock a minute to fire a bucket and rotate the leader to show why exactly one dispatcher must own each bucket to avoid double-firing.
Core Architectural Principles
- Average per bucket = pending x (bucket_minutes / 1,440); a bucketed scan reads only due buckets each tick.
- A full scan reads the whole pending backlog on every tick, so rows/sec grows with history.
- A single active dispatcher per bucket, leader-elected, prevents duplicate task firing.
Present time-bucketing or a delayed queue over a scanning cron, and explain the index-on-run_at full-scan degeneration. Cover the exactly-one-writer requirement, via leader election or partitioned bucket ownership, to avoid double execution, and durable task state versus fire-and-forget. Discuss backfill when a scheduler was down and why idempotent task handlers matter.
Time buckets make due work proportional to actual deadlines but add slot granularity and rebalancing when buckets run hot.