Edge & On-Device AI: Models at the Battery Limit
Cloud serving sends data to the model; edge serving sends the model to the data — running inference on the phone, camera, or car itself. This topic covers the four honest reasons to do it (latency, cost, privacy, offline), the compression toolbox that shrinks a model 50-500x (pruning, distillation, quantization), the mobile runtimes (LiteRT, Core ML, ExecuTorch, ONNX Runtime Mobile, llama.cpp), and the firmware-grade update and telemetry discipline that keeps it all healthy.
01.The Problem: The Datacenter Is Far Away
Your model lives in a GPU cluster. Your user lives on a phone in a tunnel.
Between those two points sit a radio network, a backbone, a load balancer, and a queue. Every inference pays that toll. Sometimes the toll is unacceptable, for one of exactly four honest reasons:
- Latency: voice wake-word detection, camera pipelines, and automotive/robotics control loops need 10-50 ms answers. Physics says a round-trip to a datacenter cannot serve a sub-10 ms control loop, no matter how fast your GPU is.
- Cost: streaming billions of raw sensor frames upstream and results back is absurd economics. Filter locally instead: the camera runs detection on-device and sends only events ("person at the door"), not video.
- Privacy: some data literally should not leave the device — keyboard models, health sensors, analysis of footage from a camera pointed inside your home. Edge is the only architecture here, with federated learning (previous topic) as the optional training complement.
- Offline and sovereignty: phones in tunnels, factories without uplinks, products sold into connectivity-poor markets. "Please connect to the internet to use your calculator" is not a product.
So the question becomes
How do we fit a trained model onto a battery-powered, thermally-limited, memory-constrained gadget — and keep it working there?
Everything in this topic is that one sentence, unpacked.
Cloud-to-Edge Artifact and Update Loop
Cloud-to-Edge Artifact and Update Loop
Compression happens in the cloud, artifacts ship through a signed update service, the runtime dispatches to NPU/DSP/GPU, and device-side telemetry (not raw data) returns to close the quality loop.
Unlock Topic #236: Edge & On-Device AI: Models at the Battery Limit
You are viewing a preview. The full in-depth technical walkthrough, worked derivations, and code notebooks for this concept, along with self-assessment quizzes, are available with Pro or Lifetime Access.
Failure modes, high-throughput bottlenecks, and real FAANG implementation decisions.
Interactive system topology diagrams, live parameter simulators, and downloadable SVG charts.
Staff-level multiple-choice quiz questions with instant feedback and answer explanations.
Firebase Google authentication automatically syncs your completed topics and quiz scores.
How clear and actionable was this distributed systems breakdown?