Model Serving: From Notebook to Production Endpoint

In the notebook, inference is one line:

prediction = model.predict(X)

It works because the notebook is a world of one: one user (you), one copy of the model in RAM, one request at a time, no deadlines, no versions, no crashes worth noticing.

Production demolishes every one of those "ones":

• Thousands of people ask simultaneously — who lines them up?
• Your model is v4.2 but rollback needs v4.1 in the same minute — where do versions live?
• The first request after boot takes 4 seconds (cold caches, compiled kernels) while the rest take 20 ms — what does the load balancer think?
• Traffic triples at 9 PM — who adds replicas, based on what signal?
• A data scientist's pickle "works on my machine" — what exactly is deployable?
• Nobody knows p99 latency quietly breached the SLO — who?

So the question becomes

That system is an inference serving stack. Everything in this topic is one of its organs: the menu (API), the host stand (gateway and load balancer), the expediter (scheduler and batching), the kitchen layout (GPU allocation), the recipe vault (model registry), the health inspector (probes), and the staffing plan (autoscaling).

A serving system is simply

One sentence each, expanded in sections 3–8:

1. Interface — a stable protocol: gRPC for internal high-QPS traffic (protobuf schemas, HTTP/2 multiplexing), REST/JSON at public edges. TF Serving and Triton expose both.
2. Versioned artifacts loaded at boot — from a registry or object store. Never "whatever pickle the data scientist emailed."
3. Scheduling — a queue, plus batching requests into one GPU call, plus optional priorities.
4. Lifecycle — warm-up (run dummy inputs before accepting traffic; the first CUDA kernel compile can take 100 ms to seconds), health probes, graceful drain on rollout.
5. Autoscaling on concurrency/queue depth, not CPU — a GPU at 20% utilization can still be queuing; KServe/KEDA-style custom metrics drive this.
6. Observability — per-model p50/p95/p99 latency, batch fill rates, GPU utilization (DCGM exporter), prediction logging for drift detection.

Miss any one and the notebook line stops being true for strangers. The rest of the topic walks the request path and the kitchen layout, using the restaurant analogy as the glue.

Capacity arithmetic first — Little's law, the only formula this topic needs:

in-flight requests = arrival rate × service time

Say your ranker answers in 50 ms of GPU time and traffic is 2,000 QPS:

in-flight = 2,000/s × 0.050 s = 100

At any moment 100 requests are inside your fleet. If one replica can hold 20 concurrent in-flight requests comfortably, you need 100 / 20 = 5 replicas (plus a safety factor). That is the whole autoscaling design problem: keep the in-flight number matched to slots.

Now the batching magic — why the GPU loves groups:

• One request through your model: 3 ms (mostly reading weights — bandwidth-bound, recall the GPU topic).
• A batch of 16: ~4 ms — nearly free, because the weight read is amortized 16 ways.

Per GPU: batched ≈ 4,000 predictions/s. Unbatched ≈ 333/s. Same chip, 12× the bill.

Which creates the serving dilemma the scheduler exists to solve: batch up (wait, raise latency) or dispatch now (waste GPU)? Dynamic batching (section 6) is the compromise with a dial on it — and why the dial exists is exactly these two numbers.

Follow one request walking through the building:

code
Copy
 client ──HTTPS──► API GATEWAY (auth, rate limits)
                      │
                      ▼
                 LOAD BALANCER (route to the instance with
                      │          the fewest outstanding requests)
                      ▼
              ┌── SERVING INSTANCE ──────────────────┐
              │  QUEUE            [r7, r2, r9, ...]  │
              │    │                                 │
              │    ▼                                 │
              │  SCHEDULER ── "wait up to 3 ms,      │
              │    │           then batch what came"  │
              │    ▼                                 │
              │  RUNTIME BACKEND (ONNX Runtime,      │
              │    │           TensorRT, Python)      │
              │    ▼                                 │
              │  GPU WORKERS (2 model instances,     │
              │    │         or one MIG slice)        │
              └────┼──────────────────────────────────┘
                   ▼
              response ──► metrics emitted: latency, queue depth,
                            batch fill, GPU util ──► autoscaler

Three observations the picture makes obvious:

• The queue is inside each instance, not at the gate — that is what "autoscale on queue depth/concurrency" means.
• The registry feeds artifacts into instances at boot (side door), not into the request path.
• Telemetry flows the other way — out of every organ — because staffing and alerting read it.

Run the whole topic through one analogy: a serving system is a restaurant.

The analogy also predicts every classic mistake. A restaurant that seats walk-ins one chef per party (no batching) pays 12×. One that staffs by chef idle-stare percentage (CPU utilization) is understaffed during the rush — queues form while everyone technically "looks busy" — which is exactly why the autoscaling signal is concurrency/queue depth. And "we serve whatever the chef wrote on a napkin" (unversioned pickle) is a rollback and audit disaster waiting for opening night.

Sections 6–9 fill in the kitchen brands (Triton vs the others), the floor plan (how many models per GPU), and the other service formats (drive-through = streaming, catering = batch inference, food trucks = edge).

You rarely build the restaurant from scratch. The known brands:

• NVIDIA Triton Inference Server — the default "hotel for models" on NVIDIA fleets: multi-framework backends (TensorRT, ONNX, PyTorch, TensorFlow, Python), concurrent model execution, dynamic batching and sequence batching, ensembles (pre/post-processing pipelines executed server-side), and decoupled streaming.
• TensorFlow Serving — SavedModel-native, servable versions with hot-swap; simpler when you are TF-only.
• TorchServe / torch.export pipelines — PyTorch-native; much of its team/best practice migrated into TorchFaaS and PyTorch-on-Kubernetes patterns and Edge (ExecuTorch) after the 2024 re-org.
• vLLM / SGLang / TensorRT-LLM — LLM-specialized servers with continuous batching and KV-cache management (the KV-cache topic). Generic servers waste themselves on autoregressive decode: a 300-token response is 300 dependent steps, not one tensor op — the wrong kitchen entirely.
• DIY FastAPI + uvicorn — fine for low-QPS CPU models or prototypes; you will reinvent queuing, batching, and metrics badly at scale.

The organs between the queue and the GPU:

• Static vs dynamic batching. Static: the client sends a pre-formed batch (their latency suffers, or the batch ships half-empty). Dynamic: the server collects individually-arrived requests for up to a bounded queue delay, then fires the fullest batch it can. The Triton config below shows the dial: preferred_batch_size tiers and max_queue_delay_microseconds. Microseconds of wait, multiples of throughput.
• Warm-up. First traffic after boot hits cold CUDA kernels, lazy builds, and unallocated pools — 100 ms-to-seconds outliers. Run dummy inputs before the readiness probe passes, or your p99 graph shows a spike every rollout.
• Health and drain. Liveness probes restart sick instances; readiness probes keep half-warm ones out of rotation; graceful drain finishes in-flight requests before a version swap.
• Sequencing. For stateful workloads (streaming ASR), Triton sequence batching keeps requests from one session on the same worker — the expediter remembering which ticket belongs to which table.
• Rollouts. New model version = new artifact pulled from the registry; canary a few percent of traffic (KServe-style), compare p95 and quality telemetry, then shift. The versioned-artifact guarantee is what makes this boring — which is the point.

The YAML sketch: one model, two concurrent GPU instances sharing a device, batching with a 3 ms deadline, and explicit warm-up — the restaurant's station staffing, order-rail policy, and opening prep in one file.

name: "ranker"
platform: "onnxruntime_onnx"
max_batch_size: 64
dynamic_batching {
  preferred_batch_size: [ 16, 64 ]
  max_queue_delay_microseconds: 3000
}
instance_group [
  { count: 2, kind: KIND_GPU, gpus: [ 0 ] }   # 2 concurrent instances share GPU
]
parameters: { key: "model_warm_up" value: { string_value: "input0;1x512" } }

Two questions define the kitchen floor plan: how many models per GPU, and how many replicas per model?

• Bin-packing many small models (each < 15% GPU utilization) onto one device raises total utilization: Triton concurrent execution or MPS share the device; MIG (the GPU topic's hardware partition) gives isolation instead of mere sharing.
• Dedicated GPU per replica for latency-critical large models; replica count straight from Little's law (section 3: arrival rate × service time × safety factor).
• Serverless GPU (Modal, RunPod, BentoCloud; CNCF Serverless / KServe combinations): scale-to-zero between calls, with a cold-start tax measured in seconds-to-minutes for big weights. Mitigations: weight streaming (RunPod network filesystem, TensorRT-LLM weight caching) and flash-boot snapshots — the food truck that keeps its pantry stocked so it can open in minutes, not hours.

And the honest trade-off to state in design reviews: bin-packing without MIG-style isolation reintroduces noisy-neighbor variance — your p99 now depends on someone else's traffic spike. Utilization and predictability are a dial, not a switch.

Request/response is one format; production speaks four:

1. Real-time single (recommender ranker in the ad path): low-latency RPC inside the user request — everything above. Budget: tens of milliseconds; the whole SLA is p99.
2. Batch inference (offline scoring): re-score the catalog nightly on spot GPUs or CPU pools, throughput-optimized. Often no endpoint at all — a Spark/Ray job reading the model straight from the registry. (Catering, not dine-in.)
3. Streaming (LLM chat): token-by-token responses over SSE or gRPC streaming (Triton decoupled protocols). The SLAs change identity: time-to-first-token and inter-token latency, not total latency — a 40-second generation feels instant if the first word lands in 300 ms and the rest pour at 50+ tokens/s.
4. Edge/on-device: the same registry artifacts compiled smaller (ONNX / TFLite / Core ML) run on the customer's hardware; the cloud tier then handles only fallback and training.

Picking the wrong pattern is a silent architecture bug: an LLM served like a ranker (one-shot RPC, naive batching) leaves 2–5× throughput on the table and breaks every UX assumption — the specific reason vLLM-style engines exist beside Triton rather than inside it.

In practice, stand the restaurant up in this order: versioned artifact + registry pull; one runtime with warm-up and readiness; dynamic batching with a measured delay budget; autoscaling on concurrency (set the target from your Little's-law in-flight number); then metrics — latency percentiles, queue depth, batch fill, GPU util — before you need them at 2 AM.