TensorRT: Compiling Models for Peak GPU Inference

You exported a clean ONNX file (previous topic). ONNX Runtime runs it everywhere — great for portability.

Then the product team turns on the faucet:

Portable execution is generic. It walks the graph mostly as written. But GPUs spend a surprising amount of their time not computing at all — they spend it moving data in and out of memory. Tiny kernels doing math on tiny tensors means the GPU is mostly a delivery truck, not a factory.

So the question becomes

A lot faster. The tool that gets you there is TensorRT.

TensorRT is simply

Two phases, and mixing them up is the beginner trap:

• Build phase (slow, once): the builder takes your graph — via ONNX (e.g. torch.onnx.export), the Model Optimizer from PyTorch/TF, or the low-level Network Definition API — studies it, experiments, and bakes everything into an engine (a .plan file).
• Run phase (fast, many times): the runtime just loads the engine and executes the pre-chosen plan.

An analogy in one line: framework execution is a musician sight-reading the score at every concert; a TensorRT engine is a rehearsal where the performance was already worked out note by note.

What the builder actually does during that expensive rehearsal:

• Fuses layers: Conv+BN+ReLU, MatMul+bias+activation become one kernel; parallel branches get horizontally fused too. The win is fewer trips to HBM (GPU memory) — the memory-bound lesson from the CUDA topic.
• Auto-tunes kernels: it literally times multiple tile/config strategies per layer and keeps the fastest. Builds take minutes-to-hours, so you cache and version the build like a binary.
• Chooses precision per layer: FP32/TF32/FP16/BF16/INT8/FP8/INT4 wherever you allow it — mixed across one graph.
• Reorders memory layouts (NCHW -> internal formats, vectorized loads) and eliminates transposes.

And here is the price tag, built into the design: the engine is opaque and not portable across GPU architectures (sm_86, sm_90, sm_100...) or major TRT versions. Your registry must store (ONNX source + build recipe + per-SM engines), not just an engine.

Fusion, counted in memory trips. Take a Conv-BatchNorm-ReLU block. Unfused, each kernel must write its output to GPU memory and the next must read it back:

• Conv writes the feature map → BN reads it, writes it → ReLU reads it, writes it.
• That is 3 writes + 2 reads = 5 round-trips to slow-ish HBM for the same numbers.
• Fused into one kernel: the intermediate values never leave the fast on-chip scratchpad. 1 write, 0 extra reads.

For small layers, arithmetic is cheap and memory traffic is the whole bill — so deleting round-trips can multiply speed even though the math is identical.

INT8 calibration, with a tiny number line. A layer's activations are floats, say values between -8.0 and +8.0 (measured, not guessed — that is what calibration does). INT8 gives you 256 slots.

• Scale done right: 16.0 / 255 ≈ 0.063 per slot. A signal of 0.5 rounds to slot 8 — preserved.
• Scale done blindly (assuming the range is -256..+256): 512 / 255 ≈ 2.0 per slot. That same 0.5 signal rounds to 0 — gone. Same weights, same network, silently destroyed accuracy.

Lesson: INT8 is a measurement task before it is a compression task, and the measurement must look like real traffic. Next, see the whole pipeline in one picture.

The whole topic compresses into this picture — an expensive, hardware-specific bake on the left, a cheap repeatable serve on the right:

code
Copy
   BUILD (slow: minutes-hours, once per GPU SKU)      RUN (fast: every request)
   +--------------------------------------------+     +---------------------+
   | ONNX graph   [Conv][BN][ReLU]... [Softmax] |     | .plan engine        |
   | calibration dataset (hundreds of samples)  | --> | fused kernels,      |--> GPU
   | builder: fuse, time kernels, pick int8     |     | int8 scales baked,  |    p99 < 10ms
   | scales, choose layouts                     |     | layouts chosen      |
   +--------------------------------------------+     +---------------------+
        CI rebuilds this per sm_86 / sm_90 / sm_100       runtime makes no decisions

Read the arrows as a trade:

• Portability axis: ONNX runs anywhere but is executed generically → the fallback path when hardware changes.
• Speed axis: the engine runs great but only on one chip family → the latency-critical path.

You do not choose "correct vs fast." You choose where the slow thinking happens: at build time (TensorRT) instead of at request time (framework). The mermaid diagram above the sections shows the same pipeline: source model → ONNX → builder (+calibration data) → engine → runtime → GPU, with ONNX Runtime as the portability fallback edge.

Carry this image through the rest of the topic: a chef who must feed thousands of guests per hour, in one specific kitchen.

• Framework eager execution = reading the recipe during service, tasting as you go, washing the pot between every course. Works in any kitchen. Slow everywhere.
• TensorRT build = a prep chef studies the menu once and rewrites it for THIS kitchen: merge three simmering pots into one (fusion), buy the fastest knife for each cut (kernel auto-tuning), pre-portion ingredients into the right trays (memory layouts), decide which dishes tolerate coarse salt and which need fine (per-layer precision). Service becomes assembly — nothing left to decide.

Now the punchlines the analogy was built for:

• Move to a bigger kitchen (a new GPU SKU, L4 → H100)? All the prep is wasted; re-prep from the menu. Engines are hardware receipts.
• The stove model changed (TRT 8 → 10)? Re-prep. Engine formats change across major versions.
• A dish needs a torch the kitchen does not own (an unsupported operator)? You can bring your own tool — that is a C++ plugin — it works, but it is the highest-maintenance option in the building.
• The prep chef portions based on the tasting menu they were given. If the tasting was Monday brunch and Friday is the dinner rush, portions are wrong, silently. That is exactly calibration data: it must taste like production traffic.
• The menu card says "banquets for 1–64 guests, ideal at 8" — that is an optimization profile (min/opt/max shapes). Book 65 and the kitchen simply refuses to cook.

Keep the chef; the next sections are all ingredients from this kitchen.

The builder can run each layer at a different numeric precision — coarse salt vs fine salt, per dish. The menu, in order you would actually try them:

• FP16/BF16: usually 2-4x faster with negligible loss; the default first switch. BF16 on Ampere+ widens the dynamic range, which matters for training-derived graphs with big-magnitude activations.
• INT8 post-training quantization (PTQ): run a calibration pass over a representative dataset (hundreds of images/samples) with entropy or percentile observers to set per-layer scales; then validate accuracy on held-out data. Expect a small, measured drop. Uncalibrated INT8 can collapse quantizers in detection models — the worked example above shows why (a 0.5 signal rounded to zero).
• QAT (quantization-aware training): fake-quant nodes during training recover INT8 accuracy for hard cases — object detection, and LLM weight-only INT4/AWQ-style setups.
• FP8 (Hopper/Ada+): the two flavors E4M3/E5M2 with per-tensor scaling; near-FP16 accuracy at roughly 2x math throughput; delayed scaling recipes were standardized in the Model Optimizer / TensorRT-Model-Optimizer repo (2024-2026).
• Per-layer precision control: builder flags plus setPrecision overrides. House rule: keep sensitive reductions (Softmax, LayerNorm) in FP16/FP32 — they amplify rounding errors.

Which one you pick is not a taste question: it is a SLO + accuracy-validation question. Build at the cheapest precision that still passes the eval.

Real traffic is ragged: batch 1 at 3 am, batch 64 at peak, sequence lengths all over. A compiled engine hates surprises — so TensorRT 8+ ships optimization profiles: each dynamic dimension gets min/opt/max bounds (say batch 1-64, optimal 8), and kernels are compiled around the opt shape. One engine then serves the whole envelope. Beyond the bounds, inference simply fails — size profiles to your true traffic envelope, not your wishlist.

The runtime choreography, in order:

1. Deserialize the engine.
2. Create execution contexts — each owns its workspace memory; multiple contexts = concurrent batch streams on one GPU.
3. Bind IO on CUDA streams → execute_async_v3.
4. CUDA graphs further cut kernel-launch overhead for fixed-shape steady state.

The operational knobs people actually fight with: tensor memory and workspace sizes are the usual OOM causes — document them for your autosizer, the same way you would document a JVM heap.

And the escape hatch: unsupported ops become C++/TensorRT plugins (custom kernels). Honest ranking — prefer ONNX graph surgery or TRT-supported decompositions first; plugins work, but they are the highest-maintenance code in your inference stack (the prep chef bringing their own torch means you now maintain the torch too).

Large language models generate one token at a time — a loop, not one forward pass — so the classic "freeze the graph and bake it" story needs a remix. NVIDIA ships TensorRT-LLM: a Python framework that generates fused kernels for autoregressive serving — FlashAttention-class attention, in-flight batching, KV cache with paged management, speculative decoding, quantized weights (INT4 AWQ/GPTQ, FP8) — compiled into engines. In production it is orchestrated by Triton's tensorrtllm backend and Dynamo (the NVIDIA inference framework, 2025 — unrelated to TorchDynamo; say which one you mean in interviews).

Meanwhile core TensorRT reached a "1.0 era" (TRT 10/11 line, 2024-2025):

• Unified Strongly Typed mode: the build respects the graph's dtypes end-to-end — fewer silent precision surprises where the builder "helpfully" reinterprets your floats.
• Split trt/trtllm ecosystems, and hardware-forward support.

The practical guidance never changed, and it is the whole discipline in one sentence: build in CI against target SKUs, calibrate with real data, validate accuracy per release.