Kernel Optimization and Operator Fusion Concepts
Most performance stories in AI infrastructure reduce to a simple question: how much useful work happens per byte moved. Modern accelerators are extraordinarily fast at arithmetic, but arithmetic is never free if the data cannot be delivered on time. Kernels and operator fusion exist to shrink waiting. They reduce memory traffic, remove overhead between operations, and keep specialized hardware units busy.
Kernel optimization can sound like an expert-only domain, but the operator view is simpler. Serving and training costs respond strongly to a handful of repeatable patterns: attention kernels, fused elementwise operations, layout changes, quantized math, and graph-level compilation that replaces thousands of small launches with a few efficient kernels.
Smart TV Pick55-inch 4K Fire TVINSIGNIA 55-inch Class F50 Series LED 4K UHD Smart Fire TV
INSIGNIA 55-inch Class F50 Series LED 4K UHD Smart Fire TV
A general-audience television pick for entertainment pages, living-room guides, streaming roundups, and practical smart-TV recommendations.
- 55-inch 4K UHD display
- HDR10 support
- Built-in Fire TV platform
- Alexa voice remote
- HDMI eARC and DTS Virtual:X support
Why it stands out
- General-audience television recommendation
- Easy fit for streaming and living-room pages
- Combines 4K TV and smart platform in one pick
Things to know
- TV pricing and stock can change often
- Platform preferences vary by buyer
What a kernel is in practical terms
A kernel is a function launched on an accelerator that runs in parallel across many threads. It reads input tensors, performs computations, and writes output tensors. The cost of a kernel includes more than math.
- Launch overhead: scheduling and dispatch cost, especially painful when thousands of tiny kernels run per step.
- Memory reads and writes: pulling inputs from HBM or VRAM and writing outputs back.
- Synchronization: barriers and dependencies between kernels, often involving intermediate tensors.
- Memory layout: the way data is arranged affects coalesced access and cache behavior.
Many models spend a surprising fraction of time moving data and managing launches rather than performing arithmetic.
Why operator graphs create hidden overhead
Deep learning frameworks describe computation as a graph of operators: matmul, layer norm, activation functions, reshapes, and so on. A naive execution path runs each operator as a separate kernel and writes each intermediate tensor back to memory.
That approach is correct but inefficient.
- Writing intermediates to memory creates extra bandwidth pressure.
- Reading those intermediates back creates additional pressure.
- Launch overhead repeats for every operator.
- Kernel boundaries force synchronization points that reduce overlap.
Operator fusion is the idea of combining multiple operators into a single kernel so that intermediates stay in registers or shared memory and the system pays launch overhead once.
The main kinds of fusion that matter in AI workloads
Operator fusion appears in several common patterns.
| Fusion pattern | What gets fused | Why it helps | Typical risk |
|---|---|---|---|
| Elementwise chains | add, mul, gelu, silu, clamp | removes intermediate writes | numerical differences from reordering |
| Normalization + activation | layer norm or RMS norm with bias and activation | reduces bandwidth and improves cache use | shape constraints and precision sensitivity |
| Attention blocks | QKV projection, attention score, softmax, value aggregation | large speedups because attention is bandwidth-heavy | requires specialized kernels and layout discipline |
| Quantize or dequantize fusion | scale and clamp fused into matmul or attention | reduces conversion overhead | accuracy drift and calibration needs |
| Bias and residual fusion | add bias and residual connections during matmul outputs | fewer kernels, fewer writes | correctness needs careful validation |
Not every operator should be fused. Fusion wins when it reduces memory traffic without introducing a worse bottleneck or blocking hardware utilization.
A mental model for fusion: bandwidth is the tax
Elementwise operations often look cheap because they involve few FLOPs. They are expensive when they cause large tensors to be read and written multiple times.
A useful planning heuristic:
- If an operation does little math per element, it is likely bandwidth-bound.
- If it is bandwidth-bound, reducing reads and writes often matters more than optimizing arithmetic.
Fusion reduces bandwidth tax by keeping values close to compute units longer and avoiding repeated trips to HBM.
Kernel launch overhead: death by a thousand cuts
Launch overhead is hard to see if benchmarks focus only on end-to-end throughput, but it becomes visible in profiles.
Launch overhead matters most when:
- Batch sizes are small.
- Sequence lengths are short.
- Models are small enough that matmuls are not dominant.
- Serving uses micro-batches and strict latency targets.
In these cases, the GPU can show low utilization even though the workload is steady. The device is not busy because it is constantly being asked to do tiny pieces of work with gaps between them.
Fusion and graph compilation reduce the number of launches and can convert a latency-bound workload into a throughput-stable workload.
Attention kernels are the highest leverage target
Attention is a hotspot because it touches large tensors and involves both matmul and softmax-like steps. It is sensitive to layout, precision, and memory access patterns.
Modern high-performance attention kernels generally share traits.
- Blocked computation: work is chunked into blocks that fit in shared memory.
- Reduced memory traffic: intermediate attention scores are not written to global memory.
- Stable numerics: softmax is computed in a way that avoids overflow.
- Efficient use of tensor cores: when precision and layout allow.
Fused attention kernels can change the economics of context length. A system that becomes unusable at long contexts can become viable when attention is implemented with the right kernel.
Layout is performance, not style
Data layout choices determine whether memory access is coalesced and whether vectorized instructions can be used. A model that is mathematically identical can run very differently depending on layout.
Common layout issues include:
- Transposes inserted between operators that force real memory movement.
- Non-contiguous tensors that prevent efficient kernels from being selected.
- Strides that force scattered reads.
- Padding and alignment that affect tensor core usage.
Kernel optimization often begins with removing unnecessary layout changes and ensuring tensors are in a form that optimized kernels expect.
When fusion harms performance
Fusion is not automatically good. It can lose performance when it reduces parallelism or increases register pressure.
Common failure modes include:
- Register spilling: a fused kernel uses too many registers, forcing spills to memory and slowing the kernel.
- Reduced occupancy: large fused kernels reduce the number of active warps, lowering throughput.
- Limited specialization: a fused kernel may be less optimized for a particular operator than a dedicated kernel.
- Debug difficulty: diagnosing correctness issues becomes harder when operators are fused.
A disciplined approach uses fusion where it reduces memory traffic and overhead without creating a new resource bottleneck.
Quantization and kernel design are connected
Quantization is not only a model decision. It is a kernel decision.
The performance benefit of quantization depends on:
- Hardware support: whether tensor cores or matrix units accelerate the chosen format.
- Kernel availability: whether the runtime has efficient kernels for the quantized operators.
- Data movement: whether quantization reduces bandwidth or adds conversion overhead.
- Calibration and accuracy: whether the format preserves output quality in the target workload.
A naive quantization path can be slower than full precision if it introduces extra conversions or forces fallback kernels.
Framework compilation stacks: where kernels come from
Kernels used by a workload are selected and generated by a stack that can include:
- Vendor libraries: cuBLAS, cuDNN, and equivalents that provide highly tuned primitives.
- Runtime kernels: specialized attention and normalization kernels packaged with a serving stack.
- Compiler-generated kernels: kernels created by graph compilers or codegen tools.
- Custom kernels: Triton, CUDA, or other custom code used for specific bottlenecks.
Understanding which layer is responsible for a hotspot helps operators choose the right lever.
- If matmul is slow, library selection and precision choice often matter.
- If attention is slow, specialized kernels and layout are usually the lever.
- If many tiny kernels dominate, fusion and graph compilation can produce the largest gains.
Measurement discipline: what to look at in profiles
Kernel optimization should be guided by measurements that map to the bottlenecks.
Useful operator-facing measurements include:
- Kernel time distribution: which kernels dominate wall time.
- Memory throughput: achieved bandwidth versus theoretical bandwidth.
- SM occupancy and utilization: whether compute units are idle.
- Launch count: total kernel launches per request or per step.
- Tensor core utilization: whether specialized matrix units are being used.
The goal is to locate whether the workload is compute-bound, bandwidth-bound, or overhead-bound. The optimization choice changes accordingly.
The infrastructure consequences: why this is not just micro-optimizing
Kernel choices and fusion strategies influence system-level behavior.
- They change cost per token by changing throughput per GPU.
- They change tail latency by reducing launch overhead and synchronization points.
- They influence hardware choices because the optimal kernel set differs by architecture.
- They affect reliability because some kernels are more sensitive to edge cases, shape variability, and precision settings.
In practice, kernel optimization is one of the few levers that can improve both cost and user experience simultaneously.
Fusion in real serving stacks: where it shows up
In production serving, fusion typically appears in a few predictable places.
- Token sampling and logits processing: softmax, top-k, top-p, temperature scaling, and repetition penalties can be fused to avoid materializing large probability tensors multiple times.
- Decoder step scheduling: continuous batching can fuse parts of scheduling and compute so that the GPU sees a steady stream of work instead of bursty micro-kernels.
- KV cache updates: kernels that combine attention computation with cache writes can reduce extra passes over memory.
- Preprocessing and postprocessing: small CPU or GPU kernels can become bottlenecks at scale; fusing them into fewer launches removes latency jitter.
These gains are often more visible in latency-sensitive systems than in throughput-only benchmarks because they reduce synchronization points and improve stability under concurrency.
Tooling patterns: when to trust libraries and when to go custom
Vendor libraries provide excellent primitives, but some workloads need kernels that libraries do not expose directly.
- Libraries win for standard matmul and convolution primitives, especially when shapes are well supported.
- Custom kernels win when an operation is structurally unique, such as a specialized attention variant, an unusual quantization scheme, or a fused sequence of small operators.
- Compiler-generated kernels win when shapes vary and the system benefits from automated specialization without hand-tuning every case.
A common operator strategy is to accept library primitives for the large dense parts, then selectively adopt custom kernels for the small but frequent bottlenecks that dominate tail latency.
Correctness guardrails: performance without reliability is a trap
Kernel and fusion changes can introduce subtle correctness problems.
- Numerical drift: changing operation order can alter rounding and accumulate error.
- Precision mismatches: mixed precision paths can behave differently across hardware generations.
- Edge-case shapes: rare shapes can trigger incorrect code paths in specialized kernels.
- Nondeterminism: race conditions or atomic behavior can change outputs across runs.
Guardrails reduce risk.
- Golden tests: fixed prompts and expected outputs checked in CI for the serving stack.
- Shape fuzzing: randomize shapes within allowed ranges to probe compiler and kernel boundaries.
- Canary rollout: route a small fraction of traffic to new kernels with tight monitoring of error rates and output distribution.
- Fallback plans: keep a stable kernel path available for rapid rollback.
These guardrails preserve the real goal of optimization: stable capacity and predictable quality.
Keep exploring on AI-RNG
- In this category
- Hardware, Compute, and Systems Overview
- Cluster Scheduling and Job Orchestration
- Serving Hardware Sizing and Capacity Planning
- Model Compilation Toolchains and Tradeoffs
- Quantization Formats and Hardware Support
- Series and navigation
- Infrastructure Shift Briefs
- Tool Stack Spotlights
- AI Topics Index
- Glossary
More Study Resources
- Category hub
- Hardware, Compute, and Systems Overview
- Related
- Cluster Scheduling and Job Orchestration
- Serving Hardware Sizing and Capacity Planning
- Model Compilation Toolchains and Tradeoffs
- Quantization Formats and Hardware Support
- Infrastructure Shift Briefs
- Tool Stack Spotlights
- AI Topics Index
- Glossary
Books by Drew Higgins
