GPU Fundamentals: Memory, Bandwidth, Utilization
GPUs sit at the center of modern AI because they are built to run a vast number of small, similar operations in parallel. The moment you start paying for serious GPU time, the conversation shifts from “how fast is this chip on paper” to “how much of the chip are we actually using.” That gap between theoretical peak and achieved throughput is where most infrastructure cost is won or lost.
This article explains GPU performance in the way operators and builders need it: as a relationship between memory movement, arithmetic work, and how well the workload keeps the device busy. The goal is not to memorize part names, but to build an intuition you can use when training is slow, inference latency is spiky, or your GPU utilization graph looks impressive while your tokens-per-second does not.
Flagship Router PickQuad-Band WiFi 7 Gaming RouterASUS ROG Rapture GT-BE98 PRO Quad-Band WiFi 7 Gaming Router
ASUS ROG Rapture GT-BE98 PRO Quad-Band WiFi 7 Gaming Router
A flagship gaming router angle for pages about latency, wired priority, and high-end home networking for gaming setups.
- Quad-band WiFi 7
- 320MHz channel support
- Dual 10G ports
- Quad 2.5G ports
- Game acceleration features
Why it stands out
- Very strong wired and wireless spec sheet
- Premium port selection
- Useful for enthusiast gaming networks
Things to know
- Expensive
- Overkill for simpler home networks
The practical model: work, movement, and waiting
A GPU does three broad things during AI workloads:
- It performs math on tensors.
- It moves data through a memory hierarchy.
- It waits on something else: CPU scheduling, synchronization, memory dependencies, I/O, or communication.
When people say “this workload is GPU-bound,” they usually mean one of two things:
- Compute-bound: arithmetic units are the limiter. Adding memory bandwidth does not help much.
- Memory-bound: the limiter is getting data to the arithmetic units fast enough. The math units are idle part of the time.
The most useful mental tool here is the roofline concept: achievable performance is bounded by whichever is smaller, the device’s compute peak or its memory bandwidth multiplied by arithmetic intensity (how much math you do per byte moved). You do not need the graph to benefit from the logic. If your kernels do little math per byte, more FLOPS will not save you; you need better locality, better fusion, or better data layout.
Memory hierarchy: why “VRAM size” is only the beginning
People shopping for GPUs often lead with VRAM capacity. Capacity matters, but performance often lives in the hierarchy:
- Registers: extremely fast, per-thread storage. Excess register use can reduce occupancy.
- Shared memory: programmer-managed on-chip memory used to stage data and reduce global memory traffic.
- L1 and L2 caches: hardware-managed, helpful when access patterns have reuse or locality.
- High-bandwidth device memory: HBM or GDDR. This is what most people mean by VRAM.
- Host memory and storage: CPU RAM and disk. Transfers here are orders of magnitude slower than on-device access.
AI workloads frequently look “simple” at the model level while being complex at the memory level. A transformer block is a chain of matrix multiplies, elementwise ops, layer norms, and attention. Whether it is fast depends on whether intermediate tensors stay on-chip, how often they spill to device memory, and how effectively kernels reuse values rather than reloading them.
Bandwidth is a budget, not a spec sheet number
Peak bandwidth numbers assume ideal access patterns. In real systems, bandwidth is shaped by:
- Access coalescing: do threads access contiguous regions or scattered addresses.
- Stride and alignment: misaligned or strided access can waste transactions.
- Cache hit rates: if data can be served from L2, global memory bandwidth pressure drops.
- Contention: multiple streams or kernels fighting for the same memory resources.
- Paging and oversubscription: if the working set does not fit, performance can collapse.
A common trap is to observe low GPU compute utilization and conclude “the GPU is not being used.” Often it is being used to move data inefficiently, so arithmetic units sit idle while memory pipelines are saturated.
Utilization: which numbers actually matter
“GPU utilization” is ambiguous. Different tools report different notions of busy. For AI, you want a small set of operator-facing signals that map to causes:
- Achieved SM occupancy: how many warps are active relative to what the hardware could host.
- Tensor core utilization or math throughput: how much of the matrix math path you are using.
- Memory throughput: achieved bandwidth as a fraction of peak, and whether it is read- or write-heavy.
- Kernel launch and scheduling overhead: small kernels can waste time on launch costs and synchronization.
- Host-to-device transfer time: whether the pipeline is starved by data movement over PCIe or similar links.
- Time in communication collectives: in multi-GPU training, all-reduce and related steps can dominate.
High occupancy is not a guarantee of high performance, and low occupancy is not always bad. But occupancy is a valuable diagnostic because it reveals when the GPU cannot keep enough work in flight to hide memory latency.
Why an AI workload can show high utilization but low throughput
It is possible to “keep the GPU busy” while still wasting money. Common reasons include:
- The GPU is busy running inefficient kernels with poor memory locality.
- The GPU is busy waiting on synchronization barriers between many small kernels.
- The GPU is busy on communication because the model is split across devices or nodes.
- The GPU is busy on non-core work like format conversion, padding, or unnecessary copies.
Operators should connect utilization to an outcome metric: tokens per second, images per second, examples per second, or cost per output. Utilization is a means, not a goal.
Compute-bound vs memory-bound in real AI workloads
Many AI building blocks are compute-heavy on paper, but become memory-limited in practice because intermediate tensors are large and reuse is limited.
Matrix multiply and attention
Dense matrix multiply is often compute-friendly because each element loaded can participate in many multiply-accumulate operations. That is why GEMM libraries are so optimized. But attention can be tricky. The score matrix and softmax path can introduce memory pressure, and in decoder-only inference the key-value cache becomes a dominant memory footprint. You can be limited by capacity and bandwidth even when the math itself is not extreme.
Elementwise chains and normalization
Layer norm, activation functions, bias adds, and similar elementwise ops can become bandwidth-limited because they do little math per byte and touch large tensors. This is where kernel fusion matters. If you can combine several elementwise steps into one kernel, you reduce memory traffic and kernel launch overhead.
Embeddings and sparse lookups
Embedding lookups can be bounded by memory and latency because access patterns can be irregular and reuse can be limited. Here, cache behavior and batching shape performance more than raw FLOPS.
Keeping the GPU fed: the hidden pipeline outside the device
A GPU can only run what you deliver to it. Many “GPU performance” issues originate upstream:
- Data loading and preprocessing: slow decoding, augmentation, or tokenization on the CPU.
- Small batch sizes: not enough parallel work to fill the device, especially in training.
- Excessive framework overhead: dispatch costs, graph breaks, Python-level control flow.
- Synchronization points: unnecessary device synchronizations that prevent overlap.
- Inefficient dataloader settings: not enough workers, lack of pinned memory, no prefetch.
For training, a strong pattern is to overlap CPU preprocessing with GPU compute, and overlap host-to-device copies with compute where possible. Pinned host memory can improve transfer efficiency, and asynchronous copies let the GPU work while data is in flight.
For inference, the key is to understand the service-level objective. If you are latency-sensitive, you may choose smaller batches, which reduces efficiency. The system problem becomes: how do we reclaim efficiency without violating latency targets. Techniques include dynamic batching, request coalescing, and using multiple model replicas.
Precision, tensor cores, and what “supported” really means
Many accelerators have specialized datapaths for lower-precision math. Using them effectively is a stack-wide choice: model, framework, kernel library, and deployment settings.
- Training commonly uses BF16 or FP16 for speed while keeping stability with loss scaling or similar techniques.
- Inference can often use INT8 or other quantized formats, reducing bandwidth and improving throughput.
But “supports INT8” does not mean your model will run fast in INT8. The operator set must be supported, calibration must be sensible, and the framework must select optimized kernels rather than falling back to slow paths. A practical approach is to test end-to-end throughput on your real model and input shapes, then profile to see which kernels dominate time.
Multi-GPU: bandwidth and latency do not scale for free
Once you use multiple GPUs, communication becomes part of the performance picture. Training large models frequently uses data parallelism, tensor parallelism, pipeline parallelism, or combinations. Each adds communication steps:
- All-reduce for gradient aggregation.
- All-gather and reduce-scatter for sharded tensors.
- Point-to-point transfers for pipelined stages.
Interconnect choice matters. Even within one server, the topology can determine whether GPUs can share data efficiently. Across nodes, networking and collective libraries become decisive. If your scaling curve flattens early, it is often a sign that communication is overtaking compute, or that load imbalance is creating idle time.
A diagnostic workflow that prevents guesswork
When performance is off, the fastest teams follow a repeatable workflow:
- Pick one objective metric: tokens per second, step time, p95 latency, or cost per output.
- Profile at the kernel level to identify where time is spent.
- Determine whether the dominant kernels are compute-bound or memory-bound.
- Fix the dominant bottleneck first, then re-measure.
Some fixes are model-level (sequence length, batch sizing, caching strategy). Others are kernel-level (fusion, better layout, better operator selection). Others are system-level (data pipeline, concurrency, replica strategy). The point is not to “optimize everything,” but to remove the limiter that currently dictates cost.
The infrastructure consequences: why this matters for AI-RNG
GPU fundamentals are not just trivia. They decide whether a deployment needs one server or ten, whether a cluster is stable under load, and whether your cost model holds when traffic spikes.
- If you do not understand memory limits, you will size VRAM for weights and forget the working set, leading to paging, failures, or latency cliffs.
- If you do not understand bandwidth, you will chase peak FLOPS while remaining memory-limited and paying for unused compute.
- If you do not understand utilization, you will interpret dashboards incorrectly and miss the true bottleneck.
The deeper point is that AI capability is increasingly constrained by infrastructure realities. Understanding how memory, bandwidth, and utilization interact is one of the clearest ways to turn AI spending into dependable output.
Keep exploring on AI-RNG
- Hardware, Compute, and Systems Overview: Hardware, Compute, and Systems Overview
- Nearby topics in this pillar
- Latency-Sensitive Inference Design Principles
- Benchmarking Hardware for Real Workloads
- Accelerator Landscape: GPUs, TPUs, NPUs, ASICs
- Training vs Inference Hardware Requirements
- Cross-category connections
- Serving Architectures: Single Model, Router, Cascades
- Use-Case Discovery and Prioritization Frameworks
- Series and navigation
- Infrastructure Shift Briefs
- Tool Stack Spotlights
- AI Topics Index
- Glossary
More Study Resources
- Category hub
- Hardware, Compute, and Systems Overview
- Related
- Latency-Sensitive Inference Design Principles
- Benchmarking Hardware for Real Workloads
- Accelerator Landscape: GPUs, TPUs, NPUs, ASICs
- Training vs Inference Hardware Requirements
- Infrastructure Shift Briefs
- Tool Stack Spotlights
- AI Topics Index
- Glossary
Books by Drew Higgins
Bible Study / Spiritual Warfare
Ephesians 6 Field Guide: Spiritual Warfare and the Full Armor of God
Spiritual warfare is real—but it was never meant to turn your life into panic, obsession, or…