IO Bottlenecks and Throughput Engineering
Compute gets the headlines, but most large AI systems are limited by the movement of data. The reason is simple: the math scales faster than the plumbing. A modern accelerator can consume tensors at a pace that turns small inefficiencies into large costs. When the input pipeline falls behind, the expensive part of the system waits. Utilization drops, wall-clock time stretches, and the budget burns without improving results.
Throughput engineering is the discipline of making data arrive where it needs to be, at the right time, with predictable tail behavior. It is not one trick. It is a stack: storage formats, filesystems, networking, queues, concurrency, caching, and measurement. IO bottlenecks show up in training, evaluation, indexing, and serving, each with slightly different failure signatures.
Gaming Laptop PickPortable Performance SetupASUS ROG Strix G16 (2025) Gaming Laptop, 16-inch FHD+ 165Hz, RTX 5060, Core i7-14650HX, 16GB DDR5, 1TB Gen 4 SSD
ASUS ROG Strix G16 (2025) Gaming Laptop, 16-inch FHD+ 165Hz, RTX 5060, Core i7-14650HX, 16GB DDR5, 1TB Gen 4 SSD
A gaming laptop option that works well in performance-focused laptop roundups, dorm setup guides, and portable gaming recommendations.
- 16-inch FHD+ 165Hz display
- RTX 5060 laptop GPU
- Core i7-14650HX
- 16GB DDR5 memory
- 1TB Gen 4 SSD
Why it stands out
- Portable gaming option
- Fast display and current-gen GPU angle
- Useful for laptop and dorm pages
Things to know
- Mobile hardware has different limits than desktop parts
- Exact variants can change over time
The IO stack in plain terms
It helps to name the path a byte takes.
- Data is stored somewhere: object storage, a distributed filesystem, a database, or local disks.
- The network moves it: switches, NICs, TCP or RDMA, congestion control, rate limits.
- The host receives it: kernel buffers, page cache, filesystem metadata, CPU copies.
- The runtime transforms it: decompression, parsing, tokenization, feature extraction.
- The accelerator consumes it: DMA transfers, pinned memory, staging buffers, device memory.
A bottleneck can live at any step. Teams often fix the visible symptom, such as adding more data loader workers, and then wonder why throughput stays flat. The limiter is usually elsewhere, often in a shared resource like metadata operations, small-file overhead, or network congestion.
Utilization is an outcome of balance
A simple mental model is that every pipeline stage must supply the next stage at the required rate.
- If storage is slow, everything downstream waits.
- If parsing is slow, adding storage bandwidth does not help.
- If host-to-device transfers are slow, faster CPUs and disks do not help.
The goal is not “maximum throughput at any cost.” The goal is stable throughput at the level the workload needs, with headroom for burst and predictable behavior under load.
Training pipelines: the classic IO trap
Training is where IO failures are most expensive, because the job is long and the compute is costly.
Common bottlenecks in training include:
- Small-file overhead
- Millions of tiny files can saturate metadata servers long before bandwidth is used.
- The symptom is high latency per open call and low aggregate throughput.
- Shuffling and random access
- Random sampling patterns create IO that defeats read-ahead and caching.
- The symptom is low disk throughput despite idle disks, because the access pattern is not sequential.
- Decompression and parsing
- Tokenization or image decode can become CPU-bound, starving the accelerator.
- The symptom is high CPU utilization in data loader workers, with modest disk and network use.
- Cross-region reads
- Pulling data over a wide-area link can create unpredictable tail latency.
- The symptom is periodic stalls that correlate with network variability.
- Contention with checkpointing
- Training reads data while also writing checkpoints.
- The symptom is throughput cliffs at checkpoint boundaries.
The practical implication is that IO design belongs in the training plan from day one. A dataset format decision can be as important as a model architecture decision, because it determines whether the system can feed the accelerators consistently.
Throughput is not only bandwidth
Teams often talk about “bandwidth” as if it is the only metric. Two other dimensions matter just as much.
- IOPS and metadata rate
- Many workloads are limited by operations per second: opens, stats, directory listings, small reads.
- A dataset with millions of small objects can fail on IOPS limits while using little bandwidth.
- Tail latency
- A pipeline that averages fast but has long stalls can destroy utilization.
- In distributed training, the slowest worker often controls progress, so tail behavior matters more than mean behavior.
Throughput engineering is partly about raising the ceiling, but also about tightening the distribution so the system behaves predictably.
Measurement that actually isolates the bottleneck
Without measurement, IO work becomes guesswork. Useful signals tend to be concrete and layered.
- End-to-end step time and accelerator utilization
- When utilization drops, the pipeline is falling behind or stalling.
- Queue depth between stages
- If the “ready batch” queue is empty, upstream is slow.
- If it is full, downstream is slow or blocked.
- Host CPU breakdown
- Parsing, decompression, and preprocessing often dominate.
- Storage metrics
- Read throughput, read latency, metadata ops, error rates, rate-limit counters.
- Network metrics
- Throughput, retransmits, congestion, p99 latency, dropped packets.
- System-level signals
- Page cache hit rates, context switch rates, disk wait times, memory pressure.
A small but powerful technique is to run a “synthetic loader” that bypasses parsing and reads raw bytes at the intended access pattern. If raw reads are slow, storage or network is the problem. If raw reads are fast but the pipeline is slow, parsing and transformation are the problem.
Data formats are throughput policies
The file format is not a neutral container. It encodes assumptions about access patterns.
Patterns that usually improve throughput:
- Large, sequentially readable shards
- Combine many samples into larger containers to reduce metadata overhead.
- Align shards with typical batch and shuffle behavior.
- Columnar or block-based layouts for structured data
- Avoid reading fields that are not needed.
- Tokenized or preprocessed datasets when CPU is scarce
- Shift expensive transforms to an offline pipeline and store ready-to-consume artifacts.
- Explicit versioning and manifests
- Make it easy to verify integrity and avoid partial datasets.
Patterns that often hurt:
- Many tiny objects in object storage without a strong index strategy
- Excessive per-sample compression that adds CPU overhead
- Formats that require expensive random seeks for common access patterns
Throughput engineering often looks like “boring” data plumbing, but it shapes the feasibility of a training plan.
Caching is a strategy, not a miracle
Caching works when the access pattern has reuse. Many training pipelines are designed to avoid reuse, because shuffling is used to improve generalization. That means caching must be approached carefully.
Practical caching patterns include:
- Hot shard caching
- Cache frequently accessed shards locally, such as recent shards in an epoch.
- Staging to local NVMe
- Preload a window of shards ahead of time.
- Host memory caching for tokenized batches
- Keep ready-to-consume batches in RAM for a short window, reducing repeated transforms.
- Shared cache layers
- A cluster-level cache can reduce repeated downloads from object storage, but only if it does not become a new contention point.
Caching is most effective when it is paired with measurement: cache hit rates, eviction rates, and the impact on tail latency. Blind caching can produce costs without benefits.
Network bottlenecks and the “shared fabric” reality
At scale, the network becomes a shared resource. Even if a single job seems fine in isolation, multiple jobs can interfere.
Common network-related constraints include:
- East-west congestion inside the cluster during distributed training
- North-south traffic to object storage or external data sources
- Rate limits on object storage endpoints
- Load balancer bottlenecks in serving paths
Throughput engineering needs a view of the whole fabric. If checkpoint uploads, dataset reads, and interconnect collectives peak at the same time, the system will produce predictable stalls.
Coordination helps. Staggering checkpoint windows, shaping bulk transfers, and isolating traffic classes can keep interactive inference stable while training runs in the background.
Host-to-device transfers: the overlooked choke point
Even with fast storage and networks, the path from host memory to accelerator memory can bottleneck.
The transfer path depends on:
- PCIe generation and topology
- NUMA placement and pinning
- Pinned memory usage and allocation strategy
- Copy count and serialization overhead in the runtime
- Kernel launch and synchronization behavior
Symptoms of host-to-device bottlenecks include:
- Storage and CPU look healthy, but device utilization remains low.
- Profilers show time spent waiting on copies or synchronization.
- Increasing data loader workers does not help.
Fixes are often architectural: reduce copy count, use pinned memory correctly, overlap transfers with compute, and ensure the data pipeline is aligned with the device’s preferred batch shapes.
Serving pipelines: throughput with strict latency
Inference has a different constraint profile. Serving must deliver consistent low latency, even under burst load.
Serving IO bottlenecks often come from:
- Model weight loading and warmup during deployment events
- KV cache pressure that forces frequent memory movement
- Tokenization and preprocessing on the critical path
- Logging and telemetry that block the request path
- Downstream tool calls that introduce long tail behavior
Throughput engineering in serving is about isolating bulk work from the critical path. Preload what can be preloaded, batch what can be batched without harming latency, and treat logging as an asynchronous stream.
A practical toolkit for throughput engineering
The most reliable improvements tend to be structural.
- Reduce metadata pressure
- Use larger shards, manifests, and fewer opens.
- Make access patterns predictable
- Prefer sequential reads where possible, and design shuffling around shard-level randomness.
- Move transforms off the critical path
- Pre-tokenize, pre-encode, or precompute features when it does not harm flexibility.
- Overlap stages
- Use prefetch windows and bounded queues so compute and IO run concurrently.
- Engineer tail behavior
- Measure p95 and p99, and address stalls, not only averages.
- Isolate traffic classes
- Keep checkpointing and bulk transfers from destabilizing interactive serving.
Throughput engineering is not glamorous, but it is how AI systems become reliable infrastructure instead of expensive experiments.
- Hardware, Compute, and Systems Overview: Hardware, Compute, and Systems Overview
- Nearby topics in this pillar
- Storage Pipelines for Large Datasets
- Checkpointing, Snapshotting, and Recovery
- Multi-Tenancy Isolation and Resource Fairness
- Hardware Monitoring and Performance Counters
- Cross-category connections
- Operational Costs of Data Pipelines and Indexing
- Cost Anomaly Detection and Budget Enforcement
- Series and navigation
- Infrastructure Shift Briefs
- Tool Stack Spotlights
- AI Topics Index
- Glossary
More Study Resources
- Category hub
- Hardware, Compute, and Systems Overview
- Related
- Storage Pipelines for Large Datasets
- Checkpointing, Snapshotting, and Recovery
- Multi-Tenancy Isolation and Resource Fairness
- Hardware Monitoring and Performance Counters
- Operational Costs of Data Pipelines and Indexing
- Cost Anomaly Detection and Budget Enforcement
- Infrastructure Shift Briefs
- Tool Stack Spotlights
- AI Topics Index
- Glossary
Books by Drew Higgins
Christian Living / Encouragement
God’s Promises in the Bible for Difficult Times
A Scripture-based reminder of God’s promises for believers walking through hardship and uncertainty.
