Interconnects and Networking: Cluster Fabrics
Modern AI clusters do not behave like a pile of independent GPUs. The moment a workload spans multiple devices, performance becomes a question of how fast devices can exchange data and how predictably that exchange happens under contention. Interconnects inside a node and networking between nodes form the fabric that turns raw compute into a coherent system.
The fabric is where scaling claims either become real or fall apart. Training can stall on collective communication. Serving can suffer tail latency from noisy neighbors and congested links. Data pipelines can compete with training traffic and cause periodic slowdowns. A clear view of cluster fabrics turns “it feels slow” into a measurable diagnosis and a targeted fix.
High-End Prebuilt PickRGB Prebuilt Gaming TowerPanorama XL RTX 5080 Gaming PC Desktop – AMD Ryzen 7 9700X Processor, 32GB DDR5 RAM, 2TB NVMe Gen4 SSD, WiFi 7, Windows 11 Pro
Panorama XL RTX 5080 Gaming PC Desktop – AMD Ryzen 7 9700X Processor, 32GB DDR5 RAM, 2TB NVMe Gen4 SSD, WiFi 7, Windows 11 Pro
A premium prebuilt gaming PC option for roundup pages that target buyers who want a powerful tower without building from scratch.
- Ryzen 7 9700X processor
- GeForce RTX 5080 graphics
- 32GB DDR5 RAM
- 2TB NVMe Gen4 SSD
- WiFi 7 and Windows 11 Pro
Why it stands out
- Strong all-in-one tower setup
- Good for gaming, streaming, and creator workloads
- No DIY build time
Things to know
- Premium price point
- Exact port mix can vary by listing
Intra-Node Versus Inter-Node: Two Different Games
Fabric decisions start with a split:
- Intra-node interconnect connects GPUs to each other and to the host inside a single machine.
- Inter-node networking connects machines to each other.
Intra-node links often have lower latency and higher bandwidth than inter-node links, and they are less exposed to congestion from unrelated traffic. That makes intra-node parallelism attractive. The catch is that the size of a single node is limited. Inter-node scale is where large training runs live.
A common cluster pattern is “fast island, slower ocean.” GPUs talk quickly inside a node, then talk more slowly across nodes. Parallelism strategies that respect this structure usually win. Strategies that assume all links are equivalent tend to produce disappointing scaling.
What the Fabric Must Carry in AI Workloads
AI workloads move a few dominant kinds of data:
- Gradients and partial reductions during training.
- Activations or partial results in pipeline or tensor-parallel setups.
- Parameter shards and optimizer state in sharded training.
- Request and response traffic, plus cache coordination, in serving systems.
- Dataset shards and feature artifacts in data pipelines.
Training traffic is often bulk and periodic. Serving traffic is often small messages with strict latency sensitivity. Mixing these on the same links without isolation is a recipe for tail-latency explosions and hard-to-debug performance cliffs.
The practical implication is that fabric design is both an engineering and a policy problem: link speed matters, and so do traffic classes, queuing behavior, and admission control.
Inside the Node: PCIe, GPU Links, and Topology Awareness
Most nodes use a host bus for device attachment. PCIe is the common baseline. It is flexible, widely supported, and improves each generation, but it is not designed specifically for all-to-all GPU traffic under heavy load. Many high-end AI nodes add dedicated GPU-to-GPU links and switching.
Topology awareness matters because “connected” is not the same as “equally connected.” A node can have:
- GPUs that share a fast link to each other.
- GPUs that must route traffic through the host.
- Non-uniform paths where some pairs have higher bandwidth than others.
Communication libraries and parallelism frameworks often attempt to detect and exploit topology. When they cannot, the workload may appear to scale until a certain device count, then flatten or regress as the worst links dominate.
Useful mental models:
- Treat the node as a graph of links with different capacities.
- Expect the slowest edge in a critical collective to set the pace.
- Watch for “islands” where a subset of GPUs communicate well internally but poorly to others.
Even without brand-specific knowledge, this perspective helps decide whether to prioritize fewer, larger nodes or more, smaller nodes with faster networking.
Between Nodes: Ethernet, RDMA, and Why Loss Matters
Inter-node networking ranges from standard Ethernet to RDMA-capable fabrics. The meaningful distinctions are:
- Latency and bandwidth per link.
- How congestion is handled.
- Whether remote direct memory access is supported and stable.
- How sensitive the fabric is to packet loss and reordering.
Distributed training often uses collective operations that can be extremely sensitive to tail behavior. A single slow link or retransmission event can stall a whole step. When the cluster is large, the probability that some link is having a bad day increases, so the system needs both speed and resilience.
Loss matters because many high-performance paths assume very low loss. When loss occurs, recovery mechanisms can introduce large stalls. That is one reason AI clusters often treat the network as a dedicated environment with carefully controlled traffic, not as a general-purpose shared corporate network.
Collectives: The Hidden Scheduler of Distributed Training
Many training stacks rely on a small set of communication patterns:
- All-reduce combines gradients across devices.
- All-gather shares shards so each device can proceed with a complete view.
- Reduce-scatter and gather are used in sharded schemes to move less data per step.
These operations can be implemented with different algorithms, such as ring-based methods or tree-based methods. The important takeaway is not the exact algorithm but the fact that communication cost grows with:
- the amount of data exchanged
- the number of participants
- the topology and link speeds
- the degree of synchronization required
When communication becomes a large fraction of step time, scaling becomes expensive. The cluster is paying for more GPUs that spend more time waiting.
A useful diagnostic is to compare compute time per step to communication time per step. If communication grows faster than compute as you scale, the fabric is the bottleneck. Fixes usually involve changing parallelism strategy, improving fabric capacity, or increasing computation per communication unit through larger batches or more work per step.
Congestion, Oversubscription, and the Source of Tail Latency
Fabric performance is rarely limited by peak link speed alone. It is often limited by congestion and queuing dynamics.
Oversubscription means the total demand from devices exceeds the capacity of an uplink or a shared segment. In a fat-tree style design, oversubscription can be controlled, but cost rises as oversubscription decreases. In practice, many clusters accept some oversubscription and rely on scheduling and traffic shaping to avoid worst-case collisions.
Tail latency arises when queues build up unpredictably. Common triggers:
- Many workers finish a compute phase at the same time and begin a collective together.
- A data pipeline performs a burst read that competes with training traffic.
- A serving system experiences a sudden burst and fans out requests to multiple services.
- A small number of problematic nodes retransmit or pause, causing head-of-line blocking.
Mitigations tend to be system-level rather than single-parameter tweaks:
- Separate training and serving traffic onto different networks or VLANs with strict QoS.
- Use topology-aware placement so jobs use nearby devices and minimize cross-cluster hops.
- Stagger phases or use gradient accumulation to reduce synchronization frequency.
- Monitor queue and drop signals, not only throughput.
Sizing and Choosing: When More Bandwidth Actually Helps
Fabric spending is justified when it increases delivered throughput or improves reliability at a given scale. A few questions sharpen the decision:
- Is the workload communication-heavy relative to compute, or compute-heavy relative to communication.
- Does the parallelism strategy demand frequent synchronization.
- Is the job sensitive to tail events or able to proceed with some asynchrony.
- Is the cluster mixing workloads, or is it dedicated to one job class.
Compute-heavy workloads with large local compute per step can tolerate slower fabrics. Communication-heavy workloads, especially those with frequent all-reduces, benefit dramatically from faster and more predictable networking.
Another practical consideration is failure behavior. A fabric that is faster but fragile can lose more time to retries, restarts, and debugging than it saves in step time. For large clusters, operational stability can be worth more than peak benchmarks.
Observability and Testing: Proving the Fabric Is the Limiter
Fabric issues are often misattributed because GPU utilization drops when communication stalls, making it look like a compute problem. Testing discipline helps separate causes.
Useful methods:
- Run microbenchmarks that measure point-to-point bandwidth and latency for GPU pairs and node pairs.
- Run collective tests that approximate training patterns at similar message sizes.
- Compare scaling curves across device counts and node counts to detect topology boundaries.
- Track per-step timing breakdowns to see when communication overtakes compute.
Operational metrics that matter:
- Retransmission and error counts.
- Queue and congestion indicators.
- Per-job communication time and variance.
- Tail latency for service-to-service calls when sharing the fabric.
A fabric is doing its job when performance is not only fast but stable. Stability is what turns a large cluster into a dependable production asset rather than a fragile experiment platform.
A Fabric-Centered View of the Infrastructure Shift
When AI becomes a compute layer, the network becomes part of the model’s runtime. The fabric shapes which architectures are feasible, which training regimes are cost-effective, and which products can meet latency targets reliably.
The best clusters treat networking as a first-class system with:
- topology-aware scheduling
- traffic separation for conflicting workload classes
- clear measurement of communication overhead
- failure handling that favors fast recovery over heroic debugging
Once those habits exist, adding compute becomes predictable. Without them, scaling turns into a lottery where each new node increases both capacity and the odds of a bad tail event.
More Study Resources
- Category hub
- Hardware, Compute, and Systems Overview
- Related
- Training vs Inference Hardware Requirements
- Memory Hierarchy: HBM, VRAM, RAM, Storage
- Cluster Scheduling and Job Orchestration
- Serving Hardware Sizing and Capacity Planning
- Infrastructure Shift Briefs
- Tool Stack Spotlights
- AI Topics Index
- Glossary
Books by Drew Higgins
