You have just deployed a GPU cluster for AI training. The hardware cost was significant. The timeline is tight. You run the first distributed training job and GPU utilisation sits at 41%.
You check the GPUs: fine.
You check the model: fine.
You check the network: nobody checked the network.
RoCEv2 congestion is the most common, most expensive and most preventable cause of AI data center underperformance. It is invisible to standard monitoring tools, it manifests in compute metrics rather than network metrics, and it almost always reaches production without ever being tested. This guide tells you exactly how to find it, measure it, and eliminate it before go-live.
What is RoCEv2 and why it matters for AI workloads
RoCEv2 (RDMA over Converged Ethernet version 2) is the network protocol that enables GPU nodes in an AI data center to exchange data directly, bypassing the CPU entirely. This direct memory access is what makes large-scale distributed AI training possible: without it, the CPU would become the bottleneck for every gradient synchronisation event during training.
RoCEv2. RDMA over Converged Ethernet v2. Enables direct GPU-to-GPU memory transfers over standard Ethernet without CPU involvement. The standard protocol for AI data center fabric since 2014.
RDMA. Remote Direct Memory Access. Transfers data between nodes without CPU processing, reducing latency from milliseconds to microseconds.
Distributed Training. Training an AI model across multiple GPU nodes simultaneously. Requires constant synchronisation, which is why the network is performance-critical.
The reason RoCEv2 matters so much for AI workloads is the traffic pattern. Traditional network traffic is relatively uniform, many small flows going in many directions. AI training traffic is synchronised and bursty: all GPU nodes transmit simultaneously during collective communication operations (AllReduce, AlltoAll, RingAllReduce), creating incast conditions that standard Ethernet was not designed to handle.
AI training workloads generate a unique class of network traffic that differs fundamentally from enterprise or web traffic patterns. Understanding these patterns is essential for understanding why standard network testing fails to predict AI data center performance.
What is RingAllReduce and why does it stress the network?
RingAllReduce is the distributed algorithm used during AI training to average gradients across all GPU nodes. Devices are arranged in a logical ring. Each GPU sends its gradient data to the next node while receiving from the previous, in two phases: ReduceScatter (aggregation) and AllGather (distribution).
The network impact is a large volume of synchronised east-west traffic between all nodes simultaneously. Every node transmits at the same time. The network fabric must handle this without congestion or packet loss, because any single delayed flow holds back the entire job.
What is AlltoAll communication and why is it the most demanding pattern?
AlltoAll is the most demanding collective communication pattern: every GPU exchanges data with every other GPU simultaneously. This creates full-mesh traffic across the switch fabric and is the primary stress test for leaf-spine topologies. Traditional testing tools that generate uniform traffic patterns completely fail to simulate AlltoAll conditions.
What is the Collective Communication Library (CCL)?
The Collective Communication Library (CCL) is the software layer that coordinates synchronisation across GPU nodes. NVIDIA's NCCL is the most widely used implementation. CCL operations -AllReduce, AllGather, ReduceScatter, Broadcast- each generate different traffic patterns with different network requirements. A complete validation must test all relevant CCL patterns for the specific AI workloads being deployed.
Above the pic: AlltoAll Example message path from GPU0 in DGX-A to GPU3 in DGX-B
RoCEv2 congestion: why it happens and how to detect it.
RoCEv2 has no native congestion control mechanism. When the network becomes congested, two control protocols activate: PFC (Priority Flow Control) and DCQCN (Data Center Quantized Congestion Notification). If either is misconfigured, performance collapses and the failure manifests in compute metrics, not network metrics, which is why it is so frequently misdiagnosed.
PCF. Priority Flow Control. Layer 2 mechanism that pauses traffic per priority class when switch buffers near overflow. Prevents packet loss but can cause deadlock if misconfigured.
DCQC. Data Center Quantized Congestion Notification. Congestion control algorithm for RoCEv2. Reduces sender transmission rate when congestion is detected via ECN marking.
ECN. Explicit Congestion Notification. IP-level mechanism that marks packets when they pass through a congested switch, signalling senders to reduce their rate.
CNP. Congestion Notification Packet. Generated by the receiver NIC when it detects ECN-marked packets. Sent to the sender to trigger rate reduction.
PFC Operation
▸ PFC pause frames appearing on GPU-facing switch ports, this signals that congestion has already reached a critical level
▸ GPU utilisation drops below 60% during distributed training without any model-level bottleneck
▸ AllReduce or AlltoAll collective operation latency spikes inconsistently across training iterations
▸ Packet retransmission rate above 0.01% on inter-node traffic, even this small amount causes visible performance degradation
▸ High Job Completion Time (JCT) variance, the same training job takes different amounts of time across runs due to network instability
The critical detail about PFC pause frames: they are the last line of defence before packet loss occurs. Their appearance in production means the congestion is already severe. By the time you see PFC pauses in your monitoring, you have already lost significant GPU performance for an unknown period.
| Metric | Healthy | Warning | Critical |
| GPU utilisation (distributed training) | 85-95% | 60-84% | < 60% |
| Packet retransmission rate | < 0.001% | 0.001-0.01% | > 0.01% |
| PFC pause frames per minute | 0 | 1-50 | > 50 |
| AllReduce latency p95 | < 50ms | 50-200ms | > 200ms |
| Job Completion Time variance | < 5% | 5-15% | > 15% |
| ECN marking rate | < 1% | 1-5% | > 5% |
Problem indicators, cause and testing insight.
Based on VIAVI Solutions' AI/ML Data Center Network Validation research and Netmetrix field experience across EMEA deployments, these are the most common failure patterns and their root causes.
| Issue observed | Most likely cause | What validation reveals |
Packet loss during peak load |
Misconfigured PFC thresholds or switch buffer overflow during synchronised training bursts |
Identifies the affected queue pair (QP) or switch port and the exact load level at which loss begins |
Long tail latency in training |
Flow path imbalance or resource contention at specific switch stages |
Reveals which links are delayed and correlates with topology and DCQCN configuration |
High JCT variance |
Inconsistent ECN/CNP responsiveness or queue buildup across iterations |
Compares algorithm performance under load, tracking JCT changes and degraded iterations |
Congestion with no rate drop |
ECMP algorithm or network topology needs optimisation |
Verifies congestion presence and validates that DCQCN rate reduction is triggering correctly |
GPU utilisation plateau |
Switch fabric bottleneck not visible in compute monitoring |
Identifies the network component limiting GPU throughput, invisible to standard compute monitoring |
How Netmetrix validates RoCEv2: the 4-phase framework
As a certified VIAVI partner operating across Italy, Spain, France, Portugal and the UK, Netmetrix applies a structured validation framework before any AI data center go-live.
The framework uses VIAVI TestCenter appliances (A1-400G and B3-800G series) to emulate realistic AI traffic patterns, including RoCEv2, RingAllReduce, AlltoAll and CCL patterns, under conditions that match actual production workloads.
1. Baseline characterisation
Measure throughput and latency for each GPU-to-GPU path in the fabric. Characterise switch buffer behaviour under increasing load. Establish PFC pause frame rate at nominal load as the reference baseline. Document ECN marking thresholds per switch.
2. Incast stress testing
Simulate AllReduce traffic patterns at 25%, 50%, 75% and 100% of design capacity. Inject incast scenarios, N nodes to 1 destination simultaneously, varied burst sizes. Measure packet loss rate, retransmission rate and throughput degradation at each load level. Identify the congestion onset point where performance degrades non-linearly.
3. DCQCN Tuning Validation
Test current DCQCN parameter settings against incast scenarios. Iterate parameter tuning and re-test until PFC pause rate drops below threshold. Validate that tuning does not introduce new failure modes such as under-aggressive congestion response. Document the final validated parameter set for production deployment.
4. Production readiness sign-off
Run full distributed AI workload simulation at design capacity for a minimum of 4 hours. Verify GPU utilisation above 85% under sustained load. Verify zero packet loss and PFC pause rate below 0.001%. Produce the test report with all metrics documented as the production readiness sign-off document.
Frequently Asked Questions
| Q: What is the difference between RoCEv2 and InfiniBand for AI data centers? |
| A: InfiniBand provides slightly lower latency and historically stronger congestion control, but requires dedicated hardware and infrastructure. RoCEv2 runs over standard Ethernet, supports Layer 3 routing, and can scale across multi-rack and multi-site deployments without dedicated switches. For most enterprise and hyperscale AI deployments, RoCEv2 over 100G/400G Ethernet is the standard choice. The validation methodology for both protocols shares the same core requirements: incast testing, congestion control verification and collective communication pattern emulation. |
| Q: Why does GPU utilisation drop during distributed AI training? |
| A: The most common cause is network congestion in the RoCEv2 fabric — specifically incast events during AllReduce operations, where all GPU nodes transmit simultaneously to synchronise gradients. When the network becomes congested, GPU nodes stall waiting for synchronisation to complete, appearing as low GPU utilisation in compute monitoring. Other causes include DCQCN misconfiguration, PFC deadlock, and ECMP routing imbalance. A structured pre-production validation that tests under realistic CCL traffic patterns identifies which cause applies before the system goes live. |
| Q: What tools are used for RoCEv2 validation in AI data centers? |
A: Professional RoCEv2 validation requires purpose-built traffic generation platforms that can emulate AI-specific patterns — including RingAllReduce, AlltoAll and CCL operations at wire speed. Netmetrix uses VIAVI TestCenter appliances (A1-400G-16-port and B3-800G series) for this purpose. Basic tools like iPerf cannot emulate the synchronised burst patterns of AI training workloads and are not suitable for pre-production validation of AI data center fabrics. For AI model validation beyond the network layer, read our guide on LLM testing and validation. |
| Q: What is DCQCN and how does it prevent RoCEv2 congestion? |
| A: DCQCN (Data Center Quantized Congestion Notification) is the congestion control algorithm designed for RoCEv2. When a switch detects congestion, it marks packets with ECN (Explicit Congestion Notification) bits. The receiver NIC generates CNP (Congestion Notification Packets) and sends them to the sender. The sender reduces its transmission rate multiplicatively when it receives a CNP. When no congestion is detected for a configurable period, the rate gradually increases. Incorrect DCQCN parameters — particularly the Alpha value and rate increase/decrease thresholds — are the most common cause of RoCEv2 performance issues in production. |
| Q: How long does RoCEv2 validation take before a data center go-live? |
| A: A complete Netmetrix RoCEv2 validation engagement — baseline characterisation, incast stress testing, DCQCN tuning and production readiness sign-off — typically takes 2 to 3 weeks. For urgent go-live timelines, an expedited 5-day assessment covering the highest-risk scenarios is available. The sign-off document produced includes all test results, validated configuration parameters and the production readiness determination. |
| Q: Can RoCEv2 validation be done without specialised equipment? |
| A: Basic characterisation can be performed with open-source tools such as ib_send_bw and NCCL tests. Full validation — including incast simulation at scale, DCQCN parameter tuning and production readiness sign-off — requires professional traffic generation platforms capable of emulating AI-specific CCL patterns at wire speed. Netmetrix uses VIAVI TestCenter for this, which provides the traffic emulation, measurement precision and reporting quality required for a documented production readiness determination. |
This article draws on concepts and research from the VIAVI Solutions white paper 'AI/ML Data Center Network Validation' (2025) and Netmetrix field experience as a certified VIAVI partner across EMEA. For the VIAVI TestCenter AI testing solution, visit viavisolutions.com
