NCP-AIN NVIDIA-Certified Professional AI Networking Questions and Answers

The correct answer isPXN (Peer eXchange Network).

From theNVIDIA NCCL Documentation:

"PXN enables communication between GPUs connected via NVLink and NICs by treating the GPUs as a distributed switch. This architecture improves bandwidth utilization by enabling any GPU to communicate with the NIC via the shortest path available, even if it's not directly connected to the NIC."

This enhances GPU-to-NIC and NIC-to-GPU transfers, leveraging the NVLink topology. It significantly boosts performance in multi-GPU setups where not every GPU is directly connected to the NIC.

Other options:

Adaptive Routingis a fabric-level feature for dynamic path rerouting.

Graph Search Optimizationis used internally for topology modeling in NCCL.

SHARP v2is a switch-based collective acceleration method, unrelated to PXN.

In Cumulus Linux, microbursts are short-lived, high-volume traffic bursts that often go undetected by coarse-grained monitoring like SNMP.

The two tools specifically used for this purpose are:

What Just Happened (WJH)

"WJH provides real-time packet drop visibility and classifies drops by reason (e.g., congestion, ACLs, etc.), enabling microburst detection."

ASIC monitoring at millisecond granularity

"Deep telemetry is enabled via the switch ASIC, which provides sub-second counters that capture microburst patterns otherwise missed by SNMP."

Incorrect Options:

AandCprovide low-frequency sampling, insufficient for microbursts which last milliseconds.

NVIDIA Air is a cloud-based network simulation tool designed to create digital twins of data center infrastructure, including Spectrum-X networks. It allows users to model switches, SuperNICs, and storage components, enabling the simulation, validation, and automation of network configurations before physical deployment. This facilitates Day 0, 1, and 2 operations, ensuring that network designs are tested and optimized for AI workloads.​

Reference Extracts from NVIDIA Documentation:

"NVIDIA Air enables cloud-scale efficiency by creating identical replicas of real-world data center infrastructure deployments."​

"NVIDIA Air allows users to model data center deployments with full software functionality, creating a digital twin. Transform and accelerate time to AI by simulating, validating, and automating changes and updates."​

"NVIDIA Air supports simulation of NVIDIA Spectrum Ethernet (Cumulus Linux and SONiC) switches and NVIDIA BlueField DPUs and SuperNICs as well as the NetQ network operations toolset."​

In InfiniBand networks,P_Keys (Partition Keys)control communication boundaries. Each port can belong to one or more partitions with eitherfullorlimitedmembership.

From NVIDIA InfiniBand Documentation (Partitioning and P_Keys):

"A limited (or partial) membership permits a port to communicate only with other ports in the same partition that have full membership. It cannot communicate with other limited members, even if they are in the same P_Key partition."

This makes limited/partial membership ideal formulti-tenant security, where tenant ports can reach infrastructure ports (full members) butnotother tenant ports (limited members).

Incorrect Options:

A & Bare not valid InfiniBand P_Key types.

C (Full membership)allows unrestricted communication within the same partition.

The Network Traffic Map in NVIDIA's Unified Fabric Manager (UFM) provides a visual representation of the network topology and traffic flows, which is particularly useful for identifying congestion points, verifying network configurations, and optimizing workload distribution.

However, when troubleshooting a single node's hardware failure, the Network Traffic Map is less effective, as it focuses on network-level issues rather than individual hardware components.​

Accessing theBlueField DPU Operating System (OS)is possible throughrshim, either over PCIe or USB, and viaSSH through the OOB interfacewhen in DPU mode.

From theNVIDIA BlueField Software Documentation:

"You can access the BlueField OS through the rshim interface. The rshim module enables host-to-DPU communication either via PCIe (default) or USB."

B. rshim over PCIe: Default when BlueField is installed in a host.

D. rshim over USB: Useful for provisioning or systems without PCIe drivers.

Incorrect Options:

A (NIC mode): BlueField acts as a transparent NIC; OS access is not available to the host.

C (Redfish): Redfish is for out-of-band management, not direct OS-level access.

TheNVIDIA UFM Cyber-AI Platformis specifically designed to enhance security and operational efficiency in InfiniBand data centers. It leverages AI-powered analytics to detect security threats, operational anomalies, and predict potential network failures. By analyzing real-time telemetry data, it identifies abnormal behaviors and performance degradation, enabling proactive maintenance and threat mitigation.​

This platform integrates with existing UFM Enterprise and Telemetry services to provide a comprehensive view of the network's health and security posture. It utilizes machine learning algorithms to establish baselines for normal operations and detect deviations that may indicate security breaches or hardware issues.​

For optimal performance in multi-node, multi-GPU AI workloads:

NVLinkprovides high-speed, low-latency communication between GPUs within the same node.

InfiniBandoffers efficient, scalable communication between nodes in a data center.Combining these technologies ensures both intra-node and inter-node communication needs are effectively met.​

NVIDIA's What Just Happened (WJH) is a feature that provides real-time visibility into network problems by analyzing all packets passing through the switch and alerting on performance issues caused by packet drops, congestion, high latency, or misconfigurations.

WJH retains the last packets that were dropped from the switch with complete packet headers and the actual drop reason. This enhances the ability to debug network problems, identify affected flows, and decrease time-to-repair.

The NVIDIA Spectrum-X Reference Architecture (RA) 1.0.1 is designed for Ethernet AI cloud deployments and includes the SN5600 Spectrum-4 switches and BlueField-3 SuperNICs. This architecture supports adaptive routing and DOCA programmable congestion control (PCC) for lossless RoCE traffic, optimizing performance for AI workloads.

The SN5600 switch offers 64 ports of 800GbE in a dense 2U form factor, providing high throughput and low latency essential for AI applications.

AnInfiniBand (IB) Routerconnects two or moreInfiniBand subnets, making it possible for nodes in different subnets to communicate throughroute-managed communication.

From the officialNVIDIA InfiniBand Routers Documentation:

"An InfiniBand router provides connectivity between two or more InfiniBand subnets, enabling communication between hosts that are not on the same subnet while preserving isolation and scalability."

Ensures fabric scalability by allowing subnet segmentation.

Uses LID routing across subnet managers (SMs).

Essential in large clusters with thousands of nodes.

Incorrect Options:

AandBare incorrect: InfiniBand does not connect directly to Ethernet or SANs without a gateway.

Cis unrelated: NVLink is a GPU interconnect, not tied to InfiniBand routers.

To enable SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) aggregation using theNCCL-SHARP plugin, the following two environment variables are required:

NCCL_COLLNET_ENABLE=1

Enables NCCL’s support for CollNet (Collective Network) operations, including SHARP.

NCCL_SHARP_AUTOINIT=1

Automatically initializes the SHARP plugin when available, activating SHARP-based collectives.

From theNVIDIA NCCL User Guide – SHARP Plugin Section:

"NCCL_COLLNET_ENABLE must be set to enable collective network acceleration features."

"NCCL_SHARP_AUTOINIT enables automatic SHARP plugin integration at NCCL runtime."

Incorrect Options:

B. NCCL_ALGO=CollNet– This variable controls the algorithm used for collectives but does not enable SHARP.

C. NCCLSPECTRUM_ENABLE– This is not a documented NCCL variable.

Role-Based Access Control (RBAC)is implemented within NVIDIA's Unified Fabric Manager (UFM) to manage user permissions effectively. RBAC allows administrators toassign roles to users, each with specific permissions, ensuring that only authorized individuals can make changes to the fabric configuration. This structured approach to access control enhances security by limiting the potential for unauthorized modifications and streamlines the management of user privileges across the network.​

TheMINHOProuting protocol, while efficient in finding minimal paths, does not inherently prevent credit loops. This can lead to deadlocks in the network. In contrast, routing protocols like UPDOWN and FAT TREE are designed to avoid such loops, ensuring more reliable network operation.​

In InfiniBand networks,Partitioning using Partition Keys (PKeys)is the standard method for implementing multi-tenancy and traffic isolation. PKeys allow administrators to define logical partitions within the fabric, ensuring that traffic is confined to designated groups of nodes. This mechanism is essential for creating secure and isolated environments in multi-tenant architectures.​

The Unified Fabric Manager (UFM) leverages PKeys to manage these partitions effectively, enabling administrators to assign and control access rights across different tenants. This approach ensures that each tenant's traffic remains isolated, maintaining both security and performance integrity within the shared fabric.​

NVIDIA AIR (AI-Ready Infrastructure) is a cloud-based simulation platform designed to model and validate data center network deployments, including Spectrum-X Ethernet networks, using realistic topologies and configurations. When creating a custom topology in NVIDIA AIR, users can define network links between devices (e.g., spine and leaf switches) using a DOT file format, which is based on the Graphviz graph visualization software. The question asks for the correct syntax to define a link between port 1 on a spine switch (spine-01) and port 41 on a leaf switch (gpu-leaf-01) in a NVIDIA AIR simulation.

According to NVIDIA’s official NVIDIA AIR documentation, the DOT file format is used to specify network topologies, including nodes (devices) and links (connections between ports). The syntax for defining a link in a DOT file uses a double dash (--) to indicate a connection between two ports, with each port specified in the format "":"". For Spectrum-X networks, which typically use Cumulus Linux or SONiC on NVIDIA Spectrum switches, ports are commonly labeled as swpX (switch port X) rather than ethX (Ethernet interface), especially for switch-to-switch connections in a leaf-spine topology. The correct syntax for the link between port 1 on spine-01 and port 41 on gpu-leaf-01 is:

"spine-01":"swp01" -- "gpu-leaf-01":"swp41"

This syntax uses swp01 and swp41 to denote switch ports, consistent with Cumulus Linux conventions, and the double dash (--) to indicate the link, as required by the DOT file format.

Exact Extract from NVIDIA Documentation:

“You can create custom topologies in Air using a DOT file, which is the file type used with the open-source graph visualization software, Graphviz. DOT files define nodes, attributes, and connections for generating a topology for a network. The following is an example of a link definition in a DOT file:

"leaf01":"swp31" -- "spine01":"swp1"

This specifies a connection between port swp31 on leaf01 and port swp1 on spine01. Port names typically follow the switch port naming convention (e.g., swpX) for Cumulus Linux-based switches.”

—NVIDIA Air Custom Topology Guide

This extract confirms that option A is the correct answer, as it uses the proper DOT file syntax with swp01 and swp41 for port names and the double dash (--) for the link, aligning with NVIDIA AIR’s topology definition process for Spectrum-X simulations.

Analysis of Other Options:

B. "spine-01":"swp1" to "gpu-leaf-01":"swp41": This option uses the correct port naming convention (swp1 and swp41) but incorrectly uses the word to as the connector instead of the double dash (--). The DOT file format requires -- to define links, making this syntax invalid for NVIDIA AIR.

C. "spine-01":"eth1" to "gpu-leaf-01":"eth41": This option uses ethX port names, which are typically used for host interfaces (e.g., servers) rather than switch ports in Cumulus Linux or SONiC environments. Switch ports in Spectrum-X topologies are labeled swpX. Additionally, the use of to instead of -- is incorrect for DOT file syntax, making this option invalid.

D. "spine-01":"eth1" - "gpu-leaf-01":"eth41": This option uses a single dash (-) instead of the required double dash (--) and incorrectly uses ethX port names instead of swpX. The ethX naming is not standard for switch ports in Spectrum-X, and the single dash is not valid DOT file syntax, making this option incorrect.

Why "spine-01":"swp01" -- "gpu-leaf-01":"swp41" is the Correct Answer:

Option A correctly adheres to the DOT file syntax used in NVIDIA AIR for defining network links:

Node and Port Naming: The nodes spine-01 and gpu-leaf-01 are specified with their respective ports swp01 and swp41, following the swpX convention for switch ports in Cumulus Linux-based Spectrum-X switches.

Link Syntax: The double dash (--) is the standard connector in DOT files to indicate a link between two ports, as required by Graphviz and NVIDIA AIR.

Spectrum-X Context: In a Spectrum-X leaf-spine topology, connections between spine and leaf switches (e.g., Spectrum-4 switches) use switch ports labeled swpX, making swp01 and swp41 appropriate for this simulation.

This syntax ensures that the NVIDIA AIR simulation accurately models the physical connection between spine-01 port 1 and gpu-leaf-01 port 41, enabling validation of the Spectrum-X network topology. The DOT file can be uploaded to NVIDIA AIR to generate the topology, as described in the documentation.

InDPU Mode, theARM cores on BlueFieldown theNIC data path, while still allowing thehost system to access the DPU OS (via OOB or virtio).

From NVIDIA BlueField Documentation:

"In DPU Mode, the data path is offloaded to the BlueField Arm cores, enabling advanced security and networking functions, while still allowing host access to the BlueField OS."

This is different from:

NIC Mode: Data path controlled by host, ARM cores inactive.

Separated Host Mode: Complete isolation; host cannot access DPU OS.

Restricted Mode: Limited host access to DPU OS, but without full offload capabilities.

MLNX_OFED (Mellanox OpenFabrics Enterprise Distribution) is an NVIDIA-tested and packaged version of the OpenFabrics Enterprise Distribution (OFED) for Linux. It provides the necessary drivers and tools to support InfiniBand and Ethernet interconnects using the same RDMA (Remote Direct Memory Access) and kernel bypass APIs. MLNX_OFED enables high-performance networking capabilities essential for AI clusters, including support for up to 400Gb/s InfiniBand and RoCE (RDMA over Converged Ethernet).​

Reference Extracts from NVIDIA Documentation:

"MLNX_OFED is an NVIDIA tested and packaged version of OFED that supports two interconnect types using the same RDMA (remote DMA) and kernel bypass APIs called OFED verbs – InfiniBand and Ethernet."​

"Up to 400Gb/s InfiniBand and RoCE (based on the RDMA over Converged Ethernet standard) over 10/25/40/50/100/200/400GbE are supported."​