NVLink

NVLink

Manufacturer	Nvidia
Type	Multi-GPU and CPU technology
Predecessor	Scalable Link Interface

NVLink is a wire-based serial multi-lane near-range communications link developed by Nvidia. Unlike PCI Express, a device can consist of multiple NVLinks, and devices use mesh networking to communicate instead of a central hub. The protocol was first announced in March 2014 and uses a proprietary high-speed signaling interconnect (NVHS). ^[1]

Principle

NVLink is developed by Nvidia for data and control code transfers in processor systems between CPUs and GPUs and solely between GPUs. NVLink specifies a point-to-point connection with data rates of 20, 25 and 50 Gbit/s (v1.0/v2.0/v3.0+ resp.) per differential pair. For NVLink 1.0 and 2.0 eight differential pairs form a "sub-link" and two "sub-links", one for each direction, form a "link". Starting from NVlink 3.0 only four differential pairs form a "sub-link". For NVLink 2.0 and higher the total data rate for a sub-link is 25 GB/s and the total data rate for a link is 50 GB/s. Each V100 GPU supports up to six links. Thus, each GPU is capable of supporting up to 300 GB/s in total bi-directional bandwidth. ^{[2] [3]} NVLink products introduced to date focus on the high-performance application space. Announced May 14, 2020, NVLink 3.0 increases the data rate per differential pair from 25 Gbit/s to 50 Gbit/s while halving the number of pairs per NVLink from 8 to 4. With 12 links for an Ampere-based A100 GPU this brings the total bandwidth to 600 GB/s. ^[4] Hopper has 18 NVLink 4.0 links enabling a total of 900 GB/s bandwidth. ^[5] Thus NVLink 2.0, 3.0 and 4.0 all have a 50 GB/s per bidirectional link, and have 6, 12 and 18 links correspondingly.

Performance

The following table shows a basic metrics comparison based upon standard specifications:

Inter­connect	Transfer rate	Line code	Modulation	Effective payload rate per lane or NVLink (unidir.)	Max. total lane length [a]	Total Links (NVLink)	Total Bandwidth (PCIe x16 or NVLink)	Realized in design
PCIe 1.x	2.5 GT/s	8b/10b		0.25 GB/s	50 cm (20 in)		8 GB/s
PCIe 2.x	5 GT/s	8b/10b		0.50 GB/s	50 cm (20 in)		16 GB/s
PCIe 3.x	8 GT/s	128b/130b		0.99 GB/s	50 cm (20 in) [6]		31.51 GB/s	Pascal, Volta, Turing
PCIe 4.0	16 GT/s	128b/130b		1.97 GB/s	20–30 cm (8–12 in) [6]		63.02 GB/s	Volta on Xavier, Ampere, POWER9
PCIe 5.0	32 GT/s [7]	128b/130b		3.94 GB/s			126.03 GB/s	Hopper
PCIe 6.0	64 GT/s	236B/256B [8]	FLIT PAM4 w/ FEC	7.56 GB/s			242 GB/s	Blackwell
NVLink 1.0	20 GT/s		NRZ	20 GB/s		4	160 GB/s	Pascal, POWER8+
NVLink 2.0	25 GT/s		NRZ	25 GB/s		6	300 GB/s	Volta, POWER9
NVLink 3.0	50 GT/s		NRZ	25 GB/s		12	600 GB/s	Ampere
NVLink 4.0	50 GT/s [9]		PAM4 differential-pair	25 GB/s		18	900 GB/s	Hopper, Nvidia Grace
NVLink 5.0 [10]	100 GT/s		PAM4 differential-pair	50 GB/s		18	1800 GB/s	Blackwell, Nvidia Grace

The following table shows a comparison of relevant bus parameters for real world semiconductors that all offer NVLink as one of their options:

Semiconductor	Board/bus delivery variant	Interconnect	Transmission technology rate (per lane)	Lanes per sub-link (out + in)	Sub-link data rate (per data direction) [b]	Sub-link or unit count	Total data rate (out + in) [b]	Total lanes (out + in)	Total data rate (out + in) [b]
Nvidia GP100	P100 SXM, [11] P100 PCI-E [12]	PCIe 3.0	08 GT/s	16 + 16 [c]	128 Gbit/s = 16 GB/s	1	16 + 16 GB/s [13]	32 [d]	032 GB/s
Nvidia GV100	V100 SXM2, [14] V100 PCI-E [15]	PCIe 3.0	08 GT/s	16 + 16 [c]	128 Gbit/s = 16 GB/s	1	016 + 016 GB/s	32 [d]	032 GB/s
Nvidia TU104	GeForce RTX 2080, Quadro RTX 5000	PCIe 3.0	08 GT/s	16 + 16 [c]	128 Gbit/s = 16 GB/s	1	016 + 016 GB/s	32 [d]	032 GB/s
Nvidia TU102	GeForce RTX 2080 Ti, Quadro RTX 6000/8000	PCIe 3.0	08 GT/s	16 + 16 [c]	128 Gbit/s = 16 GB/s	1	016 + 016 GB/s	32 [d]	032 GB/s
Nvidia GA100 [16] [17] Nvidia GA102 [18]	Ampere A100 (SXM4 & PCIe) [19]	PCIe 4.0	016 GT/s	16 + 16 [c]	256 Gbit/s = 32 GB/s	1	032 + 032 GB/s	32 [d]	064 GB/s
Nvidia GP100	P100 SXM, (not available with P100 PCI-E) [20]	NVLink 1.0	20 GT/s	08 + 08 [e]	160 Gbit/s = 20 GB/s	4	080 + 080 GB/s	64	160 GB/s
Nvidia GV100	V100 SXM2 [21] (not available with V100 PCI-E)	NVLink 2.0	25 GT/s	08 + 08 [e]	200 Gbit/s = 25 GB/s	6 [22]	150 + 150 GB/s	96	300 GB/s
Nvidia TU104	GeForce RTX 2080, Quadro RTX 5000 [23]	NVLink 2.0	25 GT/s	08 + 08 [e]	200 Gbit/s = 25 GB/s	1	025 + 025 GB/s	16	050 GB/s
Nvidia TU102	GeForce RTX 2080 Ti, Quadro RTX 6000/8000 [23]	NVLink 2.0	25 GT/s	08 + 08 [e]	200 Gbit/s = 25 GB/s	2	050 + 050 GB/s	32	100 GB/s
Nvidia GA100 [16] [17]	Ampere A100 (SXM4 & PCIe) [19]	NVLink 3.0	50 GT/s	04 + 04 [e]	200 Gbit/s = 25 GB/s	12 [24]	300 + 300 GB/s	96	600 GB/s
Nvidia GA102 [18]	GeForce RTX 3090, Quadro RTX A6000	NVLink 3.0	28.125 GT/s	04 + 04 [e]	112.5 Gbit/s = 14.0625 GB/s	4	56.25 + 56.25 GB/s	16	112.5 GB/s
NVSwitch for Hopper [25]	(fully connected 64 port switch)	NVLink 4.0	106.25 GT/s	09 + 09 [e]	450 Gbit/s	18	3600 + 3600 GB/s	128	7200 GB/s
Nvidia Grace CPU [26]	Nvidia GH200 Superchip	PCIe-5 (4x, 16x) @ 512 GB/s
Nvidia Grace CPU [26]	Nvidia GH200 Superchip	NVLink-C2C @ 900 GB/s
Nvidia Hopper GPU [26]	Nvidia GH200 Superchip	NVLink-C2C @ 900 GB/s
Nvidia Hopper GPU [26]	Nvidia GH200 Superchip	NVLink 4 (18x) @ 900 GB/s

1. PCIe: incl. 5" for PCBs
2. Data rate columns are maximum theoretical values.
3. sample value; other fractions for the PCIe lane usage should be possible.
4. a single PCIe lane transfers data over a differential pair.
5. sample value; NVLink sub-link bundling should be possible.

Real world performance could be determined by applying different encapsulation taxes as well usage rate. Those come from various sources:^{[ citation needed ]}

• 128b/130b line code (see e.g. PCI Express data transmission for versions 3.0 and higher)
• Link control characters
• Transaction header
• Buffering capabilities
• DMA usage on computer side

Those physical limitations usually reduce the data rate to between 90 and 95% of the transfer rate.^{[ citation needed ]} NVLink benchmarks show an achievable transfer rate of about 35.3 Gbit/s^{[ contradictory ]} (host to device) for a 40 Gbit/s (2 sub-lanes uplink) NVLink connection towards a P100 GPU in a system that is driven by a set of IBM POWER8 CPUs. ^[27]

Usage with plug-in boards

For the various versions of plug-in boards (a yet small number of high-end gaming and professional graphics GPU boards with this feature exist) that expose extra connectors for joining them into a NVLink group, a similar number of slightly varying, relatively compact, PCB based interconnection plugs does exist. Typically only boards of the same type will mate together due to their physical and logical design. For some setups two identical plugs need to be applied for achieving the full data rate. As of now the typical plug is U-shaped with a fine grid edge connector on each of the end strokes of the shape facing away from the viewer. The width of the plug determines how far away the plug-in cards need to be seated to the main board of the hosting computer system - a distance for the placement of the card is commonly determined by the matching plug (known available plug widths are 3 to 5 slots and also depend on board type). ^{[28] [29]} The interconnect is often referred as Scalable Link Interface (SLI) from 2004 for its structural design and appearance, even if the modern NVLink based design is of a quite different technical nature with different features in its basic levels compared to the former design. Reported real world devices are: ^[30]

• Quadro GP100 (a pair of cards will make use of up to 2 bridges; ^[31] the setup realizes either 2 or 4 NVLink connections with up to 160 GB/s ^[32] - this might resemble NVLink 1.0 with 20 GT/s)
• Quadro GV100 (a pair of cards will need up to 2 bridges and realize up to 200 GB/s ^[28] - this might resemble NVLink 2.0 with 25 GT/s and 4 links)
• GeForce RTX 2080 based on TU104 (with single bridge "GeForce RTX NVLink-Bridge" ^[33] )
• GeForce RTX 2080 Ti based on TU102 (with single bridge "GeForce RTX NVLink-Bridge" ^[29] )
• Quadro RTX 5000 ^[34] based on TU104 ^[35] (with single bridge "NVLink" up to 50 GB/s ^[36] - this might resemble NVLink 2.0 with 25 GT/s and 1 link)
• Quadro RTX 6000 ^[34] based on TU102 ^[35] (with single bridge "NVLink HB" up to 100 GB/s ^[36] - this might resemble NVLink 2.0 with 25 GT/s and 2 links)
• Quadro RTX 8000 ^[34] based on TU102 ^[37] (with single bridge "NVLink HB" up to 100 GB/s ^[36] - this might resemble NVLink 2.0 with 25 GT/s and 2 links)

Service software and programming

For the Tesla, Quadro and Grid product lines, the NVML-API (Nvidia Management Library API) offers a set of functions for programmatically controlling some aspects of NVLink interconnects on Windows and Linux systems, such as component evaluation and versions along with status/error querying and performance monitoring. ^[38] Further, with the provision of the NCCL library (Nvidia Collective Communications Library) developers in the public space shall be enabled for realizing e.g. powerful implementations for artificial intelligence and similar computation hungry topics atop NVLink. ^[39] The page "3D Settings" » "Configure SLI, Surround, PhysX" in the Nvidia Control panel and the CUDA sample application "simpleP2P" use such APIs to realize their services in respect to their NVLink features. On the Linux platform, the command line application with sub-command "nvidia-smi nvlink" provides a similar set of advanced information and control. ^[30]

History

On 5 April 2016, Nvidia announced that NVLink would be implemented in the Pascal-microarchitecture-based GP100 GPU, as used in, for example, Nvidia Tesla P100 products. ^[40] With the introduction of the DGX-1 high performance computer base it was possible to have up to eight P100 modules in a single rack system connected to up to two host CPUs. The carrier board (...) allows for a dedicated board for routing the NVLink connections – each P100 requires 800 pins, 400 for PCIe + power, and another 400 for the NVLinks, adding up to nearly 1600 board traces for NVLinks alone (...). ^[41] Each CPU has direct connection to 4 units of P100 via PCIe and each P100 has one NVLink each to the 3 other P100s in the same CPU group plus one more NVLink to one P100 in the other CPU group. Each NVLink (link interface) offers a bidirectional 20 GB/sec up 20 GB/sec down, with 4 links per GP100 GPU, for an aggregate bandwidth of 80 GB/sec up and another 80 GB/sec down. ^[42] NVLink supports routing so that in the DGX-1 design for every P100 a total of 4 of the other 7 P100s are directly reachable and the remaining 3 are reachable with only one hop. According to depictions in Nvidia's blog-based publications, from 2014 NVLink allows bundling of individual links for increased point to point performance so that for example a design with two P100s and all links established between the two units would allow the full NVLink bandwidth of 80 GB/s between them. ^[43]

At GTC2017, Nvidia presented its Volta generation of GPUs and indicated the integration of a revised version 2.0 of NVLink that would allow total I/O data rates of 300 GB/s for a single chip for this design, and further announced the option for pre-orders with a delivery promise for Q3/2017 of the DGX-1 and DGX-Station high performance computers that will be equipped with GPU modules of type V100 and have NVLink 2.0 realized in either a networked (two groups of four V100 modules with inter-group connectivity) or a fully interconnected fashion of one group of four V100 modules.

In 2017–2018, IBM and Nvidia delivered the Summit and Sierra supercomputers for the US Department of Energy ^[44] which combine IBM's POWER9 family of CPUs and Nvidia's Volta architecture, using NVLink 2.0 for the CPU-GPU and GPU-GPU interconnects and InfiniBand EDR for the system interconnects. ^[45]

In 2020, Nvidia announced that they will no longer be adding new SLI driver profiles on RTX 2000 series and older from January 1, 2021. ^[46]

See also

• UALink
• Intel QuickPath Interconnect
• HyperTransport
• Message Passing Interface
• INK (operating system)
• Compute Node Linux
• Intel Xe Link

References

1. Nvidia NVLINK 2.0 arrives in IBM servers next year by Jon Worrel on fudzilla.com on August 24, 2016
2. "NVIDIA DGX-1 With Tesla V100 System Architecture" (PDF).
3. "What Is NVLink?". Nvidia. 2014-11-14.
4. Ryan Smith (May 14, 2020). "NVIDIA Ampere Unleashed: NVIDIA Announces New GPU Architecture, A100 GPU, and Accelerator". AnandTech.
5. Jacobs, Blair (2022-03-23). "Nvidia reveals next-gen Hopper GPU architecture". Club386. Retrieved 2022-05-04.
6. "PCIe - PCI Express (1.1 / 2.0 / 3.0 / 4.0 / 5.0)". www.elektronik-kompendium.de.
7. Alcorn, Paul (17 January 2019). "PCIe 5.0 Is Ready For Prime Time". Tom's Hardware.
8. "The PCIe 6.0 Specification Webinar Q&A: A Deeper Dive into FLIT Mode, PAM4, and Forward Error Correction (FEC) PCI-SIG". pcisig.com. PCI-SIG. Retrieved 28 November 2024. We considered various FLIT sizes and settled on 256 Bytes with 236 bytes of TLP payload and a TLP efficiency of 92%.
9. "NVLink-Network Switch - NVIDIA's Switch Chip for High Communication-Bandwidth SuperPODs" (PDF). HotChips 34. 23 August 2022.
10. "NVIDIA Blackwell Architecture Technical Overview". NVIDIA. p. 8. Retrieved 28 November 2024. Fifth-generation NVLink doubles the performance of fourth- generation NVLink in NVIDIA Hopper. While the new NVLink in Blackwell GPUs also uses two high-speed differential pairs in each direction to form a single link as in the Hopper GPU, NVIDIA Blackwell doubles the effective bandwidth per link to 50 GB/sec in each direction.
11. online, heise. "NVIDIA Tesla P100 [SXM2], 16GB HBM2 (NVTP100-SXM) | heise online Preisvergleich / Deutschland". geizhals.de.
12. online, heise (14 August 2023). "PNY Tesla P100 [PCIe], 16GB HBM2 (TCSP100M-16GB-PB/NVTP100-16) ab € 4990,00 (2020) | heise online Preisvergleich / Deutschland". geizhals.de.
13. NVLink Takes GPU Acceleration To The Next Level by Timothy Prickett Morgan at nextplatform.com on May 4, 2016
14. "NVIDIA Tesla V100 SXM2 16 GB Specs". TechPowerUp. 14 August 2023.
15. online, heise (14 August 2023). "PNY Quadro GV100, 32GB HBM2, 4x DP (VCQGV100-PB) ab € 10199,00 (2020) | heise online Preisvergleich / Deutschland". geizhals.de.
16. Morgan, Timothy Prickett (May 14, 2020). "Nvidia Unifies AI Compute With "Ampere" GPU". The Next Platform.
17. "Data sheet" (PDF). www.nvidia.com. Retrieved 2020-09-15.
18. "NVIDIA ampere GA102 GPU Architecture Whitepaper" (PDF). nvidia.com. Retrieved 2 May 2023.
19. "Tensor Core GPU" (PDF). nvidia.com. Retrieved 2 May 2023.
20. Chris Williams (June 20, 2016). "All aboard the PCIe bus for Nvidia's Tesla P100 supercomputer grunt". theregister.co.uk.
21. online, heise (22 June 2017). "Nvidia Tesla V100: PCIe-Steckkarte mit Volta-Grafikchip und 16 GByte Speicher angekündigt". heise online.
22. GV100 Blockdiagramm in "GTC17: NVIDIA präsentiert die nächste GPU-Architektur Volta - Tesla V100 mit 5.120 Shadereinheiten und 16 GB HBM2" by Andreas Schilling on hardwareluxx.de on May 10, 2017
23. Angelini, Chris (14 September 2018). "Nvidia's Turing Architecture Explored: Inside the GeForce RTX 2080". Tom's Hardware . p. 7. Retrieved 28 February 2019. TU102 and TU104 are Nvidia's first desktop GPUs rocking the NVLink interconnect rather than a Multiple Input/Output (MIO) interface for SLI support. The former makes two x8 links available, while the latter is limited to one. Each link facilitates up to 50 GB/s of bidirectional bandwidth. So, GeForce RTX 2080 Ti is capable of up to 100 GB/s between cards and RTX 2080 can do half of that.
24. Schilling, Andreas (22 June 2020). "A100 PCIe: NVIDIA GA100-GPU kommt auch als PCI-Express-Variante". Hardwareluxx. Retrieved 2 May 2023.
25. "NVLINK AND NVSWITCH". www.nvidia.com. Retrieved 2021-02-07.
26. "A Big Memory Nvidia GH200 Next to Your Desk: Closer Than You Think". 23 February 2024.
27. Eliot Eshelman (January 26, 2017). "Comparing NVLink vs PCI-E with NVIDIA Tesla P100 GPUs on OpenPOWER Servers". microway.com.
28. "NVIDIA Quadro NVLink Grafikprozessor-Zusammenschaltung in Hochgeschwindigkeit". NVIDIA.
29. "Grafik neu erfunden: NVIDIA GeForce RTX 2080 Ti-Grafikkarte". NVIDIA.
30. "NVLink on NVIDIA GeForce RTX 2080 & 2080 Ti in Windows 10". Puget Systems. 5 October 2018.
31. ^[ dead link ‍]
32. Schilling, Andreas (5 February 2017). "NVIDIA präsentiert Quadro GP100 mit GP100-GPU und 16 GB HBM2". Hardwareluxx.
33. "NVIDIA GeForce RTX 2080 Founders Edition Graphics Card". NVIDIA.
34. "NVIDIA Quadro Graphics Cards for Professional Design Workstations". NVIDIA.
35. "NVIDIA Quadro RTX 6000 und RTX 5000 Ready für Pre-Order". October 1, 2018.
36. "NVLink | pny.com". www.pny.com.
37. "NVIDIA Quadro RTX 8000 Specs". TechPowerUp. 14 August 2023.
38. "NvLink Methods". docs.nvidia.com.
39. "NVIDIA Collective Communications Library (NCCL)". NVIDIA Developer. May 10, 2017.
40. "Inside Pascal: NVIDIA's Newest Computing Platform". 2016-04-05.
41. Anandtech.com
42. NVIDIA Unveils the DGX-1 HPC Server: 8 Teslas, 3U, Q2 2016 by anandtech.com on April, 2016
43. How NVLink Will Enable Faster, Easier Multi-GPU Computing by Mark Harris on November 14, 2014
44. "Whitepaper: Summit and Sierra Supercomputers" (PDF). 2014-11-01.
45. "Nvidia Volta, IBM POWER9 Land Contracts For New US Government Supercomputers". AnandTech. 2014-11-17.
46. "RIP: Nvidia slams the final nail in SLI's coffin, no new profiles after 2020". PC World. 2020-09-18.