Non-Volatile Memory Host Controller Interface Specification | |
Abbreviation | NVMe |
---|---|
Status | Published |
Year started | 2011 |
Latest version | 2.0d January 11, 2024 [1] |
Organization | NVM Express Work Group (incorporated as NVM Express in 2014) |
Website | nvmexpress |
NVM Express (NVMe) or Non-Volatile Memory Host Controller Interface Specification (NVMHCIS) is an open, logical-device interface specification for accessing a computer's non-volatile storage media usually attached via the PCI Express bus. The initial NVM stands for non-volatile memory , which is often NAND flash memory that comes in several physical form factors, including solid-state drives (SSDs), PCIe add-in cards, and M.2 cards, the successor to mSATA cards. NVM Express, as a logical-device interface, has been designed to capitalize on the low latency and internal parallelism of solid-state storage devices. [2]
Architecturally, the logic for NVMe is physically stored within and executed by the NVMe controller chip that is physically co-located with the storage media, usually an SSD. Version changes for NVMe, e.g., 1.3 to 1.4, are incorporated within the storage media, and do not affect PCIe-compatible components such as motherboards and CPUs. [3]
By its design, NVM Express allows host hardware and software to fully exploit the levels of parallelism possible in modern SSDs. As a result, NVM Express reduces I/O overhead and brings various performance improvements relative to previous logical-device interfaces, including multiple long command queues, and reduced latency. The previous interface protocols like AHCI were developed for use with far slower hard disk drives (HDD) where a very lengthy delay (relative to CPU operations) exists between a request and data transfer, where data speeds are much slower than RAM speeds, and where disk rotation and seek time give rise to further optimization requirements.
NVM Express devices are chiefly available in the form of standard-sized PCI Express expansion cards [4] and as 2.5-inch form-factor devices that provide a four-lane PCI Express interface through the U.2 connector (formerly known as SFF-8639). [5] [6] Storage devices using SATA Express and the M.2 specification which support NVM Express as the logical-device interface are a popular use-case for NVMe and have become the dominant form of solid-state storage for servers, desktops, and laptops alike. [7] [8]
Specifications for NVMe released to date include: [9]
Historically, most SSDs used buses such as SATA, SAS, or Fibre Channel for interfacing with the rest of a computer system. Since SSDs became available in mass markets, SATA has become the most typical way for connecting SSDs in personal computers; however, SATA was designed primarily for interfacing with mechanical hard disk drives (HDDs), and it became increasingly inadequate for SSDs, which improved in speed over time. [11] For example, within about five years of mass market mainstream adoption (2005–2010) many SSDs were already held back by the comparatively slow data rates available for hard drives—unlike hard disk drives, some SSDs are limited by the maximum throughput of SATA.
High-end SSDs had been made using the PCI Express bus before NVMe, but using non-standard specification interfaces. By standardizing the interface of SSDs, operating systems only need one common device driver to work with all SSDs adhering to the specification. It also means that each SSD manufacturer does not have to design specific interface drivers. This is similar to how USB mass storage devices are built to follow the USB mass-storage device class specification and work with all computers, with no per-device drivers needed. [12]
NVM Express devices are also used as the building block of the burst buffer storage in many leading supercomputers, such as Fugaku Supercomputer, Summit Supercomputer and Sierra Supercomputer, etc. [13] [14]
The first details of a new standard for accessing non-volatile memory emerged at the Intel Developer Forum 2007, when NVMHCI was shown as the host-side protocol of a proposed architectural design that had Open NAND Flash Interface Working Group (ONFI) on the memory (flash) chips side. [15] A NVMHCI working group led by Intel was formed that year. The NVMHCI 1.0 specification was completed in April 2008 and released on Intel's web site. [16] [17] [18]
Technical work on NVMe began in the second half of 2009. [19] The NVMe specifications were developed by the NVM Express Workgroup, which consists of more than 90 companies; Amber Huffman of Intel was the working group's chair. Version 1.0 of the specification was released on 1 March 2011, [20] while version 1.1 of the specification was released on 11 October 2012. [21] Major features added in version 1.1 are multi-path I/O (with namespace sharing) and arbitrary-length scatter-gather I/O. It is expected that future revisions will significantly enhance namespace management. [19] Because of its feature focus, NVMe 1.1 was initially called "Enterprise NVMHCI". [22] An update for the base NVMe specification, called version 1.0e, was released in January 2013. [23] In June 2011, a Promoter Group led by seven companies was formed.
The first commercially available NVMe chipsets were released by Integrated Device Technology (89HF16P04AG3 and 89HF32P08AG3) in August 2012. [24] [25] The first NVMe drive, Samsung's XS1715 enterprise drive, was announced in July 2013; according to Samsung, this drive supported 3 GB/s read speeds, six times faster than their previous enterprise offerings. [26] The LSI SandForce SF3700 controller family, released in November 2013, also supports NVMe. [27] [28] A Kingston HyperX "prosumer" product using this controller was showcased at the Consumer Electronics Show 2014 and promised similar performance. [29] [30] In June 2014, Intel announced their first NVM Express products, the Intel SSD data center family that interfaces with the host through PCI Express bus, which includes the DC P3700 series, the DC P3600 series, and the DC P3500 series. [31] As of November 2014 [update] , NVMe drives are commercially available.
In March 2014, the group incorporated to become NVM Express, Inc., which as of November 2014 [update] consists of more than 65 companies from across the industry. NVM Express specifications are owned and maintained by NVM Express, Inc., which also promotes industry awareness of NVM Express as an industry-wide standard. NVM Express, Inc. is directed by a thirteen-member board of directors selected from the Promoter Group, which includes Cisco, Dell, EMC, HGST, Intel, Micron, Microsoft, NetApp, Oracle, PMC, Samsung, SanDisk and Seagate. [32]
In September 2016, the CompactFlash Association announced that it would be releasing a new memory card specification, CFexpress, which uses NVMe.[ citation needed ]
NVMe Host Memory Buffer (HMB) added in version 1.2 of the NVMe specification. [33] HMB allows SSDs to utilize the host's DRAM, which can improve the I/O performance for DRAM-less SSDs. [34] For example, HMB can be used for cache the FTL table by the SSD controller, which can improve I/O performance. [35] NVMe 2.0 added Zoned Namespaces (ZNS) and support for rotating media such as hard drives. ZNS allows data to be mapped directly to its physical location in flash memory to directly access data on an SSD without a flash translation layer. [36]
There are many form factors of NVMe solid-state drive, such as AIC, U.2, U.3, M.2 etc.
Almost all early NVMe solid-state drives are HHHL (half height, half length) or FHHL (full height, half length) AIC, with a PCIe 2.0 or 3.0 interface. A HHHL NVMe solid-state drive card is easy to insert into a PCIe slot of a server.
U.2, formerly known as SFF-8639, is a computer interface for connecting solid-state drives to a computer. It uses up to four PCI Express lanes. Available servers can combine up to 48 U.2 NVMe solid-state drives. [37]
U.3 is built on the U.2 spec and uses the same SFF-8639 connector. It is a 'tri-mode' standard, combining SAS, SATA and NVMe support into a single controller. U.3 can also support hot-swap between the different drives where firmware support is available. U.3 drives are still backward compatible with U.2, but U.2 drives are not compatible with U.3 hosts.
M.2, formerly known as the Next Generation Form Factor (NGFF), uses a M.2 NVMe solid-state drive computer bus. Interfaces provided through the M.2 connector are PCI Express 3.0 or higher (up to four lanes).
NVM Express over Fabrics (NVMe-oF) is the concept of using a transport protocol over a network to connect remote NVMe devices, contrary to regular NVMe where physical NVMe devices are connected to a PCIe bus either directly or over a PCIe switch to a PCIe bus. In August 2017, a standard for using NVMe over Fibre Channel (FC) was submitted by the standards organization International Committee for Information Technology Standards (ICITS), and this combination is often referred to as FC-NVMe or sometimes NVMe/FC. [38]
As of May 2021, supported NVMe transport protocols are:
The standard for NVMe over Fabrics was published by NVM Express, Inc. in 2016. [43] [44]
The following software implements the NVMe-oF protocol:
The Advanced Host Controller Interface (AHCI) has the benefit of wide software compatibility, but has the downside of not delivering optimal performance when used with SSDs connected via the PCI Express bus. As a logical-device interface, AHCI was developed when the purpose of a host bus adapter (HBA) in a system was to connect the CPU/memory subsystem with a much slower storage subsystem based on rotating magnetic media. As a result, AHCI introduces certain inefficiencies when used with SSD devices, which behave much more like RAM than like spinning media. [7]
The NVMe device interface has been designed from the ground up, capitalizing on the lower latency and parallelism of PCI Express SSDs, and complementing the parallelism of contemporary CPUs, platforms and applications. At a high level, the basic advantages of NVMe over AHCI relate to its ability to exploit parallelism in host hardware and software, manifested by the differences in command queue depths, efficiency of interrupt processing, the number of uncacheable register accesses, etc., resulting in various performance improvements. [7] [54] : 17–18
The table below summarizes high-level differences between the NVMe and AHCI logical-device interfaces.
AHCI | NVMe | |
---|---|---|
Maximum queue depth | One command queue; Up to 32 commands per queue | Up to 65535 queues; [55] Up to 65536 commands per queue |
Uncacheable register accesses (2000 cycles each) | Up to six per non-queued command; Up to nine per queued command | Up to two per command |
Interrupt | A single interrupt | Up to 2048 MSI-X interrupts |
Parallelism and multiple threads | Requires synchronization lock to issue a command | No locking |
Efficiency for 4 KB commands | Command parameters require two serialized host DRAM fetches | Gets command parameters in one 64-byte fetch |
Data transmission | Usually half-duplex | Full-duplex |
Host Memory Buffer (HMB) | No | Yes |
The nvmecontrol
tool is used to control an NVMe disk from the command line on FreeBSD. It was added in FreeBSD 9.2. [93]
NVM-Express user space tooling for Linux. [94]
PCI Express, officially abbreviated as PCIe or PCI-e, is a high-speed serial computer expansion bus standard, designed to replace the older PCI, PCI-X and AGP bus standards. It is the common motherboard interface for personal computers' graphics cards, sound cards, hard disk drive host adapters, SSDs, Wi-Fi and Ethernet hardware connections. PCIe has numerous improvements over the older standards, including higher maximum system bus throughput, lower I/O pin count and smaller physical footprint, better performance scaling for bus devices, a more detailed error detection and reporting mechanism, and native hot-swap functionality. More recent revisions of the PCIe standard provide hardware support for I/O virtualization.
SATA is a computer bus interface that connects host bus adapters to mass storage devices such as hard disk drives, optical drives, and solid-state drives. Serial ATA succeeded the earlier Parallel ATA (PATA) standard to become the predominant interface for storage devices.
The southbridge is one of the two chips in the core logic chipset on older personal computer (PC) motherboards, the other being the northbridge. As of 2023, most personal computer devices no longer use a set of two chips, and instead have a single chip acting as the 'chipset', for example Intel's Z790 chipset.
In computing, Native Command Queuing (NCQ) is an extension of the Serial ATA protocol allowing hard disk drives to internally optimize the order in which received read and write commands are executed. This can reduce the amount of unnecessary drive head movement, resulting in increased performance for workloads where multiple simultaneous read/write requests are outstanding, most often occurring in server-type applications.
The Advanced Host Controller Interface (AHCI) is a technical standard defined by Intel that specifies the register-level interface of Serial ATA (SATA) host controllers in a non-implementation-specific manner in its motherboard chipsets.
Input/output operations per second is an input/output performance measurement used to characterize computer storage devices like hard disk drives (HDD), solid state drives (SSD), and storage area networks (SAN). Like benchmarks, IOPS numbers published by storage device manufacturers do not directly relate to real-world application performance.
A solid-state drive (SSD) is a solid-state storage device. It provides persistent data storage using no moving parts.
JMicron Technology Corporation is a Taiwan based fabless technology design house based in Hsinchu, Taiwan. As a manufacturer of integrated circuits, they produce controller chips for bridge devices.
I/O Controller Hub (ICH) is a family of Intel southbridge microchips used to manage data communications between a CPU and a motherboard, specifically Intel chipsets based on the Intel Hub Architecture. It is designed to be paired with a second support chip known as a northbridge. As with any other southbridge, the ICH is used to connect and control peripheral devices.
Intel Rapid Storage Technology (RST) is a driver SATA AHCI and a firmware-based RAID solution built into a wide range of Intel chipsets. Currently also is installed as a driver for Intel Optane temporary storage units.
A trim command allows an operating system to inform a solid-state drive (SSD) which blocks of data are no longer considered to be "in use" and therefore can be erased internally.
USB Attached SCSI (UAS) or USB Attached SCSI Protocol (UASP) is a computer protocol used to move data to and from USB storage devices such as hard drives (HDDs), solid-state drives (SSDs), and thumb drives. UAS depends on the USB protocol, and uses the standard SCSI command set. Use of UAS generally provides faster transfers compared to the older USB Mass Storage Bulk-Only Transport (BOT) drivers.
The Intel X79 is a Platform Controller Hub (PCH) designed and manufactured by Intel for their LGA 2011 and LGA 2011-1.
M.2, pronounced m dot two and formerly known as the Next Generation Form Factor (NGFF), is a specification for internally mounted computer expansion cards and associated connectors. M.2 replaces the mSATA standard, which uses the PCI Express Mini Card physical card layout and connectors. Employing a more flexible physical specification, M.2 allows different module widths and lengths, which, paired with the availability of more advanced interfacing features, makes M.2 more suitable than mSATA in general for solid-state storage applications, particularly in smaller devices such as ultrabooks and tablets.
SATA Express is a computer bus interface that supports both Serial ATA (SATA) and PCI Express (PCIe) storage devices, initially standardized in the SATA 3.2 specification. The SATA Express connector used on the host side is backward compatible with the standard SATA data connector, while it also provides two PCI Express lanes as a pure PCI Express connection to the storage device.
bhyve is a type-2 hypervisor initially written for FreeBSD. It can also be used on a number of illumos based distributions including SmartOS, OpenIndiana, and OmniOS. A port of bhyve to macOS called xhyve is also available.
Solid-state storage (SSS) is non-volatile computer storage that has no moving parts – uses only electronic circuits. This solid-state design dramatically differs from the commonly-used competing technology of electromechanical magnetic storage which uses moving media coated with magnetic material. Generally, SSS is much faster, but is more expensive.
U.2, using the port SFF-8639, is a computer interface standard for connecting solid-state drives (SSDs) to a computer. It covers the physical connector, electrical characteristics, and communication protocols.
3D XPoint is a discontinued non-volatile memory (NVM) technology developed jointly by Intel and Micron Technology. It was announced in July 2015 and was available on the open market under the brand name Optane (Intel) from April 2017 to July 2022. Bit storage is based on a change of bulk resistance, in conjunction with a stackable cross-grid data access array. Initial prices are less than dynamic random-access memory (DRAM) but more than flash memory.
NVMe is designed from the ground up to deliver high bandwidth and low latency storage access for current and future NVM technologies.