RapidIO

Last updated
RapidIO
RapidIO logo fair use.jpg
RapidIO - the unified fabric for Performance Critical Computing
Year created2000;24 years ago (2000)
Width in bitsPort widths of 1, 2, 4, 8, and 16 lanes
No. of devicesSizes of 256, 65,536, and 4,294,967,296
SpeedPer lane (each direction):
  • 1.x: 1.25, 2.5, 3.125 Gigabaud
  • 2.x: added 5 and 6.25 Gigabaud
  • 3.x: added 10.3125 Gigabaud
  • 4.x: added 12.5 and 25.3125 Gigabaud
Style Serial
Hotplugging interfaceYes
External interfaceYes, Chip-Chip, Board-Board (Backplane), Chassis-Chassis
Website www.rapidio.org

The RapidIO architecture is a high-performance packet-switched electrical connection technology. It supports messaging, read/write and cache coherency semantics. Based on industry-standard electrical specifications such as those for Ethernet, RapidIO can be used as a chip-to-chip, board-to-board, and chassis-to-chassis interconnect.

Contents

History

The RapidIO protocol was originally designed by Mercury Computer Systems and Motorola (Freescale) as a replacement for Mercury's RACEway proprietary bus and Freescale's PowerPC bus. [1] The RapidIO Trade Association was formed in February 2000, and included telecommunications and storage OEMs as well as FPGA, processor, and switch companies.

Releases

The RapidIO specification revision 1.1 (3xN Gen1), released in March 2001, defined a wide, parallel bus. This specification did not achieve extensive commercial adoption.

The RapidIO specification revision 1.2, released in June 2002, [2] defined a serial interconnection based on the XAUI physical layer. Devices based on this specification achieved significant commercial success within wireless baseband, [3] imaging and military computing. [4]

The RapidIO specification revision 1.3 was released in June 2005.

The RapidIO specification revision 2.0 (6xN Gen2), was released in March 2008. [5] This added more port widths (2×, 8×, and 16×) and increased the maximum lane speed to 6.25  GBd / 5 Gbit/s.

The RapidIO specification revision 2.1 was released in September 2009.

The RapidIO specification revision 2.2 was released in May 2011.

The RapidIO specification revision 3.0 (10xN Gen3) released in October 2013. [6] The following changes were made:

The RapidIO specification revision 3.1, was released in October 2014. [7] It was developed through a collaboration between the RapidIO Trade Association and NGSIS. Revision 3.1 has the following changes compared to the 3.0 specification:

The RapidIO specification revision 3.2 was released in February 2016.

The RapidIO specification revision 4.0 (25xN Gen4) was released in June 2016. [8] It had the following changes compared to the 3.x specifications:

The RapidIO specification revision 4.1 was released in July 2017. [9]

Wireless infrastructure

RapidIO fabrics are used in cellular infrastructure 3G, 4G and LTE networks with millions of RapidIO ports shipped [10] into wireless base stations worldwide. RapidIO fabrics were originally designed to support connecting different types of processors from different manufacturers together in a single system. This flexibility has driven the widespread use of RapidIO in wireless infrastructure equipment where there is a need to combine heterogeneous, DSP, FPGA and communication processors together in a tightly coupled system with low latency and high reliability.

Data centers

Data center and HPC analytics systems have been deployed using a RapidIO 2D Torus Mesh Fabric, [11] that provides a high speed general purpose interface among the system cartridges. This allows for applications that benefit from high bandwidth to low latency node-to-node communication. The RapidIO 2D Torus unified fabric is routed as a torus ring configuration connecting up to 45 server cartridges. Hence, capable of providing 5Gbs per lane connections in each direction to its north, south, east and west neighbors. This allows the system to meet many unique HPC applications where efficient localized traffic is needed.

Also, using an open modular data center and compute platform, [12] a heterogeneous HPC system has showcased the low latency attribute of RapidIO to enable real-time analytics. [13] In March 2015 a top-of-rack switch was announced to drive RapidIO into mainstream data center applications. [14]

Aerospace

The interconnect or "bus" is one of the critical technologies in the design and development of spacecraft avionic systems that dictate its architecture and level of complexity. There are a host of existing architectures that are still in use given their level of maturity. These existing systems are sufficient for a given type of architecture need and requirement. Unfortunately, for next generation missions a more capable avionics architecture is desired; which is well beyond the capabilities levied by existing architectures. A viable option for the design and development of these next generation architectures is to leverage existing commercial protocols capable of accommodating high levels of data transfer.

In 2012, RapidIO was selected by the Next Generation Spacecraft Interconnect Standard (NGSIS) working group to serve as the foundation for standard communication interconnects to be used in spacecraft. The NGSIS is an umbrella standards effort that includes RapidIO Version 3.1 development, and a box level hardware standards effort under VITA 78 called SpaceVPX or High ReliabilityVPX. The NGSIS requirements committee developed extensive requirements criteria with 47 different elements for the NGSIS interconnect. Independent trade study results by NGSIS member companies demonstrated the superiority of RapidIO over other existing commercial protocols, such as InfiniBand, Fibre Channel, and 10G Ethernet. As a result, the group decided that RapidIO offered the best overall interconnect for the needs of next-generation spacecraft. [15]

PHY roadmap

The RapidIO roadmap aligns with Ethernet PHY development. RapidIO specifications for 50 GBd and higher links are under investigation. [16]

Terminology

Link Partner
One end of a RapidIO link.
Endpoint
A device that can originate and/or terminate RapidIO packets.
Processing Element
A device which has at least one RapidIO port
Switch
A device that can route RapidIO packets.

Protocol overview

The RapidIO protocol is defined in a 3-layered specification:

System specifications include:

Physical layer

The RapidIO electrical specifications are based on industry-standard Ethernet and Optical Interconnect Forum standards:

The RapidIO PCS/PMA layer supports two forms of encoding/framing:

Every RapidIO processing element transmits and receives three kinds of information: Packets, control symbols, and an idle sequence.

Packets

Every packet has two values that control the physical layer exchange of that packet. The first is an acknowledge ID (ackID), which is the link-specific, unique, 5-, 6-, or 12-bit value that is used to track packets exchanged on a link. Packets are transmitted with serially increasing ackID values. Because the ackID is specific to a link, the ackID is not covered by CRC, but by protocol. This allows the ackID to change with each link it passes over, while the packet CRC can remain a constant end-to-end integrity check of the packet. When a packet is successfully received, it is acknowledged using the ackID of the packet. A transmitter must retain a packet until it has been successfully acknowledged by the link partner.

The second value is the packet's physical priority. The physical priority is composed of the Virtual Channel (VC) identifier bit, the Priority bits, and the Critical Request Flow (CRF) bit. The VC bit determines if the Priority and CRF bits identify a Virtual Channel from 1 to 8, or are used as the priority within Virtual Channel 0. Virtual Channels are assigned guaranteed minimum bandwidths. Within Virtual Channel 0, packets of higher priority can pass packets of lower priority. Response packets must have a physical priority higher than requests in order to avoid deadlock.

The physical layer contribution to RapidIO packets is a 2-byte header at the beginning of each packet that includes the ackID and physical priority, and a final 2-byte CRC value to check the integrity of the packet. Packets larger than 80 bytes also have an intermediate CRC after the first 80 bytes. With one exception a packet's CRC value(s) acts as an end-to-end integrity check.

Control symbols

RapidIO control symbols can be sent at any time, including within a packet. This gives RapidIO the lowest possible in-band control path latency, enabling the protocol to achieve high throughput with smaller buffers than other protocols.

Control symbols are used to delimit packets (Start of Packet, End of Packet, Stomp), to acknowledge packets (Packet Acknowledge, Packet Not Acknowledged), reset (Reset Device, Reset Port) and to distribute events within the RapidIO system (Multicast Event Control Symbol). Control symbols are also used for flow control (Retry, Buffer Status, Virtual Output Queue Backpressure) and for error recovery.

The error recovery procedure is very fast. When a receiver detects a transmission error in the received data stream, the receiver causes its associated transmitter to send a Packet Not Accepted control symbol. When the link partner receives a Packet Not Accepted control symbol, it stops transmitting new packets and sends a Link Request/Port Status control symbol. The Link Response control symbol indicates the ackID that should be used for the next packet transmitted. Packet transmission then resumes.

IDLE sequence

The IDLE sequence is used during link initialization for signal quality optimization. It is also transmitted when the link does not have any control symbols or packets to send.

Transport layer

Every RapidIO endpoint is uniquely identified by a Device Identifier (deviceID). Each RapidIO packet contains two device IDs. The first is the destination ID (destID), which indicates where the packet should be routed. The second is the source ID (srcID), which indicates where the packet originated. When an endpoint receives a RapidIO request packet that requires a response, the response packet is composed by swapping the srcID and destID of the request.

RapidIO switches use the destID of received packets to determine the output port or ports that should forward the packet. Typically, the destID is used to index into an array of control values. The indexing operation is fast and low cost to implement. RapidIO switches support a standard programming model for the routing table, which simplifies system control.

The RapidIO transport layer supports any network topology, from simple trees and meshes to n-dimensional hypercubes, multi-dimensional toroids, and more esoteric architectures such as entangled networks.

The RapidIO transport layer enables hardware virtualization (for example, a RapidIO endpoint can support multiple device IDs). Portions of the destination ID of each packet can be used to identify specific pieces of virtual hardware within the endpoint.

Logical layer

The RapidIO logical layer is composed of several specifications, each providing packet formats and protocols for different transaction semantics.

Logical I/O

The logical I/O layer defines packet formats for read, write, write-with-response, and various atomic transactions. Examples of atomic transactions are set, clear, increment, decrement, swap, test-and-swap, and compare-and-swap.

Messaging

The Messaging specification defines Doorbells and Messages. Doorbells communicate a 16-bit event code. Messages transfer up to 4KiB of data, segmented into up to 16 packets each with a maximum payload of 256 bytes. Response packets must be sent for each Doorbell and Message request. The response packet status value indicates done, error, or retry. A status of retry requests the originator of the request to send the packet again. The logical level retry response allows multiple senders to access a small number of shared reception resources, leading to high throughput with low power.

Flow control

The Flow Control specification defines packet formats and protocols for simple XON/XOFF flow control operations. Flow control packets can be originated by switches and endpoints. Reception of a XOFF flow control packet halts transmission of a flow or flows until an XON flow control packet is received or a timeout occurs. Flow Control packets can also be used as a generic mechanism for managing system resources.

CC-NUMA

The Globally Shared Memory specification defines packet formats and protocols for operating a cache coherent shared memory system over a RapidIO network.

Data streaming

The Data Streaming specification supports messaging with different packet formats and semantics than the Messaging specification. Data Streaming packet formats support the transfer of up to 64K of data, segmented over multiple packets. Each transfer is associated with a Class of Service and Stream Identifier, enabling thousands of unique flows between endpoints.

The Data Streaming specification also defines Extended Header flow control packet formats and semantics to manage performance within a client-server system. Each client uses extended header flow control packets to inform the server of the amount of work that could be sent to the server. The server responds with extended header flow control packets that use XON/XOFF, rate, or credit based protocols to control how quickly and how much work the client sends to the server.

System initialization

Systems with a known topology can be initialized in a system specific manner without affecting interoperability. The RapidIO system initialization specification supports system initialization when system topology is unknown or dynamic. System initialization algorithms support the presence of redundant hosts, so system initialization need not have a single point of failure.

Each system host recursively enumerates the RapidIO fabric, seizing ownership of devices, allocating device IDs to endpoints and updating switch routing tables. When a conflict for ownership occurs, the system host with the larger deviceID wins. The "losing" host releases ownership of its devices and retreats, waiting for the "winning" host. The winning host completes enumeration, including seizing ownership of the losing host. Once enumeration is complete, the winning host releases ownership of the losing host. The losing host then discovers the system by reading the switch routing tables and registers on each endpoint to learn the system configuration. If the winning host does not complete enumeration in a known time period, the losing host determines that the winning host has failed and completes enumeration.

System enumeration is supported in Linux by the RapidIO subsystem.

Error management

RapidIO supports high availability, fault tolerant system design, including hot swap. The error conditions that require detection, and standard registers to communicate status and error information, are defined. A configurable isolation mechanism is also defined so that when it is not possible to exchange packets on a link, packets can be discarded to avoid congestion and enable diagnosis and recovery activities. In-band (port-write packet) and out-of-band (interrupt) notification mechanisms are defined.

Form factors

The RapidIO specification does not discuss the subjects of form factors and connectors, leaving this to specific application-focussed communities. RapidIO is supported by the following form factors:

Software

Processor-agnostic RapidIO support is found in the Linux kernel.[ citation needed ]

Applications

The RapidIO interconnect is used extensively in the following applications[ citation needed ]:

RapidIO is expanding into supercomputing, server, and storage applications. [ citation needed ]

Competing protocols

PCI Express is targeted at the host to peripheral market, as opposed to embedded systems. Unlike RapidIO, PCIe is not optimized for peer-to-peer multi processor networks. PCIe is ideal for host to peripheral communication. PCIe does not scale as well in large multiprocessor peer-to-peer systems, as the basic PCIe assumption of a "root complex" creates fault tolerance and system management issues.

Another alternative interconnect technology is Ethernet. Ethernet is a robust approach to linking computers over large geographic areas, where network topology may change unexpectedly, the protocols used are in flux, and link latencies are large. To meet these challenges, systems based on Ethernet require significant amounts of processing power, software and memory throughout the network to implement protocols for flow control, data transfer, and packet routing. RapidIO is optimized for energy efficient, low latency, processor-to-processor communication in fault tolerant embedded systems that span geographic areas of less than one kilometer.

SpaceFibre is a competing technology for space applications. [17]

Time Triggered Ethernet is a competing technology for more complex backplane (VPX) and backbone applications for space (launchers and human-rated integrated avionics).

See also

Related Research Articles

<span class="mw-page-title-main">OSI model</span> Model of communication of seven abstraction layers

The Open Systems Interconnection model is a reference model from the International Organization for Standardization (ISO) that "provides a common basis for the coordination of standards development for the purpose of systems interconnection." In the OSI reference model, the communications between systems are split into seven different abstraction layers: Physical, Data Link, Network, Transport, Session, Presentation, and Application.

<span class="mw-page-title-main">USB</span> Standard for computer data connections

Universal Serial Bus (USB) is an industry standard that allows data exchange and delivery of power between many various types of electronics. It specifies its architecture, in particular its physical interface, and communication protocols for data transfer and power delivery to and from hosts, such as personal computers, to and from peripheral devices, e.g. displays, keyboards, and mass storage devices, and to and from intermediate hubs, which multiply the number of a host's ports.

Intermediate System to Intermediate System is a routing protocol designed to move information efficiently within a computer network, a group of physically connected computers or similar devices. It accomplishes this by determining the best route for data through a packet switching network.

A network switch is networking hardware that connects devices on a computer network by using packet switching to receive and forward data to the destination device.

Trivial File Transfer Protocol (TFTP) is a simple lockstep File Transfer Protocol which allows a client to get a file from or put a file onto a remote host. One of its primary uses is in the early stages of nodes booting from a local area network. TFTP has been used for this application because it is very simple to implement.

<span class="mw-page-title-main">PCI Express</span> Computer expansion bus standard

PCI Express, officially abbreviated as PCIe or PCI-e, is a high-speed serial computer expansion bus standard, designed to replace the older PCI, PCI-X and AGP bus standards. It is the common motherboard interface for personal computers' graphics cards, sound cards, hard disk drive host adapters, SSDs, Wi-Fi and Ethernet hardware connections. PCIe has numerous improvements over the older standards, including higher maximum system bus throughput, lower I/O pin count and smaller physical footprint, better performance scaling for bus devices, a more detailed error detection and reporting mechanism, and native hot-swap functionality. More recent revisions of the PCIe standard provide hardware support for I/O virtualization.

A virtual private network (VPN) is a mechanism for creating a secure connection between a computing device and a computer network, or between two networks, using an insecure communication medium such as the public Internet.

<span class="mw-page-title-main">Medium access control</span> Service layer in IEEE 802 network standards

In IEEE 802 LAN/MAN standards, the medium access control (MAC), also called media access control, is the layer that controls the hardware responsible for interaction with the wired or wireless transmission medium. The MAC sublayer and the logical link control (LLC) sublayer together make up the data link layer. The LLC provides flow control and multiplexing for the logical link, while the MAC provides flow control and multiplexing for the transmission medium.

<span class="mw-page-title-main">Profinet</span> Computer network protocol

Profinet is an industry technical standard for data communication over Industrial Ethernet, designed for collecting data from, and controlling equipment in industrial systems, with a particular strength in delivering data under tight time constraints. The standard is maintained and supported by Profibus and Profinet International, an umbrella organization headquartered in Karlsruhe, Germany.

<span class="mw-page-title-main">IEEE 1355</span>

IEEE Standard 1355-1995, IEC 14575, or ISO 14575 is a data communications standard for Heterogeneous Interconnect (HIC).

The Link Layer Discovery Protocol (LLDP) is a vendor-neutral link layer protocol used by network devices for advertising their identity, capabilities, and neighbors on a local area network based on IEEE 802 technology, principally wired Ethernet. The protocol is formally referred to by the IEEE as Station and Media Access Control Connectivity Discovery specified in IEEE 802.1AB with additional support in IEEE 802.3 section 6 clause 79.

<span class="mw-page-title-main">Computer network</span> Network that allows computers to share resources and communicate with each other

A computer network is a set of computers sharing resources located on or provided by network nodes. Computers use common communication protocols over digital interconnections to communicate with each other. These interconnections are made up of telecommunication network technologies based on physically wired, optical, and wireless radio-frequency methods that may be arranged in a variety of network topologies.

In mobile-telephone technology, the UniPro protocol stack follows the architecture of the classical OSI Reference Model. In UniPro, the OSI Physical Layer is split into two sublayers: Layer 1 and Layer 1.5 which abstracts from differences between alternative Layer 1 technologies. The actual physical layer is a separate specification as the various PHY options are reused in other MIPI Alliance specifications.

Data center bridging (DCB) is a set of enhancements to the Ethernet local area network communication protocol for use in data center environments, in particular for use with clustering and storage area networks.

MOST is a high-speed multimedia network technology for the automotive industry. It can be used for applications inside or outside the car. The serial MOST bus uses a daisy-chain topology or ring topology and synchronous serial communication to transport audio, video, voice and data signals via plastic optical fiber (POF) or electrical conductor physical layers.

RDMA over Converged Ethernet (RoCE) or InfiniBand over Ethernet (IBoE) is a network protocol which allows remote direct memory access (RDMA) over an Ethernet network. It does this by encapsulating an InfiniBand (IB) transport packet over Ethernet. There are multiple RoCE versions. RoCE v1 is an Ethernet link layer protocol and hence allows communication between any two hosts in the same Ethernet broadcast domain. RoCE v2 is an internet layer protocol which means that RoCE v2 packets can be routed. Although the RoCE protocol benefits from the characteristics of a converged Ethernet network, the protocol can also be used on a traditional or non-converged Ethernet network.

RTP-MIDI is a protocol to transport MIDI messages within Real-time Transport Protocol (RTP) packets over Ethernet and WiFi networks. It is completely open and free, and is compatible both with LAN and WAN application fields. Compared to MIDI 1.0, RTP-MIDI includes new features like session management, device synchronization and detection of lost packets, with automatic regeneration of lost data. RTP-MIDI is compatible with real-time applications, and supports sample-accurate synchronization for each MIDI message.

Time-Sensitive Networking (TSN) is a set of standards under development by the Time-Sensitive Networking task group of the IEEE 802.1 working group. The TSN task group was formed in November 2012 by renaming the existing Audio Video Bridging Task Group and continuing its work. The name changed as a result of the extension of the working area of the standardization group. The standards define mechanisms for the time-sensitive transmission of data over deterministic Ethernet networks.

Deterministic Networking (DetNet) is an effort by the IETF DetNet Working Group to study implementation of deterministic data paths for real-time applications with extremely low data loss rates, packet delay variation (jitter), and bounded latency, such as audio and video streaming, industrial automation, and vehicle control.

<span class="mw-page-title-main">Audio Video Bridging</span> Specifications for synchronized, low-latency streaming through IEEE 802 networks

Audio Video Bridging (AVB) is a common name for the set of technical standards which provide improved synchronization, low-latency, and reliability for switched Ethernet networks. AVB embodies the following technologies and standards:

References

  1. ↑ Fuller, Sam (27 December 2004). "Preface". RapidIO: The Embedded System Interconnect. John Wiley & Sons Ltd. ISBN   0-470-09291-2.
  2. ↑ "RapidIO Standard Revision 1.2". www.rapidio.org. RapidIO Trade Association. 26 June 2002. Retrieved 9 October 2014.
  3. ↑ "Integrated Device Technology 2011 Annual Report" (PDF). www.idt.com. Integrated Device Technology Inc. 6 June 2011. p. 4. Retrieved 9 October 2014.
  4. ↑ Jag Bolaria (October 15, 2013). "RapidIO Reaches for the Clouds". www.linleygroup.com. The Linley Group. Retrieved 9 October 2014.
  5. ↑ "RapidIO Standard Revision 2.0". www.rapidio.org. RapidIO Trade Association. 23 February 2005. Retrieved 9 October 2014.
  6. ↑ "RapidIO Standard Revision 3.0". www.rapidio.org. RapidIO Trade Association. 10 November 2013. Retrieved 9 October 2014.
  7. ↑ "RapidIO Standard Revision 3.1" (PDF). www.rapidio.org. RapidIO Trade Association. 13 October 2014. Retrieved 18 October 2014.
  8. ↑ "RapidIO Standard Revision 4.0". www.rapidio.org. RapidIO Trade Association. June 2016. Retrieved 15 August 2016.
  9. ↑ "RapidIO Standard Revision 4.1". www.rapidio.org. RapidIO Trade Association. July 2017. Retrieved 11 August 2019.
  10. ↑ "Reader Forum: Cloud radio access and small cell networks based on RapidIO". www.rcrwireless.com. 3 December 2012.
  11. ↑ "PayPal Finds Order from Chaos with HPC". hpcwire.com. 24 September 2014.
  12. ↑ "Prodrive Technologies announces its Datacenter - HPC system (DCCP-280) with RapidIO and 10 Gigabit Ethernet - Prodrive Technologies". prodrive-technologies.com. 30 January 2014.
  13. ↑ "IDT, Orange Silicon Valley, NVIDIA Accelerate Computing Breakthrough With RapidIO-based Clusters Ideal for Gaming, Analytics". businesswire.com (Press release). 18 November 2014.
  14. ↑ "Prodrive Technologies Launches PRSB-760G2 for large RapidIO networks - Prodrive Technologies". prodrive-technologies.com. 2 March 2015.
  15. ↑ Patrick Collier (14 October 2013). "Next Generation Space Interconnect Standard (NGSIS): A Modular Open Standards Approach for High Performance Interconnects for Space" (PDF). Reinventing Space Conference. p. 5. Retrieved 9 October 2014.
  16. ↑ "RapidIO Roadmap". www.rapidio.com. RapidIO Trade Association. 10 June 2012. p. 4. Retrieved 9 October 2014.
  17. ↑ "SpaceFibre Overview" (PDF). STAR-Dundee. Archived from the original (PDF) on 22 October 2014. Retrieved 21 October 2014.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.