 | This article needs to be updated.(June 2023) |
ATA over Ethernet (AoE) is a network protocol developed by the Brantley Coile Company, designed for simple, high-performance access of block storage devices over Ethernet networks. It is used to build storage area networks (SANs) with low-cost, standard technologies.
AoE runs on layer 2 Ethernet. AoE does not use Internet Protocol (IP); it cannot be accessed over the Internet or other IP networks. In this regard it is more comparable to Fibre Channel over Ethernet than iSCSI.
With fewer protocol layers, this approach makes AoE fast and lightweight. It also makes the protocol relatively easy to implement and offers linear scalability with high performance. The AoE specification is 12 pages [1] compared with iSCSI's 257 pages. [2]
0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 0 | Ethernet Destination MAC Address | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 4 | Ethernet Destination (cont) | Ethernet Source MAC Address | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 8 | Ethernet Source MAC Address (cont) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 12 | Ethernet Type (0x88A2) | Ver | Flags | Error | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 16 | Major | Minor | Command | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 20 | Tag | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 24 | Arg | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
AoE has the IEEE assigned EtherType 0x88A2.
SATA (and older PATA) hard drives use the Advanced Technology Attachment (ATA) protocol to issue commands, such as read, write, and status. AoE encapsulates those commands inside Ethernet frames and lets them travel over an Ethernet network instead of a SATA or 40-pin ribbon cable. Although internally AoE uses the ATA protocol, it presents the disks as SCSI to the operating system. Also the actual disks can be SCSI or any other kind, AoE is not limited to disks that use the ATA command set. By using an AoE driver, the host operating system is able to access a remote disk as if it were directly attached.
The encapsulation of ATA provided by AoE is simple and low-level, allowing the translation to happen either at high performance or inside a small, embedded device, or both.
AoE is a layer 2 protocol running at the data-link layer, unlike some other SAN protocols which run on top of layer 3 utilizing IP. While this reduces the significant processing overhead of TCP/IP, this means that routers cannot route AoE data across disparate networks (such as a campus network or the Internet). Instead, AoE packets can only travel within a single local Ethernet storage area network (e.g., a set of computers connected to the same switch or in the same LAN Subnet or VLAN).
The non-routability of AoE is the only security mechanism (i.e., an intruder can't connect through a router—they must physically plug into the local Ethernet switch where Ethernet frame tunneling over routed networks is not in use). However, there are no AoE-specific mechanisms for password verification or encryption. The protocol provides for AoE targets such as Coraid Storage appliances, vblade and GGAOED to establish access lists ("masks") allowing connections only from specific MAC addresses (although these can be spoofed). Most secure AoE by using Ethernet VLANs.
The AoE protocol provides a mechanism for host-based cooperative locking. When more than one AoE initiator is using an AoE target they must communicate to avoid interfering with one another as they read and write the config string data on the shared AoE device. Without this cooperation file-system corruption and data loss is likely, unless access is strictly read-only or a cluster file system is used.
One option provided by AoE is to use the storage device itself as the mechanism for determining specific host access. This is the AoE "config string" feature. [3] The config string can record who is using the device, as well as other information. If more than one host tries to set the config string simultaneously, only one succeeds. The other host is informed of the conflict.
The following operating systems provide ATA over Ethernet (AoE) support:
| OS | Support | Third-party drivers |
|---|---|---|
| Linux | Native (2.6.11+) | Coraid |
| Windows | Third-party | StarWind Software AoE Initiator, [4] WinAoE, [5] WinVBlock [6] |
| Mac OS X 10.4 and up | Third-party | From 2006 through 2010, 2°Frost Technologies developed proprietary software and marketed AoE storage solutions in the Windows and Mac markets. The Mac implementation was its own and the Windows version was OEM-ed from StarWind Software. |
| Mac OS X 10.5 and 10.6 | Third-party | Small Tree Communications [7] |
| Solaris | Third-party | Coraid |
| FreeBSD | Third-party | Coraid (outdated) |
| OpenBSD | Native (4.5 to 5.6 [8] ) | |
| VMware | Third-party | Coraid |
| Plan 9 from Bell Labs [9] | Native |
Coraid offered an array of AoE SAN appliances under the EtherDrive brand, along with diskless gateways that add network-attached storage functionality, using the NFS or SMB protocols, to one or more AoE appliances. The Coraid brand is now owned by SouthSuite, Inc., a company founded by Brantley Coile who founded Coraid.
In 2007, LayerWalker [10] announced AoE hardware called miniSAN [11] running at both Fast and Gigabit Ethernet. The miniSAN product family offers standard AoE server functions plus other management features that targets PC, consumer and small and medium businesses markets.
Although AoE is a simple network protocol, it opens up a complex realm of storage possibilities. To understand and evaluate these storage scenarios, it helps to be familiar with a few concepts.
A SAN allows the physical hard drive to be removed from the server that uses it, and placed on the network. A SAN interface is similar in principle to non-networked interfaces such as SATA or SCSI. Most users will not use a SAN interface directly. Instead, they will connect to a server that uses a SAN disk instead of a local disk. Direct connection, however, can also be used.
When using a SAN network to access storage, there are several potential advantages over a local disk:
To use a SAN disk, the host must format it with a filesystem. Unlike a SATA or SCSI disk, however, a SAN hard drive may be accessed by multiple machines. This is a source of both danger and opportunity.
Traditional filesystems (such as FAT or ext3) are designed to be accessed by a single host, and will cause unpredictable behavior if accessed by multiple machines. Such filesystems may be used, and AoE provides mechanisms whereby an AoE target can be guarded against simultaneous access (see: Config String).
Shared disk file systems allow multiple machines to use a single hard disk safely by coordinating simultaneous access to individual files. These filesystems can be used to allow multiple machines access to the same AoE target without an intermediate server or filesystem (and at higher performance).
Small Computer System Interface is a set of standards for physically connecting and transferring data between computers and peripheral devices, best known for its use with storage devices such as hard disk drives. SCSI was introduced in the 1980s and has seen widespread use on servers and high-end workstations, with new SCSI standards being published as recently as SAS-4 in 2017.
Internet Small Computer Systems Interface or iSCSI is an Internet Protocol-based storage networking standard for linking data storage facilities. iSCSI provides block-level access to storage devices by carrying SCSI commands over a TCP/IP network. iSCSI facilitates data transfers over intranets and to manage storage over long distances. It can be used to transmit data over local area networks (LANs), wide area networks (WANs), or the Internet and can enable location-independent data storage and retrieval.
Fibre Channel (FC) is a high-speed data transfer protocol providing in-order, lossless delivery of raw block data. Fibre Channel is primarily used to connect computer data storage to servers in storage area networks (SAN) in commercial data centers.
In computer hardware a host controller, host adapter or host bus adapter (HBA) connects a computer system bus which acts as the host system to other network and storage devices. The terms are primarily used to refer to devices for connecting SCSI, SAS, NVMe, Fibre Channel and SATA devices. Devices for connecting to FireWire, USB and other devices may also be called host controllers or host adapters.
A disk array controller is a device that manages the physical disk drives and presents them to the computer as logical units. It almost always implements hardware RAID, thus it is sometimes referred to as RAID controller. It also often provides additional disk cache.
TCP offload engine (TOE) is a technology used in some network interface cards (NIC) to offload processing of the entire TCP/IP stack to the network controller. It is primarily used with high-speed network interfaces, such as gigabit Ethernet and 10 Gigabit Ethernet, where processing overhead of the network stack becomes significant. TOEs are often used as a way to reduce the overhead associated with Internet Protocol (IP) storage protocols such as iSCSI and Network File System (NFS).
In computing, Serial Attached SCSI (SAS) is a point-to-point serial protocol that moves data to and from computer-storage devices such as hard disk drives and tape drives. SAS replaces the older Parallel SCSI bus technology that first appeared in the mid-1980s. SAS, like its predecessor, uses the standard SCSI command set. SAS offers optional compatibility with Serial ATA (SATA), versions 2 and later. This allows the connection of SATA drives to most SAS backplanes or controllers. The reverse, connecting SAS drives to SATA backplanes, is not possible.
In computer data storage, a SCSI initiator is the endpoint that initiates a SCSI session, that is, sends a SCSI command. The initiator usually does not provide any Logical Unit Numbers (LUNs).
The USB mass storage device class is a set of computing communications protocols, specifically a USB Device Class, defined by the USB Implementers Forum that makes a USB device accessible to a host computing device and enables file transfers between the host and the USB device. To a host, the USB device acts as an external hard drive; the protocol set interfaces with a number of storage devices.
Network Direct Attached Storage (NDAS) is a proprietary storage area network system, originally marketed by the company Ximeta, for connecting external digital storage devices such as hard-disks, flash memory and tape drives via the Ethernet family of computer networks. Unlike other more common forms of networked storage, NDAS does not use TCP/IP to communicate over the network. Instead a Lean Packet Exchange (LPX) protocol is used. NDAS also supports some limited RAID functions such as aggregation and mirroring.
On Linux, network block device (NBD) is a network protocol that can be used to forward a block device from one machine to a second machine. As an example, a local machine can access a hard disk drive that is attached to another computer.
LayerWalker Technology, Inc. is a fabless integrated circuit design company that announced a network storage system on a chip (SoC). Their products targeted digital home, small business and consumer electronics markets.
Fibre Channel over Ethernet (FCoE) is a computer network technology that encapsulates Fibre Channel frames over Ethernet networks. This allows Fibre Channel to use 10 Gigabit Ethernet networks while preserving the Fibre Channel protocol. The specification was part of the International Committee for Information Technology Standards T11 FC-BB-5 standard published in 2009. FCoE did not see widespread adoption.
SCSI / ATA Translation (SAT) is a set of standards developed by the T10 subcommittee, defining how to communicate with ATA devices through a SCSI application layer. The standard attempts to be consistent with the SCSI architectural model, the SCSI Primary Commands, and the SCSI Block Commands standards.
A storage area network (SAN) or storage network is a computer network which provides access to consolidated, block-level data storage. SANs are primarily used to access data storage devices, such as disk arrays and tape libraries from servers so that the devices appear to the operating system as direct-attached storage. A SAN typically is a dedicated network of storage devices not accessible through the local area network (LAN).
EtherDrive is a brand name for storage area network devices based upon the ATA over Ethernet (AoE) protocol. It was registered with the United States Patent and Trademark Office in 2004. The word was invented by Brantley Coile as a portmanteau of the words Ethernet and disk drive. EtherDrive was a trademark by Coraid from 2002 until 2015 when it was purchased by The Brantley Coile Company as part of the purchase of Coraid's software copyrights, trademarks and trade secrets in May 2015.
Coraid, Inc. is a computer data storage vendor that provides storage area network (SAN) products that use Ethernet, headquartered in Athens, Georgia.
The Linux-IOTarget (LIO) is an open-source SCSI target implementation that is included with the Linux kernel. Internally, LIO does not initiate sessions, but instead provides one or more Logical Unit Numbers (LUNs), waits for SCSI commands from a SCSI initiator, and performs required input/output data transfers. LIO supports common storage fabrics, including FCoE, Fibre Channel, IEEE 1394, iSCSI, iSCSI Extensions for RDMA (iSER), SCSI RDMA Protocol (SRP) and USB. It is included in some Linux distributions; native support for LIO in QEMU/KVM, libvirt, and OpenStack makes LIO also a storage option for cloud deployments.
Brantley Coile is an inventor and founder of network technology companies, he worked for John Mayes as a programmer whose companies products include PIX Firewall, the first stateful-inspection firewall and Cisco Systems' first load-balancer, LocalDirector. Coile's patents include the fundamental patents on Network Address Translation (NAT).
Hard disk drives are accessed over one of a number of bus types, including parallel ATA, Serial ATA (SATA), SCSI, Serial Attached SCSI (SAS), and Fibre Channel. Bridge circuitry is sometimes used to connect hard disk drives to buses with which they cannot communicate natively, such as IEEE 1394, USB, SCSI, NVMe and Thunderbolt.