Linux DM Multipath

Linux DM Multipath
Initial release	17 June 2005;19 years ago Linux Kernel 2.6.12
Repository	https://git.opensvc.com/gitweb.cgi?p=multipath-tools%2F.git;a=log
Written in	C
Operating system	Linux

Last updated August 10, 2024

Device Mapper Multipath Input Output often shortened to DM-Multipathing and abbreviated as DM-MPIO provides input-output (I/O) fail-over and load-balancing by using multipath I/O within Linux for block devices.^[1]^[2]^[3] By utilizing device-mapper, the multipathd daemon provides the host-side logic to use multiple paths of a redundant network to provide continuous availability and higher-bandwidth connectivity between the host server and the block-level device.^[4] DM-MPIO handles the rerouting of block I/O to an alternate path in the event of a path failure. DM-MPIO can also balance the I/O load across all of the available paths that are typically utilized in Fibre Channel (FC) and iSCSI SAN environments.^[5] DM-MPIO is based on the device mapper,^[6] which provides the basic framework that maps one block device onto another.

Considerations

When utilizing Linux DM-MPIO in a datacenter that has other operating systems and multipath solutions, key components of path management must be considered.

Load balancing— The workload is distributed across the available hardware components. Goal: Reduce I/O completion time, maximize throughput, and optimize resources
Path failover and recovery— Utilizes redundant I/O channels to redirect application reads and writes when one or more paths are no longer available.

History

DM-MPIO started as a patch set created by Joe Thornber, and was later maintained by Alasdair G Kergon at Red Hat. It was included in mainline Linux with kernel version 2.6.12, which was released on June 17, 2005.^[7]

Components

DM-MPIO in Linux consists of kernel components and user-space components.

Kernel – device-mapper – block subsystem that provides layering mechanism for block devices.
- dm-multipath – kernel module implementing the multipath device-mapper target.
User-space – multipath-tools – provides the tools to manage multipathed devices by instructing the device-mapper multipath module what to do. The tools consist of:
- Multipath: scans the system for multipathed devices, assembles them, updates the device-mapper's map.^[5]
- Multipathd: daemon that waits for maps events, and then executes multipath and monitors the paths. Marks a path as failed when the path becomes faulty. Depending on the failback policy, it can reactivate the path.^[5]
- Devmap-name: provides a meaningful device-name to udev for devmaps.^[5]
- Kpartx: maps linear devmaps to device partitions to make multipath maps partitionable.^[5]
- Multipath.conf: configuration file for the multipath daemon. Used to overwrite the built-in configuration table of multipathd.

Configuration file

The configuration file /etc/multipath.conf makes many of the DM-MPIO features user-configurable. The multipath command and the kernel daemon multipathd use information found in this file. The file is only consulted during the configuration of the multipath devices. Changes must be made prior to running the multipath command. Changes to the file afterward will require multipath to be executed again.

The multipath.conf has five sections:^[8]

System level defaults (defaults): User can override system level defaults.
Blacklisted devices (blacklist): User specifies the list of devices that are not to be under the control of DM-MPIO.
Blacklist exceptions (blacklist_exceptions): Specific devices to be treated as multipath devices even if listed in the blacklist.
Storage controller specific settings (devices): User specified configuration settings will be applied to devices with specified "Vendor" and "Product" information.
Device specific settings (multipaths): Fine tune the configuration settings for individual LUNs.

Terminology

HBA: Host bus adapters provide the physical interface between the input/output (I/O) host bus of Fibre Channel devices and the underlying Fibre Channel network.^[9]
Path: Connection from the server through the HBA to a specific LUN.
DM Path States: The device mapper's view of the path condition. Only two conditions are possible:
- Active: The last I/O operation sent through this path successfully completed. Analogous to ready path state.
- Failed: The last I/O operation sent through this path did not successfully complete. Analogous to faulty path state.
Failover: When a path is determined to be in a failed state, a path that is in a ready state will be made active.^[10]
Failback: When a failed path is determined to be active again, multipathd may choose to failback to the path as determined by the failback policy.^[11]
Failback Policy: Four options as set in the multipath.conf configuration file.
- Immediate: Immediately failback to the highest priority path.
- Manual: The failed path is not monitored, requires user intervention to failback.
- Followover(for clusters): Only perform automatic failback when the first path of a pathgroup becomes active. This keeps a node from automatically failing back when another node requested the failover.
- Number of seconds: Wait for a specified number of seconds to allow the I/O to stabilize, then failback to the highest priority path.
Active/Active: In a system that has two storage controllers, each controller can process I/O.^[12]
Active/Passive: In a system that has two storage controllers, only one controller at a time is able to process I/O, the other (passive) is in a standby mode.^[12]
LUN: SCSI Logical Unit Number
WWID: Worldwide Identifier is an identifier for the multipath device that is guaranteed to be globally unique and unchanging.

Related Research Articles

In the context of an operating system, a device driver is a computer program that operates or controls a particular type of device that is attached to a computer or automaton. A driver provides a software interface to hardware devices, enabling operating systems and other computer programs to access hardware functions without needing to know precise details about the hardware being used.

XFS is a high-performance 64-bit journaling file system created by Silicon Graphics, Inc (SGI) in 1993. It was the default file system in SGI's IRIX operating system starting with its version 5.3. XFS was ported to the Linux kernel in 2001; as of June 2014, XFS is supported by most Linux distributions; Red Hat Enterprise Linux uses it as its default file system.

Security-Enhanced Linux (SELinux) is a Linux kernel security module that provides a mechanism for supporting access control security policies, including mandatory access controls (MAC).

In computing, the Global File System 2 or GFS2 is a shared-disk file system for Linux computer clusters. GFS2 allows all members of a cluster to have direct concurrent access to the same shared block storage, in contrast to distributed file systems which distribute data throughout the cluster. GFS2 can also be used as a local file system on a single computer.

In computer storage, multipath I/O is a fault-tolerance and performance-enhancement technique that defines more than one physical path between the CPU in a computer system and its mass-storage devices through the buses, controllers, switches, and bridge devices connecting them.

In Linux, Logical Volume Manager (LVM) is a device mapper framework that provides logical volume management for the Linux kernel. Most modern Linux distributions are LVM-aware to the point of being able to have their root file systems on a logical volume.

Veritas Storage Foundation (VSF), previously known as Veritas Foundation Suite, is a computer software product made by Veritas Software that combines Veritas Volume Manager (VxVM) and Veritas File System (VxFS) to provide online-storage management. Symantec Corporation developed and maintained VSF until January 29, 2016, at which point Veritas and Symantec separated. The latest product version, 7.0, was re-branded as "Veritas InfoScale 7.0".

The device mapper is a framework provided by the Linux kernel for mapping physical block devices onto higher-level virtual block devices. It forms the foundation of the logical volume manager (LVM), software RAIDs and dm-crypt disk encryption, and offers additional features such as file system snapshots.

NetworkManager is a daemon that sits on top of libudev and other Linux kernel interfaces and provides a high-level interface for the configuration of the network interfaces.

Solaris Multiplexed I/O (MPxIO), known also as Sun StorageTek Traffic Manager, is multipath I/O software for Solaris/illumos. It enables a storage device to be accessed through multiple host controller interfaces from a single operating system instance. The MPxIO architecture helps protect against I/O outages due to I/O controller failures. Should one I/O controller fail, MPxIO automatically switches to an alternate controller.

Enterprise Volume Management System (EVMS) was a flexible, integrated volume management software used to manage storage systems under Linux.

mdadm is a Linux utility used to manage and monitor software RAID devices. It is used in modern Linux distributions in place of older software RAID utilities such as raidtools2 or raidtools.

The Linux booting process involves multiple stages and is in many ways similar to the BSD and other Unix-style boot processes, from which it derives. Although the Linux booting process depends very much on the computer architecture, those architectures share similar stages and software components, including system startup, bootloader execution, loading and startup of a Linux kernel image, and execution of various startup scripts and daemons. Those are grouped into 4 steps: system startup, bootloader stage, kernel stage, and init process. When a Linux system is powered up or reset, its processor will execute a specific firmware/program for system initialization, such as Power-on self-test, invoking the reset vector to start a program at a known address in flash/ROM, then load the bootloader into RAM for later execution. In personal computer (PC), not only limited to Linux-distro PC, this firmware/program is called BIOS, which is stored in the mainboard. In embedded Linux system, this firmware/program is called boot ROM. After being loaded into RAM, bootloader will execute to load the second-stage bootloader. The second-stage bootloader will load the kernel image into memory, decompress and initialize it then pass control to this kernel image. Second-stage bootloader also performs several operation on the system such as system hardware check, mounting the root device, loading the necessary kernel modules, etc. Finally, the first user-space process starts, and other high-level system initializations are performed.

Btrfs is a computer storage format that combines a file system based on the copy-on-write (COW) principle with a logical volume manager, developed together. It was founded by Chris Mason in 2007 for use in Linux, and since November 2013, the file system's on-disk format has been declared stable in the Linux kernel.

The NOOP scheduler is the simplest I/O scheduler for the Linux kernel. This scheduler was developed by Jens Axboe.

Ceph is a free and open-source software-defined storage platform that provides object storage, block storage, and file storage built on a common distributed cluster foundation. Ceph provides completely distributed operation without a single point of failure and scalability to the exabyte level, and is freely available. Since version 12 (Luminous), Ceph does not rely on any other conventional filesystem and directly manages HDDs and SSDs with its own storage backend BlueStore and can expose a POSIX filesystem.

oVirt is a free, open-source virtualization management platform. It was founded by Red Hat as a community project on which Red Hat Virtualization is based. It allows centralized management of virtual machines, compute, storage and networking resources, from an easy-to-use web-based front-end with platform independent access. KVM on x86-64, PowerPC64 and s390x architecture are the only hypervisors supported, but there is an ongoing effort to support ARM architecture in a future releases.

In computing, Linux-IO (LIO) Target is an open-source implementation of the SCSI target that has become the standard one included in the Linux kernel. Internally, LIO does not initiate sessions, but instead provides one or more Logical Unit Numbers (LUNs), waits for SCSI commands from a SCSI initiator, and performs required input/output data transfers. LIO supports common storage fabrics, including FCoE, Fibre Channel, IEEE 1394, iSCSI, iSCSI Extensions for RDMA (iSER), SCSI RDMA Protocol (SRP) and USB. It is included in some Linux distributions; native support for LIO in QEMU/KVM, libvirt, and OpenStack makes LIO also a storage option for cloud deployments.

Dracut is a set of tools that provide enhanced functionality for automating the Linux boot process. The tool named dracut is used to create a Linux boot image (initramfs) by copying tools and files from an installed system and combining it with the Dracut framework, which is usually found in /usr/lib/dracut/modules.d.

Stratis is a user-space configuration daemon that configures and monitors existing components from Linux's underlying storage components of logical volume management (LVM) and XFS filesystem via D-Bus.

References

↑ Goggin, Edward; Kergon, Alasdair; Varoqui, Christophe; Olien, David. "Linux Multipathing" (PDF).
↑ Varoqui, Christophe. "The Linux multipath implementation".
↑ Oberg, Michael. "Exploration of Parallel Storage Architectures for a Blue Gene/L on the TeraGrid" (PDF). Archived from the original (PDF) on 2013-12-02. Retrieved 2012-02-13.
↑ van Vugt, Sander (8 March 2010). A Practical Guide to XEN High Availability. Books4Brains. ISBN 9789072389084.
1 2 3 4 5 SUSE. "Storage Administration Guide, SUSE Linux Enterprise Server 11 SP1" (PDF). SLES11 Documentation, pg. 49.
↑ "Multipath support in the device mapper [LWN.net]".
↑ Linux 2.6.12 kernelnewbies.org
↑ RedHat. "Using Device-Mapper Multipath". Archived from the original on 2011-12-29. Retrieved 2012-01-30.
↑ Gupta, Meeta (2002). Storage Area Network Fundamentals. Indianapolis, IN: Cisco Press. p. 81. ISBN 1-58705-065-X.
↑ Anderson, Michael. "SCSI Mid-Level Multipath" (PDF).
↑ "Storage Administration Guide, SLES11 Documentation" (PDF). SUSE. p. 73.
1 2 Centos. "Overview of DM-Multipath". Using Device-Mapper Multipath. Archived from the original on 2012-03-05. Retrieved 2012-01-30.

External links

multipath-tools, homepage of the upstream project used to drive the Device Mapper multipathing driver.

This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.

[1] Goggin, Edward; Kergon, Alasdair; Varoqui, Christophe; Olien, David. "Linux Multipathing" (PDF).

[2] Varoqui, Christophe. "The Linux multipath implementation".

[3] Oberg, Michael. "Exploration of Parallel Storage Architectures for a Blue Gene/L on the TeraGrid" (PDF). Archived from the original (PDF) on 2013-12-02. Retrieved 2012-02-13.

[4] van Vugt, Sander (8 March 2010). A Practical Guide to XEN High Availability. Books4Brains. ISBN 9789072389084.

[SUSE-5] 1 2 3 4 5 SUSE. "Storage Administration Guide, SUSE Linux Enterprise Server 11 SP1" (PDF). SLES11 Documentation, pg. 49.

[6] "Multipath support in the device mapper [LWN.net]".

[7] Linux 2.6.12 kernelnewbies.org

[8] RedHat. "Using Device-Mapper Multipath". Archived from the original on 2011-12-29. Retrieved 2012-01-30.

[9] Gupta, Meeta (2002). Storage Area Network Fundamentals. Indianapolis, IN: Cisco Press. p. 81. ISBN 1-58705-065-X.

[10] Anderson, Michael. "SCSI Mid-Level Multipath" (PDF).

[11] "Storage Administration Guide, SLES11 Documentation" (PDF). SUSE. p. 73.

[Centos-12] 1 2 Centos. "Overview of DM-Multipath". Using Device-Mapper Multipath. Archived from the original on 2012-03-05. Retrieved 2012-01-30.

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]