Data scrubbing

Last updated

Data scrubbing is an error correction technique that uses a background task to periodically inspect main memory or storage for errors, then corrects detected errors using redundant data in the form of different checksums or copies of data. Data scrubbing reduces the likelihood that single correctable errors will accumulate, leading to reduced risks of uncorrectable errors.

Contents

Data integrity is a high-priority concern in writing, reading, storage, transmission, or processing of the computer data in computer operating systems and in computer storage and data transmission systems. However, only a few of the currently existing and used file systems provide sufficient protection against data corruption. [1] [2] [3]

To address this issue, data scrubbing provides routine checks of all inconsistencies in data and, in general, prevention of hardware or software failure. This "scrubbing" feature occurs commonly in memory, disk arrays, file systems, or FPGAs as a mechanism of error detection and correction. [4] [5] [6]

RAID

With data scrubbing, a RAID controller may periodically read all hard disk drives in a RAID array and check for defective blocks before applications might actually access them. This reduces the probability of silent data corruption and data loss due to bit-level errors. [7]

In Dell PowerEdge RAID environments, a feature called "patrol read" can perform data scrubbing and preventive maintenance. [8]

In OpenBSD, the bioctl(8) utility allows the system administrator to control these patrol reads through the BIOCPATROL ioctl on the /dev/bio pseudo-device; as of 2019, this functionality is supported in some device drivers for LSI Logic and Dell controllers — this includes mfi(4) since OpenBSD 5.8 (2015) and mfii(4) since OpenBSD 6.4 (2018). [9] [10]

In FreeBSD and DragonFly BSD, patrol can be controlled through a RAID controller-specific utility mfiutil(8) since FreeBSD 8.0 (2009) and 7.3 (2010). [11] The implementation from FreeBSD was used by the OpenBSD developers for adding patrol support to their generic bio(4) framework and the bioctl utility, without a need for a separate controller-specific utility.

In NetBSD in 2008, the bio(4) framework from OpenBSD was extended to feature support for consistency checks, which was implemented for /dev/bio pseudo-device under BIOCSETSTATE ioctl command, with the options being start and stop (BIOC_SSCHECKSTART_VOL and BIOC_SSCHECKSTOP_VOL, respectively); this is supported only by a single driver as of 2019 — arcmsr(4). [12]

Linux MD RAID, as a software RAID implementation, makes data consistency checks available and provides automated repairing of detected data inconsistencies. Such procedures are usually performed by setting up a weekly cron job. Maintenance is performed by issuing operations check, repair, or idle to each of the examined MD devices. Statuses of all performed operations, as well as general RAID statuses, are always available. [13] [14] [15]

File systems

Btrfs

As a copy-on-write (CoW) file system for Linux, Btrfs provides fault isolation, corruption detection and correction, and file-system scrubbing. If the file system detects a checksum mismatch while reading a block, it first tries to obtain (or create) a good copy of this block from another device – if its internal mirroring or RAID techniques are in use. [16]

Btrfs can initiate an online check of the entire file system by triggering a file system scrub job that is performed in the background. The scrub job scans the entire file system for integrity and automatically attempts to report and repair any bad blocks it finds along the way. [17] [18]

ZFS

The features of ZFS, which is a combined file system and logical volume manager, include the verification against data corruption modes, continuous integrity checking, and automatic repair. Sun Microsystems designed ZFS from the ground up with a focus on data integrity and to protect the data on disks against issues such as disk firmware bugs and ghost writes.[ failed verification ] [19]

ZFS provides a repair utility called scrub that examines and repairs silent data corruption caused by data rot and other problems.

Memory

Due to the high integration density of contemporary computer memory chips, the individual memory cell structures became small enough to be vulnerable to cosmic rays and/or alpha particle emission. The errors caused by these phenomena are called soft errors. This can be a problem for DRAM- and SRAM-based memories.

Memory scrubbing does error-detection and correction of bit errors in computer RAM by using ECC memory, other copies of the data, or other error-correction codes.

FPGA

Scrubbing is a technique used to reprogram an FPGA. It can be used periodically to avoid the accumulation of errors without the need to find one in the configuration bitstream, thus simplifying the design.

Numerous approaches can be taken with respect to scrubbing, from simply reprogramming the FPGA to partial reconfiguration. The simplest method of scrubbing is to completely reprogram the FPGA at some periodic rate (typically 1/10 the calculated upset rate). However, the FPGA is not operational during that reprogram time, on the order of micro to milliseconds. For situations that cannot tolerate that type of interruption, partial reconfiguration is available. This technique allows the FPGA to be reprogrammed while still operational. [20]

See also

Related Research Articles

Data integrity is the maintenance of, and the assurance of, data accuracy and consistency over its entire life-cycle and is a critical aspect to the design, implementation, and usage of any system that stores, processes, or retrieves data. The term is broad in scope and may have widely different meanings depending on the specific context – even under the same general umbrella of computing. It is at times used as a proxy term for data quality, while data validation is a prerequisite for data integrity. Data integrity is the opposite of data corruption. The overall intent of any data integrity technique is the same: ensure data is recorded exactly as intended. Moreover, upon later retrieval, ensure the data is the same as when it was originally recorded. In short, data integrity aims to prevent unintentional changes to information. Data integrity is not to be confused with data security, the discipline of protecting data from unauthorized parties.

RAID is a data storage virtualization technology that combines multiple physical disk drive components into one or more logical units for the purposes of data redundancy, performance improvement, or both. This was in contrast to the previous concept of highly reliable mainframe disk drives referred to as "single large expensive disk" (SLED).

fsck System tool for checking the consistency of a file system

The system utility fsck is a tool for checking the consistency of a file system in Unix and Unix-like operating systems, such as Linux, macOS, and FreeBSD. A similar command, CHKDSK, exists in Microsoft Windows and its ancestor, MS-DOS.

Data degradation is the gradual corruption of computer data due to an accumulation of non-critical failures in a data storage device. The phenomenon is also known as data decay, data rot or bit rot.

In computer storage, logical volume management or LVM provides a method of allocating space on mass-storage devices that is more flexible than conventional partitioning schemes to store volumes. In particular, a volume manager can concatenate, stripe together or otherwise combine partitions into larger virtual partitions that administrators can re-size or move, potentially without interrupting system use.

Data striping

In computer data storage, data striping is the technique of segmenting logically sequential data, such as a file, so that consecutive segments are stored on different physical storage devices.

A disk array controller is a device that manages the physical disk drives and presents them to the computer as logical units. It almost always implements hardware RAID, thus it is sometimes referred to as RAID controller. It also often provides additional disk cache.

Data corruption Errors in computer data that introduce unintended changes to the original data

Data corruption refers to errors in computer data that occur during writing, reading, storage, transmission, or processing, which introduce unintended changes to the original data. Computer, transmission, and storage systems use a number of measures to provide end-to-end data integrity, or lack of errors.

The device mapper is a framework provided by the Linux kernel for mapping physical block devices onto higher-level virtual block devices. It forms the foundation of the logical volume manager (LVM), software RAIDs and dm-crypt disk encryption, and offers additional features such as file system snapshots.

Disk mirroring

In data storage, disk mirroring is the replication of logical disk volumes onto separate physical hard disks in real time to ensure continuous availability. It is most commonly used in RAID 1. A mirrored volume is a complete logical representation of separate volume copies.

ECC memory Self-correcting computer data storage

Error correction code memory is a type of computer data storage that uses an error correction code (ECC) to detect and correct n-bit data corruption which occurs in memory. ECC memory is used in most computers where data corruption cannot be tolerated, like industrial control applications, critical databases, and infrastructural memory caches.

In computer main memory, auxiliary storage and computer buses, data redundancy is the existence of data that is additional to the actual data and permits correction of errors in stored or transmitted data. The additional data can simply be a complete copy of the actual data, or only select pieces of data that allow detection of errors and reconstruction of lost or damaged data up to a certain level.

Vinum is a logical volume manager, also called software RAID, allowing implementations of the RAID-0, RAID-1 and RAID-5 models, both individually and in combination. The original Vinum was part of the base distribution of the FreeBSD operating system since 3.0, and also NetBSD between 2003-10-10 and 2006-02-25, as well as descendants of FreeBSD, including DragonFly BSD; in more recent versions of FreeBSD, it has been replaced with gvinum, which was first introduced around FreeBSD 6. Vinum source code is maintained in the FreeBSD and DragonFly source trees. Vinum supports RAID levels 0, 1, 5, and JBOD. Vinum was inspired by Veritas Volume Manager.

In computing, error recovery control (ERC) is a feature of hard disks which allow a system administrator to configure the amount of time a drive's firmware is allowed to spend recovering from a read or write error. Limiting the recovery time allows for improved error handling in hardware or software RAID environments. In some cases, there is a conflict as to whether error handling should be undertaken by the hard drive or by the RAID implementation, which leads to drives being marked as unusable and significant performance degradation, when this could otherwise have been avoided.

Although all RAID implementations differ from the specification to some extent, some companies and open-source projects have developed non-standard RAID implementations that differ substantially from the standard. Additionally, there are non-RAID drive architectures, providing configurations of multiple hard drives not referred to by RAID acronyms.

Btrfs is a computer storage format that combines a file system based on the copy-on-write (COW) principle with a logical volume manager, developed together. It was initially designed at Oracle Corporation in 2007 for use in Linux, and since November 2013, the file system's on-disk format has been declared stable in the Linux kernel. According to Oracle, Btrfs "is not a true acronym".

HAMMER is a high-availability 64-bit file system developed by Matthew Dillon for DragonFly BSD using B+ trees. Its major features include infinite NFS-exportable snapshots, master-multislave operation, configurable history retention, fsckless-mount, and checksums to deal with data corruption. HAMMER also supports data block deduplication, meaning that identical data blocks will be stored only once on a file system. A successor, HAMMER2, was announced in 2011 and became the default in Dragonfly 5.2.

Resilient File System (ReFS), codenamed "Protogon", is a Microsoft proprietary file system introduced with Windows Server 2012 with the intent of becoming the "next generation" file system after NTFS.

ZFS File system

ZFS combines a file system with a volume manager. It began as part of the Sun Microsystems Solaris operating system in 2001. Large parts of Solaris – including ZFS – were published under an open source license as OpenSolaris for around 5 years from 2005, before being placed under a closed source license when Oracle Corporation acquired Sun in 2009/2010. During 2005 to 2010, the open source version of ZFS was ported to Linux, Mac OS X and FreeBSD. In 2010, the illumos project forked a recent version of OpenSolaris, to continue its development as an open source project, including ZFS. In 2013, OpenZFS was founded to coordinate the development of open source ZFS. OpenZFS maintains and manages the core ZFS code, while organizations using ZFS maintain the specific code and validation processes required for ZFS to integrate within their systems. OpenZFS is widely used in Unix-like systems.

The bio(4) pseudo-device driver and the bioctl(8) utility implement a generic RAID volume management interface in OpenBSD and NetBSD. The idea behind this software is similar to ifconfig, where a single utility from the operating system can be used to control any RAID controller using a generic interface, instead of having to rely on many proprietary and custom RAID management utilities specific for each given hardware RAID manufacturer. Features include monitoring of the health status of the arrays, controlling identification through blinking the LEDs and managing of sound alarms, and specifying hot spare disks. Additionally, the softraid configuration in OpenBSD is delegated to bioctl as well; whereas the initial creation of volumes and configuration of hardware RAID is left to card BIOS as non-essential after the operating system has already been booted. Interfacing between the kernel and userland is performed through the ioctl system call through the /dev/bio pseudo-device.

References

  1. ↑ "Checking ZFS File System Integrity". Oracle Solaris ZFS Administration Guide. Oracle. Archived from the original on 31 January 2013. Retrieved 25 November 2012.
  2. ↑ Vijayan Prabhakaran (2006). "IRON FILE SYSTEMS" (PDF). Doctor of Philosophy in Computer Sciences. University of Wisconsin-Madison. Archived (PDF) from the original on 29 April 2011. Retrieved 9 June 2012.
  3. ↑ Andrew Krioukov; Lakshmi N. Bairavasundaram; Garth R. Goodson; Kiran Srinivasan; Randy Thelen; Andrea C. Arpaci-Dusseau; Remzi H. Arpaci-Dusseau (2008). "Parity Lost and Parity Regained". In Mary Baker; Erik Riedel (eds.). FAST'08: Proceedings of the 6th USENIX Conference on File and Storage Technologies. Archived from the original on 2020-08-26. Retrieved 2021-05-28.
  4. ↑ "An Analysis of Data Corruption in the Storage Stack" (PDF). Archived (PDF) from the original on 2010-06-15. Retrieved 2012-11-26.
  5. ↑ "Impact of Disk Corruption on Open-Source DBMS" (PDF). Archived (PDF) from the original on 2010-06-15. Retrieved 2012-11-26.
  6. ↑ "Baarf.com". Baarf.com. Archived from the original on November 5, 2011. Retrieved November 4, 2011.
  7. ↑ Ulf Troppens, Wolfgang Mueller-Friedt, Rainer Erkens, Rainer Wolafka, Nils Haustein. Storage Networks Explained: Basics and Application of Fibre Channel SAN, NAS, ISCSI, InfiniBand and FCoE. John Wiley and Sons, 2009. p.39
  8. ↑ "About PERC 6 and CERC 6i Controllers". Archived from the original on 2013-05-29. Retrieved 2013-06-20. The Patrol Read feature is designed as a preventative measure to ensure physical disk health and data integrity. Patrol Read scans for and resolves potential problems on configured physical disks.
  9. ↑ "/sys/dev/ic/mfi.c — LSI Logic & Dell MegaRAID SAS RAID controller". BSD Cross Reference. OpenBSD.
  10. ↑ "/sys/dev/pci/mfii.c — LSI Logic MegaRAID SAS Fusion RAID controller". BSD Cross Reference. OpenBSD.
  11. ↑ "mfiutil — Utility for managing LSI MegaRAID SAS controllers". BSD Cross Reference. FreeBSD.
  12. ↑ "sys/dev/pci/arcmsr.c — Areca Technology Corporation SATA/SAS RAID controller". BSD Cross Reference. NetBSD.
  13. ↑ "RAID Administration". kernel.org. Archived from the original on 2013-09-21. Retrieved 2013-09-20.
  14. ↑ "Software RAID and LVM: Data scrubbing". archlinux.org. Archived from the original on 2013-09-21. Retrieved 2013-09-20.
  15. ↑ "Linux kernel documentation: Documentation/md.txt". kernel.org. Archived from the original on 2013-09-21. Retrieved 2013-09-20.
  16. ↑ "btrfs Wiki: Features". The btrfs Project. Archived from the original on 2012-04-25. Retrieved 2013-09-20.
  17. ↑ Bierman, Margaret; Grimmer, Lenz (August 2012). "How I Use the Advanced Capabilities of Btrfs". Archived from the original on 2014-01-02. Retrieved 2013-09-20.
  18. ↑ Coekaerts, Wim (2011-09-28). "btrfs scrub – go fix corruptions with mirror copies please!". Archived from the original on 2013-09-21. Retrieved 2013-09-20.
  19. ↑ Bonwick, Jeff (2005-12-08). "ZFS End-to-End Data Integrity". Archived from the original on 2017-05-06. Retrieved 2013-09-19.
  20. ↑ "Xcell journal, issue 50" (PDF). FPGAs on Mars. Xilinx. 2004. p. 9. Archived (PDF) from the original on 2019-08-30. Retrieved 2013-10-16.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.