ReFS

Last updated

ReFS
Developer(s) Microsoft
Full nameResilient File System
Introduced1 August 2012;11 years ago (2012-08-01) [1] with Windows Server 2012
Structures
Directory contents B+ tree [2]
Limits
Max volume size35 petabytes [3]
Max file size35 petabytes [3]
Features
AttributesYes
Transparent
compression
No
Data deduplication Yes, since v3.2 debuting in 2016 v1709 [4]
Copy-on-write Yes
Other
Supported
operating systems
Website learn.microsoft.com/en-US/windows-server/storage/refs/refs-overview

Resilient File System (ReFS), [6] codenamed "Protogon", [7] is a Microsoft proprietary file system introduced with Windows Server 2012 with the intent of becoming the "next generation" file system after NTFS.

Contents

ReFS was designed to overcome problems that had become significant over the years since NTFS was conceived, which are related to how data storage requirements have changed. These requirements arose from two major changes in storage systems and usage – the size of storage in use (large or massive arrays of multi-terabyte drives now being fairly common), and the need for continual reliability. As a result, the file system needs to be self-repairing (to prevent disk checking from being impractically slow or disruptive), along with abstraction or virtualization between physical disks and logical volumes.

The key design advantages of ReFS include automatic integrity checking and data scrubbing, elimination of the need for running chkdsk, protection against data degradation, built-in handling of hard disk drive failure and redundancy, integration of RAID functionality, a switch to copy/allocate on write for data and metadata updates, handling of very long paths and filenames, and storage virtualization and pooling, including almost arbitrarily sized logical volumes (unrelated to the physical sizes of the used drives).

Comparison with NTFS

Major new features

Improved reliability for on-disk structures

ReFS uses B+ trees for all on-disk structures, including all metadata and file data. [2] [8] Metadata and file data are organized into tables similar to a relational database. The file size, number of files in a folder, total volume size, and number of folders in a volume are limited by 64-bit numbers; as a result, ReFS supports a maximum file size of 35 petabytes, and a maximum volume size of 35 petabytes. [3]

Built-in resilience

ReFS employs an allocation-on-write update strategy for metadata, [2] which allocates new chunks for every update transaction and uses large IO batches. All ReFS metadata have 64-bit checksums which are stored independently. The file data can have an optional checksum in a separate "integrity stream", in which case the file update strategy also implements allocation-on-write for file data; this is controlled by a new "integrity" attribute applicable to both files and directories. If file data or metadata become corrupt, the file can be deleted without taking the whole volume offline for maintenance, and then be restored from the backup. As a result of built-in resiliency, administrators do not need to periodically run error-checking tools such as CHKDSK when using ReFS.

Compatibility with existing APIs and technologies

ReFS supports only a subset of NTFS features, and only supports Win32 APIs that are "widely adopted". It does not require new system APIs, and most file system filters continue to work with ReFS volumes. [2] ReFS supports many existing Windows and NTFS features such as BitLocker encryption, Access Control Lists, USN Journal, change notifications, [9] symbolic links, junction points, mount points, reparse points, volume snapshots, file IDs, and oplock. ReFS seamlessly integrates with Storage Spaces, [2] a storage virtualization layer that allows data mirroring and striping, as well as sharing storage pools between machines. [10] ReFS resiliency features enhance the mirroring feature provided by Storage Spaces and can detect whether any mirrored copies of files become corrupt using a data scrubbing process, [8] which periodically reads all mirror copies and verifies their checksums, then replaces bad copies with good ones.

Microsoft Windows and Windows Server include ReFSUtil, a command-line utility that can be used to diagnose heavily damaged ReFS volumes, identify remaining files, and copy those files to another volume. [11]

Removed features

Some NTFS features are not implemented in ReFS. These include object IDs, 8.3 filename, NTFS compression, Encrypting File System (EFS), transactional NTFS, extended attributes, and disk quotas. [7] [2] [12] Dynamic disks with mirrored or striped volumes are replaced with mirrored or striped storage pools provided by Storage Spaces; however, automated error-correction is only supported on mirrored spaces. Data deduplication was missing in early versions of ReFS. [2] It was implemented in v3.2, debuting in Windows Server v1709. [4]

Support for alternate data streams and hard links was initially not implemented in ReFS. In Windows 8.1 64-bit and Server 2012 R2, the file system reacquired support for alternate data streams, with lengths of up to 128K, and automatic correction of corruption when integrity streams are used on parity spaces. [13] ReFS had initially been unsuitable for Microsoft SQL Server instance allocation due to the absence of alternate data streams. [14] Hard links were introduced with preview versions of Windows Server 2022 but are not yet available in Windows 11.

Implementations

ReFS was initially added to Windows Server 2012 only, with the aim of gradual migration to consumer systems in future versions; this was achieved as of Windows 8.1. [3] The initial versions removed some NTFS features, such as disk quotas, alternate data streams, and extended attributes. Some of these were re-implemented in later versions of ReFS.

In early versions (2012–2013), ReFS was similar to or slightly faster than NTFS in most tests, [15] but far slower when full integrity checking was enabled, a result attributed to the relative newness of ReFS. [16] [ self-published source ] [17] [ self-published source ]

The ability to create ReFS volumes was removed in Windows 10's 2017 Fall Creators Update for all editions except Enterprise and Pro for Workstations. [5] [ why? ]

Starting with Windows Server 2022 and Windows 11 21H2, the boot environment natively supports ReFS, allowing the system to be installed and run in a special way on a volume formatted with ReFS v3. If it is a volume formatted with ReFS v1, it cannot be booted with ReFS. [18]

Starting with Windows 11 build 22621.2338, ReFS is re-introduced via a Dev Drive feature; allowing fixed storage drives and VHDs to be formatted as ReFS, with special file and Microsoft Defender policies added during use.

The cluster size of a ReFS volume is either 4 KB or 64 KB. [19]

History

Server 2016

At the Storage Developer Conference 2015, a Microsoft developer presented enhancements of ReFS expected to be released with Windows Server 2016 and included in Technical Preview 4, titled "ReFS v2". [20] It highlighted that ReFS now included capabilities for very high speed moving, reordering, and cloning of blocks between files [21] (which can be done for all blocks of a file). This is particularly needed for virtualization, and is stated to allow fast provisioning, diff merging, and tiering. Other enhancements cover the redo log (for synchronous disk writes), parallelization, efficient tracking of uninitialized sparse data and files, and efficient 4k I/O. [20]

Server 2022

Windows Server 2022 (using ReFS version 3.7) supports file-level snapshots. [3]

Windows Insider Preview 22H2 and 23H2 (builds 226** and 25***) support ReFS volume compression using LZ4 and zstd algorithms. [22]

Versions

ReFS has some different versions, with various degrees of compatibility between operating system versions. Aside from development versions of the filesystem, usually, later operating system versions can mount filesystems created with earlier OS versions (backwards compatibility). Some features may not be compatible with the feature set of the OS. The version, cluster size and other features of the filesystem can be queried with the command fsutil fsinfo refsinfo volumename.

Known problems

Issues identified or suggested for ReFS, when running on Storage Spaces, include:

Comparison with other file systems

Like ReFS, ZFS and Btrfs are designed to integrate data protection, snapshots, and background error correction.

In 2012, Phoronix wrote an analysis [25] of ReFS vs Btrfs. At the time, their features were similar, with both supporting checksums, RAID-like use of multiple disks, and error correction. However, ReFS lacked copy-on-write snapshots and compression, both found in Btrfs and ZFS.

In 2014, BetaNews wrote a review of ReFS and assessed its readiness for production use. [26] The review concluded that ReFS had at least some advantages over two of its main file system competitors.

Reverse engineering and internals

As of November 2019, Microsoft has not published any specifications for ReFS, nor have any working open-source drivers been made. A third-party open-source project to document ReFS is on GitHub. [28] [29]

Paragon Software Group provides a closed-source driver for Windows and Linux.

See also

Related Research Articles

New Technology File System (NTFS) is a proprietary journaling file system developed by Microsoft. Starting with Windows NT 3.1, it is the default file system of the Windows NT family. It superseded File Allocation Table (FAT) as the preferred filesystem on Windows and is supported in Linux and BSD as well. NTFS reading and writing support is provided using a free and open-source kernel implementation known as NTFS3 in Linux and the NTFS-3G driver in BSD. By using the convert command, Windows can convert FAT32/16/12 into NTFS without the need to rewrite all files. NTFS uses several files typically hidden from the user to store metadata about other files stored on the drive which can help improve speed and performance when reading data. Unlike FAT and High Performance File System (HPFS), NTFS supports access control lists (ACLs), filesystem encryption, transparent compression, sparse files and file system journaling. NTFS also supports shadow copy to allow backups of a system while it is running, but the functionality of the shadow copies varies between different versions of Windows.

Universal Disk Format (UDF) is an open, vendor-neutral file system for computer data storage for a broad range of media. In practice, it has been most widely used for DVDs and newer optical disc formats, supplanting ISO 9660. Due to its design, it is very well suited to incremental updates on both write-once and re-writable optical media. UDF was developed and maintained by the Optical Storage Technology Association (OSTA).

<span class="mw-page-title-main">Disk partitioning</span> Creation of separate accessible storage areas on a secondary computer storage device

Disk partitioning or disk slicing is the creation of one or more regions on secondary storage, so that each region can be managed separately. These regions are called partitions. It is typically the first step of preparing a newly installed disk, before any file system is created. The disk stores the information about the partitions' locations and sizes in an area known as the partition table that the operating system reads before any other part of the disk. Each partition then appears to the operating system as a distinct "logical" disk that uses part of the actual disk. System administrators use a program called a partition editor to create, resize, delete, and manipulate the partitions. Partitioning allows the use of different filesystems to be installed for different kinds of files. Separating user data from system data can prevent the system partition from becoming full and rendering the system unusable. Partitioning can also make backing up easier. A disadvantage is that it can be difficult to properly size partitions, resulting in having one partition with too much free space and another nearly totally allocated.

HPFS is a file system created specifically for the OS/2 operating system to improve upon the limitations of the FAT file system. It was written by Gordon Letwin and others at Microsoft and added to OS/2 version 1.2, at that time still a joint undertaking of Microsoft and IBM, and released in 1988.

Disk formatting is the process of preparing a data storage device such as a hard disk drive, solid-state drive, floppy disk, memory card or USB flash drive for initial use. In some cases, the formatting operation may also create one or more new file systems. The first part of the formatting process that performs basic medium preparation is often referred to as "low-level formatting". Partitioning is the common term for the second part of the process, dividing the device into several sub-devices and, in some cases, writing information to the device allowing an operating system to be booted from it. The third part of the process, usually termed "high-level formatting" most often refers to the process of generating a new file system. In some operating systems all or parts of these three processes can be combined or repeated at different levels and the term "format" is understood to mean an operation in which a new disk medium is fully prepared to store files. Some formatting utilities allow distinguishing between a quick format, which does not erase all existing data and a long option that does erase all existing data.

HFS Plus or HFS+ is a journaling file system developed by Apple Inc. It replaced the Hierarchical File System (HFS) as the primary file system of Apple computers with the 1998 release of Mac OS 8.1. HFS+ continued as the primary Mac OS X file system until it was itself replaced with the Apple File System (APFS), released with macOS High Sierra in 2017. HFS+ is also one of the formats supported by the iPod digital music player.

<span class="mw-page-title-main">File system</span> Format or program for storing files and directories

In computing, a file system or filesystem is a method and data structure that the operating system uses to control how data is stored and retrieved. Without a file system, data placed in a storage medium would be one large body of data with no way to tell where one piece of data stopped and the next began, or where any piece of data was located when it was time to retrieve it. By separating the data into pieces and giving each piece a name, the data are easily isolated and identified. Taking its name from the way a paper-based data management system is named, each group of data is called a "file". The structure and logic rules used to manage the groups of data and their names is called a "file system."

Filesystem in Userspace (FUSE) is a software interface for Unix and Unix-like computer operating systems that lets non-privileged users create their own file systems without editing kernel code. This is achieved by running file system code in user space while the FUSE module provides only a bridge to the actual kernel interfaces.

<span class="mw-page-title-main">Shadow Copy</span> Microsoft technology for storage snapshots

Shadow Copy is a technology included in Microsoft Windows that can create backup copies or snapshots of computer files or volumes, even when they are in use. It is implemented as a Windows service called the Volume Shadow Copy service. A software VSS provider service is also included as part of Windows to be used by Windows applications. Shadow Copy technology requires either the Windows NTFS or ReFS filesystems in order to create and store shadow copies. Shadow Copies can be created on local and external volumes by any Windows component that uses this technology, such as when creating a scheduled Windows Backup or automatic System Restore point.

In computer data storage, a volume or logical drive is a single accessible storage area with a single file system, typically resident on a single partition of a hard disk. Although a volume might be different from a physical disk drive, it can still be accessed with an operating system's logical interface. However, a volume differs from a partition.

The Logical Disk Manager (LDM) is an implementation of a logical volume manager for Microsoft Windows NT, developed by Microsoft and Veritas Software. It was introduced with the Windows 2000 operating system, and is supported in Windows XP, Windows Server 2003, Windows Vista, Windows 7, Windows 8, Windows 10 and Windows 11. The MMC-based Disk Management snap-in hosts the Logical Disk Manager. On Windows 8 and Windows Server 2012, Microsoft deprecated LDM in favor of Storage Spaces.

The following tables compare general and technical information for a number of file systems.

ext4 is a journaling file system for Linux, developed as the successor to ext3.

<span class="mw-page-title-main">Microsoft Drive Optimizer</span> Windows utility which defragments a hard drive

Microsoft Drive Optimizer is a utility in Microsoft Windows designed to increase data access speed by rearranging files stored on a disk to occupy contiguous storage locations, a technique called defragmentation. Defragmenting a disk minimizes head travel, which reduces the time it takes to read files from and write files to the disk. As a result of the decreased read and write times, Microsoft Drive Optimizer decreases system startup times for systems starting from magnetic storage devices such as a hard drive. However, defragmentation is not helpful on storage devices such as solid state drives, USB drives or SD cards that use flash memory to increase speeds, as these drives do not use a head. Defragmentation may decrease lifespan for certain technologies, e.g. solid state drives. Microsoft Drive Optimizer was first officially shipped with Windows XP.

An NTFS reparse point is a type of NTFS file system object. It is available with the NTFS v3.0 found in Windows 2000 or later versions. Reparse points provide a way to extend the NTFS filesystem. A reparse point contains a reparse tag and data that are interpreted by a filesystem filter driver identified by the tag. Microsoft includes several default tags including NTFS symbolic links, directory junction points, volume mount points and Unix domain sockets. Also, reparse points are used as placeholders for files moved by Windows 2000's Remote Storage Hierarchical Storage System. They also can act as hard links, but are not limited to pointing to files on the same volume: they can point to directories on any local volume. The feature is inherited to ReFS.

The NTFS file system defines various ways to redirect files and folders, e.g., to make a file point to another file or its contents without making a copy of it. The object being pointed to is called the target. Such file is called a hard or symbolic link depending on a way it's stored on the filesystem.

Single-instance storage (SIS) is a system's ability to take multiple copies of content and replace them by a single shared copy. It is a means to eliminate data duplication and to increase efficiency. SIS is frequently implemented in file systems, e-mail server software, data backup, and other storage-related computer software. Single-instance storage is a simple variant of data deduplication. While data deduplication may work at a segment or sub-block level, single-instance storage works at the whole-file level and eliminates redundant copies of entire files or e-mail messages.

In computing, data deduplication is a technique for eliminating duplicate copies of repeating data. Successful implementation of the technique can improve storage utilization, which may in turn lower capital expenditure by reducing the overall amount of storage media required to meet storage capacity needs. It can also be applied to network data transfers to reduce the number of bytes that must be sent.

<span class="mw-page-title-main">Windows Server 2012</span> Sixth version of Windows Server, released in 2012

Windows Server 2012, codenamed "Windows Server 8", is the sixth version of the Windows Server operating system by Microsoft, as part of the Windows NT family of operating systems. It is the server version of Windows based on Windows 8 and succeeds Windows Server 2008 R2, which is derived from the Windows 7 codebase, released nearly three years earlier. Two pre-release versions, a developer preview and a beta version, were released during development. The software was officially launched on September 4, 2012, which was the month before the release of Windows 8. It was succeeded by Windows Server 2012 R2 in 2013. Mainstream support for Windows Server 2012 ended on October 9, 2018, and extended support ended on October 10, 2023. Windows Server 2012 is eligible for the paid Extended Security Updates (ESU) program, which offers continued security updates until October 13, 2026.

References

  1. Snover, Jeffrey (1 August 2012). "Windows Server 2012 released to manufacturing!". Windows Server Blog. Microsoft Corporation. Archived from the original on 22 December 2015. Retrieved 2 September 2014.
  2. 1 2 3 4 5 6 7 Verma, Surendra (16 January 2012). Sinofsky, Steven (ed.). "Building the next generation file system for Windows: ReFS". Building Windows 8. Microsoft. MSDN blogs. Archived from the original on 31 July 2012. Retrieved 20 January 2013.
  3. 1 2 3 4 5 6 "Resilient File System (ReFS) overview". Microsoft. Archived from the original on 25 September 2020. Retrieved 22 September 2020.
  4. 1 2 Sarkar, Dona (13 July 2017). "Announcing Windows Server Insider Preview Build 16237". Windows Experience Blog. Microsoft. Archived from the original on 23 July 2017. Retrieved 20 July 2017.
  5. 1 2 3 "Windows 10 features and functionality removed in Windows 10". docs.microsoft.com. 30 January 2020. Archived from the original on 18 November 2019. Retrieved 23 February 2020. Creation ability will be available in the following editions only: Windows 10 Enterprise and Windows 10 Pro for Workstations. Creation ability will be removed from all other editions. All other editions will have Read and Write ability.
  6. Lucas, Martin (1 January 2013). "Windows Server 2012: Does ReFS replace NTFS? When should I use it?". Ask Premier Field Engineering (PFE) Platforms. Microsoft. TechNet blogs. Archived from the original on 2 June 2013. Retrieved 20 January 2013.
  7. 1 2 Foley, Mary Jo (16 January 2012). "Microsoft goes public with plans for its new Windows 8 file system". ZDNet . CBS Interactive. Archived from the original on 21 May 2013. Retrieved 21 January 2013.
  8. 1 2 Meyer, David (17 January 2012). "Microsoft takes wraps off new Windows file system". ZDNet . CBS Interactive. Archived from the original on 21 May 2013. Retrieved 20 January 2013.
  9. "ReadDirectoryChangesW function". MSDN Library . Microsoft. 8 November 2012. Archived from the original on 4 May 2013. Retrieved 21 January 2013.
  10. Nagar, Rajeev (5 January 2012). Sinofsky, Steven (ed.). "Virtualizing storage for scale, resiliency, and efficiency". Building Windows 8. Microsoft. MSDN blogs. Archived from the original on 6 January 2012. Retrieved 18 January 2013.
  11. "ReFSUtil | Microsoft Docs". Archived from the original on 27 August 2021. Retrieved 13 July 2021.
  12. Windows Server 2012: Does ReFS replace NTFS? When should I use it? Archived 2 June 2013 at the Wayback Machine – Martin Lucas, TechNet
  13. "Resilient File System Overview". Microsoft Docs . Microsoft. 29 February 2012. Retrieved 24 August 2022.
  14. Cherry, Denny (21 August 2012). "SQL Server and Windows Server 2012's ReFS File System | Troubleshooting SQL Server Storage Problems". SQL Server Pro magazine. Penton Media. Archived from the original on 14 July 2018. Retrieved 9 May 2020.
  15. "ReFS vs NTFS Benchmark – First Ever – Exclusive Technology News". FirstEver.eu. Archived from the original on 8 July 2014.
  16. "Relative Performance of Parity .vs. Mirror in Storage Spaces". Baris Eris's Blog. 22 October 2012. Archived from the original on 22 June 2014. Retrieved 17 June 2014.
  17. Eckert, Jason (25 February 2014). "Microsoft ReFS vs Oracle ZFS – Fight!". Archived from the original on 13 January 2017. Retrieved 12 January 2017.
  18. Windows 11 on ReFS, archived from the original on 10 July 2022, retrieved 10 July 2022
  19. "Storage at Microsoft: Cluster size recommendations for ReFS and NTFS". 13 January 2017. Retrieved 22 August 2022.
  20. 1 2 J. R. Tipton. "ReFS v2 – Cloning, projecting, and moving data" (PDF). Retrieved 20 May 2023.
  21. "Block Cloning". Microsoft Docs. Microsoft. 7 January 2021. Retrieved 20 May 2023.
  22. REFS_COMPRESSION_FORMATS - NTIFS.h Windows SDK Insider Preview Archived 14 May 2022 at the Wayback Machine build 22621
  23. 1 2 Pakhomova, Elena (8 August 2012). "Data Recovery Tales: When Windows Storage Spaces Go Bad". Archived from the original on 30 July 2016. Retrieved 17 June 2014.
  24. "What I learned after 30 minutes with ReFS". Ctrl blog. 18 May 2017. Archived from the original on 19 May 2017. Retrieved 30 May 2017.
  25. "Microsoft's ReFS File-System: Competitor To Btrfs?". phoronix.com. Archived from the original on 8 September 2016. Retrieved 14 September 2016.
  26. "Windows Storage Spaces and ReFS: Is it time to ditch RAID for good?". BetaNews. 15 January 2014. Archived from the original on 31 January 2014. Retrieved 17 June 2014.
  27. "How To Size Main Memory for ZFS Deduplication". www.oracle.com. Retrieved 24 August 2022.
  28. "libyal/libfsrefs". GitHub. 18 October 2019. Archived from the original on 9 February 2019. Retrieved 11 November 2019.
  29. "ReFS documentation project - PDF document of the ReFS filing system (direct link)". GitHub . Archived from the original on 10 July 2022. Retrieved 28 June 2020.