NILFS

Last updated
NILFS
Developer(s) Nippon Telegraph and Telephone Cyber Space Laboratories
Full nameNew Implementation of a Log-structured File System
Introduced2005;19 years ago (2005) with Linux kernel 2.6.13
Structures
File allocation B-tree
Limits
Max volume size8 EiB
Max file size8 EiB
Max filename length255 bytes
Features
Dates recordedatime (planned)
Date resolution64-bit on-disk timestamps
AttributesPOSIX, with plans for extended attributes
File system
permissions
POSIX, with plans for ACLs [1]
Transparent
compression
No
Transparent
encryption
No
Copy-on-write Yes
Other
Supported
operating systems
Linux kernel, (ReadOnly for NetBSD)

NILFS or NILFS2 (New Implementation of a Log-structured File System) is a log-structured file system implementation for the Linux kernel. It was developed by Nippon Telegraph and Telephone Corporation (NTT) CyberSpace Laboratories and a community from all over the world. NILFS was released under the terms of the GNU General Public License (GPL).

Contents

Design

"NILFS is a log-structured file system, in that the storage medium is treated like a circular buffer and new blocks are always written to the end.[…]Log-structured file systems are often used for flash media since they will naturally perform wear-leveling;[…]NILFS emphasizes snapshots. The log-structured approach is a specific form of copy-on-write behavior, so it naturally lends itself to the creation of file system snapshots. The NILFS developers talk about the creation of "continuous snapshots" which can be used to recover from user-initiated file system problems[…]." [2]

Using a copy-on-write technique known as a log-structured file system, NILFS records all data in a continuous log-like format that is only appended to, never overwritten, an approach that is designed to reduce seek times, as well as minimize the kind of data loss that occurs after a crash with conventional file systems. For example, data loss occurs on ext3 file systems when the system crashes during a write operation. When the system reboots, the journal notes that the write did not complete, and any partial data writes are lost.

Some file systems, like UFS-derived file systems used by the Solaris operating system and BSDs, provide a snapshot feature that prevents[ citation needed ] such data loss, but the snapshot configuration can be lengthy on large file systems. NILFS, in contrast, can "continuously and automatically [save] instantaneous states of the file system without interrupting service", according to NTT Labs. [3]

The "instantaneous states" that NILFS continuously saves can actually be mounted, read-only, at the same time that the actual file system is mounted read-write — a capability useful for data recovery after hardware failures and other system crashes. The "lscp" (list checkpoint) command of an interactive NILFS "inspect" utility is first used to find the checkpoint's address, in this case "2048":

# inspect /dev/sda2 ... nilfs> listcp    1     6 Tue Jul 12 14:55:57 2005 MajorCP|LogiBegin|LogiEnd 2048  2352 Tue Jul 12 14:55:58 2005 MajorCP|LogiEnd ... nilfs> quit

The checkpoint address is then used to mount the checkpoint:

# mount -t nilfs -r -o cp=2048 /dev/sda2 /nilfs-cp # df Filesystem           1K-blocks      Used Available Use% Mounted on /dev/sda2             70332412   8044540  62283776  12% /nilfs /dev/sda2             70332412   8044540  62283776  12% /nilfs-cp

Features

NILFS provides continuous snapshotting. In addition to the versioning capability of the entire file system, users can even restore files mistakenly overwritten or deleted at any recent time. Since NILFS can keep consistency like conventional LFS, it achieves quick recovery after system crashes.

Continuous snapshotting is not provided by most file systems, including those supporting point-in-time snapshotting (e.g. Btrfs)

NILFS creates a number of checkpoints every few seconds or per synchronous write basis (unless there is no change). Users can select significant versions among continuously created checkpoints, and can change them into snapshots which will be preserved until they are changed back to checkpoints.

There is no limit on the number of snapshots until the volume gets full. Each snapshot is mountable as a read-only file system. It is mountable concurrently with a writable mount and other snapshots, and this feature is convenient for making consistent backups during use.

Possible uses of NILFS include versioning, tamper detection, SOX compliance logging, data loss recovery.

The current major version of NILFS is version 2, referred to as NILFS2. NILFS2 implements online garbage collection to reclaim disk space while keeping multiple snapshots.

Other NILFS features include:

Current status

Issues

As of 2023, NILFS lacks a dedicated consistency checking utility (fsck), and thus can't recover from severe errors that cause it to fail to find a valid checkpoint. [4] [5]

Supported features

  • Basic POSIX file system features
  • Snapshots
    • Automatically and continuously taken
    • No limit on the number of snapshots until the volume gets full
    • Mountable as read-only file systems
    • Mountable concurrently with the writable mount (convenient to make consistent backups during use)
    • Quick listing
  • Background Garbage Collection (GC)
    • Can maintain multiple snapshots
    • Selectable GC Policy, which is given by a userland daemon.
  • Quick crash recovery on-mount
  • Read-ahead for meta data files as well as data files
  • Block sizes smaller than page size (e.g. 1KB or 2KB)
  • Online resizing (since Linux-3.x and nilfs-utils 2.1)
  • Related utilities (by contribution of Jiro SEKIBA)

Additional features

Compatibility

NILFS is available in various GNU/Linux distributions like Arch Linux, [8] Debian [9] (since version 5.0), Fedora, Gentoo, [10] Linux Mint, NixOS, Ubuntu (since version 9.10), etc. To use it, users typically need to install the nilfs-utils or nilfs-tools package. A boot-cd with NILFS is also available on PrRescue.

It is also supported by partition-editing applications like GParted. [11]

A separate, BSD licensed implementation, currently with read-only support, is included in NetBSD. [12]

Relative performance

In the January 2015 presentation SD cards and file systems for embedded systems at Linux.conf.au, it was stated: [13]

If you've got a workload that's latency sensitive, you might want to use NILFS. If you've got one that's throughput sensitive, you might want to use F2FS.

Peter Chubb

NILFS2 works much better for the lots of small files case than F2FS or EXT4.

Peter Chubb

License

The NILFS2 file system utilities are made available under the GNU General Public License version 2, with the exception of the lib/nilfs libraries and their header files, which are made available under the GNU Lesser General Public License Version 2.1.

Developers

The Japanese primary authors and major contributors to the nilfs-utils who worked or are working at labs of NTT Corporation are:

Other major contributors are:

See also

Related Research Articles

ext3, or third extended filesystem, is a journaled file system that is commonly used by the Linux kernel. It used to be the default file system for many popular Linux distributions. Stephen Tweedie first revealed that he was working on extending ext2 in Journaling the Linux ext2fs Filesystem in a 1998 paper, and later in a February 1999 kernel mailing list posting. The filesystem was merged with the mainline Linux kernel in November 2001 from 2.4.15 onward. Its main advantage over ext2 is journaling, which improves reliability and eliminates the need to check the file system after an unclean shutdown. Its successor is ext4.

fsck System tool for checking the consistency of a file system

The system utility fsck is a tool for checking the consistency of a file system in Unix and Unix-like operating systems, such as Linux, macOS, and FreeBSD. The equivalent programs on MS-DOS and Microsoft Windows are CHKDSK, SFC, and SCANDISK.

Unionfs is a filesystem service for Linux, FreeBSD and NetBSD which implements a union mount for other file systems. It allows files and directories of separate file systems, known as branches, to be transparently overlaid, forming a single coherent file system. Contents of directories which have the same path within the merged branches will be seen together in a single merged directory, within the new, virtual filesystem.

In information technology, a backup, or data backup is a copy of computer data taken and stored elsewhere so that it may be used to restore the original after a data loss event. The verb form, referring to the process of doing so, is "back up", whereas the noun and adjective form is "backup". Backups can be used to recover data after its loss from data deletion or corruption, or to recover data from an earlier time. Backups provide a simple form of disaster recovery; however not all backup systems are able to reconstitute a computer system or other complex configuration such as a computer cluster, active directory server, or database server.

mv is a Unix command that moves one or more files or directories from one place to another. If both filenames are on the same filesystem, this results in a simple file rename; otherwise the file content is copied to the new location and the old file is removed. Using mv requires the user to have write permission for the directories the file will move between. This is because mv changes the content of both directories involved in the move. When using the mv command on files located on the same filesystem, the file's timestamp is not updated.

Filesystem in Userspace (FUSE) is a software interface for Unix and Unix-like computer operating systems that lets non-privileged users create their own file systems without editing kernel code. This is achieved by running file system code in user space while the FUSE module provides only a bridge to the actual kernel interfaces.

A versioning file system is any computer file system which allows a computer file to exist in several versions at the same time. Thus it is a form of revision control. Most common versioning file systems keep a number of old copies of the file. Some limit the number of changes per minute or per hour to avoid storing large numbers of trivial changes. Others instead take periodic snapshots whose contents can be accessed using methods similar as those for normal file access.

rm (Unix) Unix command utility

rm is a basic command on Unix and Unix-like operating systems used to remove objects such as computer files, directories and symbolic links from file systems and also special files such as device nodes, pipes and sockets, similar to the del command in MS-DOS, OS/2, and Microsoft Windows. The command is also available in the EFI shell.

The Log-Structured File System is an implementation of a log-structured file system, originally developed for BSD. It was removed from FreeBSD and OpenBSD; the NetBSD implementation was nonfunctional until work leading up to the 4.0 release made it viable again as a production file system.

<span class="mw-page-title-main">Snapshot (computer storage)</span> Recorded state of a computer storage system at a particular point in time

In computer systems, a snapshot is the state of a system at a particular point in time. The term was coined as an analogy to that in photography.

Yaffs is a file system designed and written by Charles Manning for the company Aleph One.

Extended file attributes are file system features that enable users to associate computer files with metadata not interpreted by the filesystem, whereas regular attributes have a purpose strictly defined by the filesystem. Unlike forks, which can usually be as large as the maximum file size, extended attributes are usually limited in size to a value significantly smaller than the maximum file size. Typical uses include storing the author of a document, the character encoding of a plain-text document, or a checksum, cryptographic hash or digital certificate, and discretionary access control information.

The following tables compare general and technical information for a number of file systems.

chattr is the command in Linux that allows a user to set certain attributes of a file. lsattr is the command that displays the attributes of a file.

Btrfs is a computer storage format that combines a file system based on the copy-on-write (COW) principle with a logical volume manager, developed together. It was created by Chris Mason in 2007 for use in Linux, and since November 2013, the file system's on-disk format has been declared stable in the Linux kernel.

HAMMER is a high-availability 64-bit file system developed by Matthew Dillon for DragonFly BSD using B+ trees. Its major features include infinite NFS-exportable snapshots, master–multislave operation, configurable history retention, fsckless-mount, and checksums to deal with data corruption. HAMMER also supports data block deduplication, meaning that identical data blocks will be stored only once on a file system. A successor, HAMMER2, was announced in 2011 and became the default in Dragonfly 5.2.

A journaling file system is a file system that keeps track of changes not yet committed to the file system's main part by recording the goal of such changes in a data structure known as a "journal", which is usually a circular log. In the event of a system crash or power failure, such file systems can be brought back online more quickly with a lower likelihood of becoming corrupted.

Bcachefs is a copy-on-write (COW) file system for Linux-based operating systems. Its primary developer, Kent Overstreet, first announced it in 2015, and it was added to the Linux kernel beginning with 6.7. It is intended to compete with the modern features of ZFS or Btrfs, and the speed and performance of ext4 or XFS.

References

  1. "NILFS Current Status".
  2. "BTRFS and NILFS [LWN.net]".
  3. "An article about NILFS" . Retrieved 2008-07-28.
  4. Corrupted NILFS2 partition (/var) - Opinions on a better FS for /var?
  5. Kernel archives: Can't mount nilfs - error searching super root.
  6. the NILFS version 1: overview
  7. does not verify: Re: Does nilfs2 checksum all data?; kernel 4.4.38-v7+ does not verify, too.
  8. "Arch Linux - nilfs-utils 2.2.9-1 (X86_64)".
  9. "Debian -- Details of package nilfs-tools in bullseye".
  10. "Sys-fs/Nilfs-utils – Gentoo Packages".
  11. "GParted -- Features".
  12. NiLFS(2) source commit
  13. Peter Chubb. "SD cards and filesystems for embedded systems". Linux.conf.au.
  14. https://www.complang.tuwien.ac.at/Diplomarbeiten/rohner18.pdf [ bare URL PDF ]