File locking is a mechanism that restricts access to a computer file, or to a region of a file, by allowing only one user or process to modify or delete it at a specific time and to prevent reading of the file while it's being modified or deleted.
Systems implement locking to prevent the classic interceding update scenario, which is a typical example of a race condition, by enforcing the serialization of update processes to any given file. The following example illustrates the interceding update problem:
Process A reads a customer record from a file containing account information, including the customer's account balance and phone number.
Process B now reads the same record from the same file, so it has its own copy.
Process A changes the account balance in its copy of the customer record and writes the record back to the file.
Process B, which still has the original stale value for the account balance in its copy of the customer record, updates the account balance and writes the customer record back to the file.
Process B has now written its stale account-balance value to the file, causing the changes made by process A to be lost.
Most operating systems support the concept of record locking, which means that individual records within any given file may be locked, thereby increasing the number of concurrent update processes. Database maintenance uses file locking, whereby it can serialize access to the entire physical file underlying a database. Although this does prevent any other process from accessing the file, it can be more efficient than individually locking many regions in the file by removing the overhead of acquiring and releasing each lock.
Poor use of file locks, like any computer lock, can result in poor performance or in deadlocks. File locking may also refer to additional security applied by a computer user either by using Windows security, NTFS permissions or by installing a third party file locking software.
In mainframes
IBM pioneered file locking in 1963 for use in mainframe computers using OS/360, where it was termed "exclusive control".[1]
In Microsoft Windows
Microsoft Windows uses three distinct mechanisms to manage access to shared files:
using share-access controls that allow applications to specify whole-file access-sharing for read, write, or delete[2]
using byte-range locks to arbitrate read and write access to regions within a single file[3]
by Windows file systems disallowing executing files from being opened for write or delete access
Windows inherits the semantics of share-access controls from the MS-DOS system, where sharing was introduced in MS-DOS3.3. Thus, an application must explicitly allow sharing when it opens a file; otherwise it has exclusive read, write, and delete access to the file until closed (other types of access, such as those to retrieve the attributes of a file are allowed.)
For a file opened with shared access, applications may then use byte-range locking to control access to specific regions of the file. Such byte-range locks specify a region of the file (offset and length) and the type of lock (shared or exclusive). Note that the region of the file being locked is not required to have data within the file, and applications sometimes exploit this ability to implement their functionality.
For applications that use the file read/write APIs in Windows, byte-range locks are enforced (also referred to as mandatory locks) by the file systems that execute within Windows. For applications that use the file mapping APIs in Windows, byte-range locks are not enforced (also referred to as advisory locks.) Byte-range locking may also have other side-effects on the Windows system. For example, the Windows file-sharing mechanism will typically disable client side caching of a file for all clients when byte-range locks are used by any client. The client will observe slower access because read and write operations must be sent to the server where the file is stored.
Improper error-handling in an application program can lead to a scenario where a file is locked (either using "share" access or with byte-range file locking) and cannot be accessed by other applications. If so, the user may be able to restore file access by manually terminating the malfunctioning program. This is typically done through the Task Manager utility.
The sharing mode (dwShareMode) parameter of the CreateFile[2] function (used to open files) determines file-sharing. The sharing mode can be specified to allow sharing the file for read, write, or delete access, or any combination of these. Subsequent attempts to open the file must be compatible with all previously granted sharing-access to the file. When the file is closed, sharing-access restrictions are adjusted to remove the restrictions imposed by that specific file open.
Byte-range locking type is determined by the dwFlags parameter in the LockFileEx[4] function used to lock a region of a file. The Windows API function LockFile[5] can also be used and acquires an exclusive lock on the region of the file.
Any file containing an executable program file that is currently running on the computer system as a program (e.g. an EXE, COM, DLL, CPL or other binary program file format) is normally locked by the operating system itself, preventing any application from modifying or deleting it. Any attempt to do so will be denied with a sharing violation error, despite the fact that the program file is not opened by any application. However, some access is still allowed. For example, a running application file can be renamed or copied (read) even when executing.
Files are accessed by applications in Windows by using file handles. These file handles can be explored with the Process Explorer utility. This utility can also be used to force-close handles without needing to terminate the application holding them. This can cause an undefined behavior, since the program will receive an unexpected error when using the force-closed handle and may even operate on an unexpected file since the handle number may be recycled.[citation needed]
Microsoft Windows XP and Server 2003 editions have introduced volume snapshot (VSS) capability to NTFS, allowing open files to be accessed by backup software despite any exclusive locks. However, unless software is rewritten to specifically support this feature, the snapshot will be crash consistent only, while properly supported applications can assist the operating system in creating "transactionally consistent" snapshots. Other commercial software for accessing locked files under Windows include File Access Manager and Open File Manager. These work by installing their own drivers to access the files in kernel mode.
In Unix-like systems
Unix-like operating systems (including Linux and Apple's macOS) do not normally automatically lock open files. Several kinds of file-locking mechanisms are available in different flavors of Unix, and many operating systems support more than one kind for compatibility. The most common mechanism is fcntl. Two other such mechanisms are flock(2) and lockf(3), each of which may be implemented atop fcntl or may be implemented separately from fcntl. Although some types of locks can be configured to be mandatory, file locks under Unix are by default advisory. This means that cooperating processes may use locks to coordinate access to a file among themselves, but uncooperative processes are also free to ignore locks and access the file in any way they choose. In other words, file locks lock out other file lockers only, not I/O.
Two kinds of locks are offered: shared locks and exclusive locks. In the case of fcntl, different kinds of locks may be applied to different sections (byte ranges) of a file, or else to the whole file. Shared locks can be held by multiple processes at the same time, but an exclusive lock can only be held by one process, and cannot coexist with a shared lock. To acquire a shared lock, a process must wait until no processes hold any exclusive locks. To acquire an exclusive lock, a process must wait until no processes hold either kind of lock. Unlike locks created by fcntl, those created by flock are preserved across forks, making them useful in forking servers. It is therefore possible for more than one process to hold an exclusive lock on the same file, provided these processes share a filial relationship and the exclusive lock was initially created in a single process before being duplicated across a fork.
Shared locks are sometimes called "read locks" and exclusive locks are sometimes called "write locks". However, because locks on Unix are advisory, this isn't enforced. Thus it is possible for a database to have a concept of "shared writes" vs. "exclusive writes"; for example, changing a field in place may be permitted under shared access, whereas garbage-collecting and rewriting the database may require exclusive access.
File locks apply to the actual file, rather than the file name. This is important since Unix allows multiple names to refer to the same file. Together with non-mandatory locking, this leads to great flexibility in accessing files from multiple processes. On the other hand, the cooperative locking approach can lead to problems when a process writes to a file without obeying file locks set by other processes.
For this reason, some Unix-like operating systems also offer limited support for mandatory locking.[6] On such systems, a file whose setgid bit is on but whose group execution bit is off when that file is opened will be subject to automatic mandatory locking if the underlying filesystem supports it. However, non-local NFS partitions tend to disregard this bit.[7] If a file is subject to mandatory locking, attempts to read from a region that is locked with an exclusive lock, or to write to a region that is locked with a shared or exclusive lock, will block until the lock is released. This strategy first originated in SystemV, and can be seen today in the Solaris, HP-UX, and Linux operating systems. It is not part of POSIX, however, and BSD-derived operating systems such as FreeBSD, OpenBSD, NetBSD, and Apple's macOS do not support it.[8] Linux also supports mandatory locking through the special -o mand parameter for file system mounting (mount(8)), but this is rarely used.
Some Unix-like operating systems prevent attempts to open the executable file of a running program for writing; this is a third form of locking, separate from those provided by fcntl and flock.
Problems
More than one process can hold an exclusive flock on a given file if the exclusive lock was duplicated across a later fork. This simplifies coding for network servers and helps prevent race conditions, but can be confusing to the unaware.
Mandatory locks have no effect on the unlink system call. Consequently, certain programs may, effectively, circumvent mandatory locking. Stevens & Rago (2005) observed that the ed editor indeed did that.[9]
Whether and how flock locks work on network filesystems, such as NFS, is implementation dependent. On BSD systems, flock calls on a file descriptor open to a file on an NFS-mounted partition are successful no-ops. On Linux prior to 2.6.12, flock calls on NFS files would act only locally. Kernel2.6.12 and above implement flock calls on NFS files using POSIX byte-range locks. These locks will be visible to other NFS clients that implement fcntl-style POSIX locks, but invisible to those that do not.[10]
Lock upgrades and downgrades release the old lock before applying the new lock. If an application downgrades an exclusive lock to a shared lock while another application is blocked waiting for an exclusive lock, the latter application may get the exclusive lock and lock the first application out. This means that lock downgrades can block, which may be counter-intuitive.
Allfcntl locks associated with a file for a given process are removed when any file descriptor for that file is closed by that process, even if a lock was never requested for that file descriptor. Also, fcntl locks are not inherited by a child process. The fcntl close semantics are particularly troublesome for applications that call subroutine libraries that may access files. Neither of these "bugs" occurs using real flock-style locks.
Preservation of the lock status on open file descriptors passed to another process using a Unix domain socket is implementation dependent.
Buffered I/O problems
One source of lock failure occurs when buffered I/O has buffers assigned in the user's local workspace, rather than in an operating system buffer pool. fread and fwrite are commonly used to do buffered I/O, and once a section of a file is read, another attempt to read that same section will, most likely, obtain the data from the local buffer. The problem is another user attached to the same file has their own local buffers, and the same thing is happening for them. An fwrite of data obtained from the buffer by fread will not be obtaining the data from the file itself, and some other user could have changed it. Both could use flock to ensure exclusive access, which prevents simultaneous writes, but since the reads are reading from the buffer and not the file itself, any data changed by user#1 can be lost by user#2 (over-written). The best solution to this problem is to use unbuffered I/O (read and write) with flock, which also means using lseek instead of fseek and ftell. Of course, you'll have to make adjustments for function parameters and results returned. Generally speaking, buffered I/O is unsafe when used with shared files.
In AmigaOS
In AmigaOS, a lock on a file (or directory) can be acquired using the Lock function (in the dos.library). A lock can be shared (other processes can read the file/directory, but can't modify or delete it), or exclusive so that only the process which successfully acquires the lock can access or modify the object. The lock is on the whole object and not part of it. The lock must be released with the UnLock function: unlike in Unix, the operating system does not implicitly unlock the object when the process terminates.
Lock files
Shell scripts and other programs often use a strategy similar to the use of file locking: creation of lock files, which are files whose contents are irrelevant (although often one will find the process identifier of the holder of the lock in the file) and whose sole purpose is to signal by their presence that some resource is locked. A lock file is often the best approach if the resource to be controlled is not a regular file at all, so using methods for locking files does not apply. For example, a lock file might govern access to a set of related resources, such as several different files, directories, a group of disk partitions, or selected access to higher level protocols like servers or database connections.
When using lock files, care must be taken to ensure that operations are atomic. To obtain a lock, the process must verify that the lock file does not exist and then create it, whilst preventing another process from creating it in the meantime. Various methods to do this include:
Using the lockfile command (a conditional semaphore-file creator distributed in the procmail package).
System calls that create a file, but fail if the file already exists. (System calls are available from languages such as C or C++, and shell scripts can make use of noclobber)
Using the mkdir command and checking the exit code for failure[11]
Lock files are often named with a tilde (~) prefixed to the name of the file they are locking, or a duplicate of the full file name suffixed with .LCK. If they are locking a resource other than a file, they may be named more arbitrarily.
Unlocker software
An unlocker is a utility used to determine what process is locking a file, and displays a list of processes as well as choices on what to do with the process (kill task, unlock, etc.) along with a list of file options such as delete or rename. Their purpose is to remove improper or stale file locks, which often arise from anomalous situations, such as crashed or hung processes, that lead to file locks that persist despite the owning process having died already. On some Unix-like systems, utilities such as fstat and lockf can be used to inspect the state of file locks by process, by filename, or both.[citation needed]
On Windows systems, if a file is locked, it's possible to schedule its moving or deletion to be performed on the next reboot. This approach is typically used by installers to replace locked system files.
Version control systems
In version control systems file locking is used to prevent two users changing the same file version in parallel and then when saving, the second user to overwrite what first user changed. This is implemented by marking locked files as read-only in the file system. A user wanting to change the file performs an unlock (also called checkout) operation, and until a check-in (store) operation is done, or the lock is reverted, nobody else is allowed to unlock the file.
NT File System (NTFS) is a proprietary journaling file system developed by Microsoft in the 1990s.
Network File System (NFS) is a distributed file system protocol originally developed by Sun Microsystems (Sun) in 1984, allowing a user on a client computer to access files over a computer network much like local storage is accessed. NFS, like many other protocols, builds on the Open Network Computing Remote Procedure Call system. NFS is an open IETF standard defined in a Request for Comments (RFC), allowing anyone to implement the protocol.
In computer security, an access-control list (ACL) is a list of permissions associated with a system resource. An ACL specifies which users or system processes are granted access to resources, as well as what operations are allowed on given resources. Each entry in a typical ACL specifies a subject and an operation. For instance,
A Berkeley (BSD) socket is an application programming interface (API) for Internet domain sockets and Unix domain sockets, used for inter-process communication (IPC). It is commonly implemented as a library of linkable modules. It originated with the 4.2BSD Unix operating system, which was released in 1983.
In computing, tar is a computer software utility for collecting many files into one archive file, often referred to as a tarball, for distribution or backup purposes. The name is derived from "tape archive", as it was originally developed to write data to sequential I/O devices with no file system of their own, such as devices that use magnetic tape. The archive data sets created by tar contain various file system parameters, such as name, timestamps, ownership, file-access permissions, and directory organization. POSIX abandoned tar in favor of pax, yet tar sees continued widespread use.
Server Message Block (SMB) is a communication protocol used to share files, printers, serial ports, and miscellaneous communications between nodes on a network. On Microsoft Windows, the SMB implementation consists of two vaguely named Windows services: "Server" and "Workstation". It uses NTLM or Kerberos protocols for user authentication. It also provides an authenticated inter-process communication (IPC) mechanism.
A virtual file system (VFS) or virtual filesystem switch is an abstract layer on top of a more concrete file system. The purpose of a VFS is to allow client applications to access different types of concrete file systems in a uniform way. A VFS can, for example, be used to access local and network storage devices transparently without the client application noticing the difference. It can be used to bridge the differences in Windows, classic Mac OS/macOS and Unix filesystems, so that applications can access files on local file systems of those types without having to know what type of file system they are accessing.
The Andrew File System (AFS) is a distributed file system which uses a set of trusted servers to present a homogeneous, location-transparent file name space to all the client workstations. It was developed by Carnegie Mellon University as part of the Andrew Project. Originally named "Vice", "Andrew" refers to Andrew Carnegie and Andrew Mellon. Its primary use is in distributed computing.
In computing, mmap(2) is a POSIX-compliant Unix system call that maps files or devices into memory. It is a method of memory-mapped file I/O. It implements demand paging because file contents are not immediately read from disk and initially use no physical RAM at all. The actual reads from disk are performed after a specific location is accessed, in a lazy manner. After the mapping is no longer needed, the pointers must be unmapped with munmap(2). Protection information—for example, marking mapped regions as executable—can be managed using mprotect(2), and special treatment can be enforced using madvise(2).
In Unix and Unix-like computer operating systems, a file descriptor is a process-unique identifier (handle) for a file or other input/output resource, such as a pipe or network socket.
In computing, a named pipe is an extension to the traditional pipe concept on Unix and Unix-like systems, and is one of the methods of inter-process communication (IPC). The concept is also found in OS/2 and Microsoft Windows, although the semantics differ substantially. A traditional pipe is "unnamed" and lasts only as long as the process. A named pipe, however, can last as long as the system is up, beyond the life of the process. It can be deleted if no longer used. Usually a named pipe appears as a file, and generally processes attach to it for IPC.
df is a standard Unix command used to display the amount of available disk space for file systems on which the invoking user has appropriate read access. df is typically implemented using the statfs or statvfs system calls.
In computer security, mandatory access control (MAC) refers to a type of access control by which a secured environment constrains the ability of a subject or initiator to access or modify on an object or target. In the case of operating systems, the subject is a process or thread, while objects are files, directories, TCP/UDP ports, shared memory segments, or IO devices. Subjects and objects each have a set of security attributes. Whenever a subject attempts to access an object, the operating system kernel examines these security attributes, examines the authorization rules in place, and decides whether to grant access. A database management system, in its access control mechanism, can also apply mandatory access control; in this case, the objects are tables, views, procedures, etc.
In computing, a file system or filesystem governs file organization and access. A local file system is a capability of an operating system that services the applications running on the same computer. A distributed file system is a protocol that provides file access between networked computers.
The Access Database Engine is a database engine on which several Microsoft products have been built. The first version of Jet was developed in 1992, consisting of three modules which could be used to manipulate a database.
In computing, ioctl is a system call for device-specific input/output operations and other operations which cannot be expressed by regular file semantics. It takes a parameter specifying a request code; the effect of a call depends completely on the request code. Request codes are often device-specific. For instance, a CD-ROM device driver which can instruct a physical device to eject a disc would provide an ioctl request code to do so. Device-independent request codes are sometimes used to give userspace access to kernel functions which are only used by core system software or still under development.
Remote File Sharing (RFS) is a Unix operating system component for sharing resources, such as files, devices, and file system directories, across a network, in a network-independent manner, similar to a distributed file system. It was developed at Bell Laboratories of AT&T in the 1980s, and was first delivered with UNIX System V Release 3 (SVR3). RFS relied on the STREAMS Transport Provider Interface feature of this operating system. It was also included in UNIX System V Release 4, but as that also included the Network File System (NFS) which was based on TCP/IP and more widely supported in the computing industry, RFS was little used. Some licensees of AT&T UNIX System V Release 4 did not include RFS support in SVR4 distributions, and Sun Microsystems removed it from Solaris 2.4.
The following tables compare general and technical information for a number of file systems.
In Unix-like operating systems, a device file, device node, or special file is an interface to a device driver that appears in a file system as if it were an ordinary file. There are also special files in DOS, OS/2, and Windows. These special files allow an application program to interact with a device by using its device driver via standard input/output system calls. Using standard system calls simplifies many programming tasks, and leads to consistent user-space I/O mechanisms regardless of device features and functions.
↑ Viega, John; Messier, Matt (2003). "2.8 Locking Files". Secure Programming Cookbook for C and C++ (1sted.). Sabastopol, CA: O'Reilly Media. p.792. ISBN978-0-596-00394-4. Support for mandatory locks varies greatly from one Unix variant to another. Both Linux and Solaris support mandatory locks, but Darwin, FreeBSD, NetBSD, and OpenBSD do not, even though they export the interface used by Linux and Solaris to support them. On such systems, this interface creates advisory locks. Support for mandatory locking does not extend to NFS.
↑ Stevens, W. Richard; Rago, Stephen A. (27 June 2005). Advanced Programming in the UNIX Environment (Seconded.). Addison-Wesley Professional. p.456. ISBN978-0201433074.
File-private POSIX locks, a LWN.net article on file locks supported on Linux that behave differently from POSIX locks regarding inheritance and behavior on close
This page is based on this Wikipedia article Text is available under the CC BY-SA 4.0 license; additional terms may apply. Images, videos and audio are available under their respective licenses.
