Zarr is an open standard for storing large multidimensional array data. It specifies a protocol and data format, and is designed to be "cloud ready" including random access, by dividing data into subsets referred to as chunks.[1][2] Zarr can be used within many programming languages, including Python, Java, JavaScript, C++, Rust and Julia.[3] It has been used by organizations such as Google and Microsoft to publish large datasets.[4][5] Early versions of Zarr were first released in 2015 by Alistair Miles.[6][7]
Zarr is designed to support high-throughput distributed I/O on different storage systems, which is a common requirement in cloud computing. Multiple read operations can efficiently occur to a Zarr array in parallel, or multiple write operations in parallel.[8]
Format description
An illustration of Zarr's chunking data format.
The main data format in Zarr is multidimensional arrays. For parallelisable access, these arrays are stored and accessed as a grid of so-called "chunks". The actual data format on disk depends on the compressor and storage plugins selected by the user.[8]
Zarr's design was influenced by that of HDF5, and so it includes similar features for metadata and grouping: arrays can be grouped into named hierarchies, and they can also be annotated with key-value metadata stored alongside the array.[8]
Due to its efficient handling of tensors, Zarr is being used to publish weather and satellite data [9] and energy data,[10] among others.
For bioimaging such as microscopy, a consortium called the Open Microscopy Environment (OME) created a format called "OME-Zarr", based on Zarr with some discipline-specific extensions.[11] The .zarr specification enables granular representation of outputs of complex experiments, such as high content screening assays. Each plate read in the microscope contains multiple wells, and to scan each well, multiple fields are needed. Each image may have up to 5 dimensions (time points, imaging channels and the three space dimensions). It may also include resolution pyramids, enabling better performance of visualization tools. As Zarr uses multiple directories for organizing data, each of these different fields can be specified and retrieved independently, for example by retrieving a custom URL from object storage databases. [11]
↑ Sansal, Altay; Kainkaryam, Sribharath; Lasscock, Ben; Valenciano, Alejandro (2023). "MDIO: Open-source format for multidimensional energy data". The Leading Edge. 42 (7). Society of Exploration Geophysicists: 465–473. Bibcode:2023LeaEd..42..465S. doi:10.1190/tle42070465.1. ISSN1938-3789.
This page is based on this Wikipedia article Text is available under the CC BY-SA 4.0 license; additional terms may apply. Images, videos and audio are available under their respective licenses.
