Gfarm file system

Last updated
Gfarm file system
Developer(s) NPO Tsukuba OSS Technical Support Center
Stable release
2.7.22 / March 29, 2023
Operating system Linux, FreeBSD, NetBSD, OS X, Solaris
Type Distributed file system
License X11
Website Gfarm file system

Gfarm file system is an open-source distributed file system, generally used for large-scale cluster computing and wide-area data sharing, and provides features to manage replica location explicitly. The name is derived from the Grid Data Farm architecture it implements.

Contents

Grid Datafarm is a petascale data-intensive computing project initiated in Japan. [1] The project is a collaboration among High Energy Accelerator Research Organization (KEK), National Institute of Advanced Industrial Science and Technology (AIST), [2] the University of Tokyo, Tokyo Institute of Technology and University of Tsukuba. The challenge involves construction of a Peta- to Exascale parallel filesystem exploiting local storage of PCs spread over the worldwide Grid.

See also

Related Research Articles

Grid computing is the use of widely distributed computer resources to reach a common goal. A computing grid can be thought of as a distributed system with non-interactive workloads that involve many files. Grid computing is distinguished from conventional high-performance computing systems such as cluster computing in that grid computers have each node set to perform a different task/application. Grid computers also tend to be more heterogeneous and geographically dispersed than cluster computers. Although a single grid can be dedicated to a particular application, commonly a grid is used for a variety of purposes. Grids are often constructed with general-purpose grid middleware software libraries. Grid sizes can be quite large.

<span class="mw-page-title-main">High-performance computing</span> Computing with supercomputers and clusters

High-performance computing (HPC) uses supercomputers and computer clusters to solve advanced computation problems.

Oracle Grid Engine, previously known as Sun Grid Engine (SGE), CODINE or GRD, was a grid computing computer cluster software system, acquired as part of a purchase of Gridware, then improved and supported by Sun Microsystems and later Oracle. There have been open source versions and multiple commercial versions of this technology, initially from Sun, later from Oracle and then from Univa Corporation.

Ian Tremere Foster is a New Zealand-American computer scientist. He is a distinguished fellow, senior scientist, and director of the Data Science and Learning division at Argonne National Laboratory, and a professor in the department of computer science at the University of Chicago.

<span class="mw-page-title-main">David Bader (computer scientist)</span> American computer scientist

David A. Bader is a Distinguished Professor and Director of the Institute for Data Science at the New Jersey Institute of Technology. Previously, he served as the Chair of the Georgia Institute of Technology School of Computational Science & Engineering, where he was also a founding professor, and the executive director of High-Performance Computing at the Georgia Tech College of Computing. In 2007, he was named the first director of the Sony Toshiba IBM Center of Competence for the Cell Processor at Georgia Tech. Bader has served on the Computing Research Association's Board of Directors, the National Science Foundation's Advisory Committee on Cyberinfrastructure, and on the IEEE Computer Society's Board of Governors. He is an expert in the design and analysis of parallel and multicore algorithms for real-world applications such as those in cybersecurity and computational biology. His main areas of research are at the intersection of high-performance computing and real-world applications, including cybersecurity, massive-scale analytics, and computational genomics. Bader built the first Linux supercomputer using commodity processors and a high-speed interconnection network.

<span class="mw-page-title-main">University of Tsukuba</span> National University in Tsukuba, Ibaraki, Japan

The University of Tsukuba is a public research university located in Tsukuba, Ibaraki, Japan. It is a top 10 Designated National University, and was ranked Type A by the Japanese government as part of the Top Global University Project.

GPFS is high-performance clustered file system software developed by IBM. It can be deployed in shared-disk or shared-nothing distributed parallel modes, or a combination of these. It is used by many of the world's largest commercial companies, as well as some of the supercomputers on the Top 500 List. For example, it is the filesystem of the Summit at Oak Ridge National Laboratory which was the #1 fastest supercomputer in the world in the November 2019 TOP500 list of supercomputers. Summit is a 200 Petaflops system composed of more than 9,000 POWER9 processors and 27,000 NVIDIA Volta GPUs. The storage filesystem called Alpine has 250 PB of storage using Spectrum Scale on IBM ESS storage hardware, capable of approximately 2.5TB/s of sequential I/O and 2.2TB/s of random I/O.

<span class="mw-page-title-main">Charlie Catlett</span> American computer scientist

Charlie Catlett is a senior computer scientist at Argonne National Laboratory and a visiting senior fellow at the Mansueto Institute for Urban Innovation at the University of Chicago. From 2020 to 2022 he was a senior research scientist at the University of Illinois Discovery Partners Institute. He was previously a senior computer scientist at Argonne National Laboratory and a senior fellow in the Computation Institute, a joint institute of Argonne National Laboratory and The University of Chicago, and a senior fellow at the University of Chicago's Harris School of Public Policy.

The D-Grid Initiative was a government project to fund computer infrastructure for education and research (e-Science) in Germany. It uses the term grid computing. D-Grid started September 1, 2005 with six community projects and an integration project (DGI) as well as several partner projects.

A grid file system is a computer file system whose goal is improved reliability and availability by taking advantage of many smaller file storage areas.

Grid-oriented Storage (GOS) was a term used for data storage by a university project during the era when the term grid computing was popular.

<span class="mw-page-title-main">Computer cluster</span> Set of computers configured in a distributed computing system

A computer cluster is a set of computers that work together so that they can be viewed as a single system. Unlike grid computers, computer clusters have each node set to perform the same task, controlled and scheduled by software.

<span class="mw-page-title-main">Fabric computing</span>

Fabric computing or unified computing involves constructing a computing fabric consisting of interconnected nodes that look like a weave or a fabric when seen collectively from a distance.

<span class="mw-page-title-main">MTA SZTAKI Laboratory of Parallel and Distributed Systems</span> Hungarian research laboratory

The Laboratory of Parallel and Distributed Systems (LPDS), as a department of MTA SZTAKI, is a research laboratory in distributed grid and cloud technologies. LPDS is a founding member of the Hungarian Grid Competence Centre, the Hungarian National Grid Initiative, and the Hungarian OpenNebula Community, and also coordinates several European grid/cloud projects.

<span class="mw-page-title-main">Róbert Lovas</span> Hungarian computer scientist

Róbert Lovas is a Hungarian computer scientist at SZTAKI, Budapest, Hungary.

The Hitachi SR2201 was a distributed memory parallel system that was introduced in March 1996 by Hitachi. Its processor, the 150 MHz HARP-1E based on the PA-RISC 1.1 architecture, solved the cache miss penalty by pseudo vector processing (PVP). In PVP, data was loaded by prefetching to a special register bank, bypassing the cache. Each processor had a peak performance of 300 MFLOPS, giving the SR2201 a peak performance of 600 GFLOPS. Up to 2048 RISC processors could be connected via a high-speed three-dimensional crossbar network, which was able to transfer data at 300 MB/s over each link.

Ignacio Martín Llorente is an entrepreneur, researcher and educator in the field of cloud and distributed computing. He is the director of OpenNebula, a visiting scholar at Harvard University and a full professor at Complutense University.

<span class="mw-page-title-main">Data grid</span> Set of services used to access, modify and transfer geographical data

A data grid is an architecture or set of services that gives individuals or groups of users the ability to access, modify and transfer extremely large amounts of geographically distributed data for research purposes. Data grids make this possible through a host of middleware applications and services that pull together data and resources from multiple administrative domains and then present it to users upon request. The data in a data grid can be located at a single site or multiple sites where each site can be its own administrative domain governed by a set of security restrictions as to who may access the data. Likewise, multiple replicas of the data may be distributed throughout the grid outside their original administrative domain and the security restrictions placed on the original data for who may access it must be equally applied to the replicas. Specifically developed data grid middleware is what handles the integration between users and the data they request by controlling access while making it available as efficiently as possible. The adjacent diagram depicts a high level view of a data grid.

A distributed file system for cloud is a file system that allows many clients to have access to data and supports operations on that data. Each data file may be partitioned into several parts called chunks. Each chunk may be stored on different remote machines, facilitating the parallel execution of applications. Typically, data is stored in files in a hierarchical tree, where the nodes represent directories. There are several ways to share files in a distributed architecture: each solution must be suitable for a certain type of application, depending on how complex the application is. Meanwhile, the security of the system must be ensured. Confidentiality, availability and integrity are the main keys for a secure system.

<span class="mw-page-title-main">Ishfaq Ahmad (computer scientist)</span> Computer scientist and university professor

Ishfaq Ahmad is a computer scientist, IEEE Fellow and Professor of Computer Science and Engineering at the University of Texas at Arlington (UTA). He is the Director of the Center For Advanced Computing Systems (CACS) and has previously directed IRIS at UTA. He is widely recognized for his contributions to scheduling techniques in parallel and distributed computing systems, and video coding.

References

  1. Osamu Tatebe, Kohei Hiraga, Noriyuki Soda, "Gfarm Grid File System", New Generation Computing, Ohmsha, Ltd. and Springer, Vol. 28, No. 3, pp.257-275, 2010.
  2. High Performance Computing Infrastructure project