Singularity (software)

Last updated

Singularity
Original author(s) Gregory Kurtzer (gmk), et al.
Developer(s) Community
Gregory Kurtzer
Stable release
3.8.7 / 17 March 2022;2 years ago (2022-03-17) [1]
Repository
Written in Go [2]
Operating system Linux
Type Operating-system-level virtualization
License 3-clause BSD License [3]
Website apptainer.org OOjs UI icon edit-ltr-progressive.svg

Singularity is a free and open-source computer program that performs operating-system-level virtualization also known as containerization. [4]

Contents

One of the main uses of Singularity is to bring containers and reproducibility to scientific computing and the high-performance computing (HPC) world. [5]

The need for reproducibility requires the ability to use containers to move applications from system to system. [6]

Using Singularity containers, developers can work in reproducible environments of their choosing and design, and these complete environments can easily be copied and executed on other platforms. [7]

In 2021 the Singularity open source project split into two projects called Apptainer and SingularityCE.

Usage workflow for Singularity containers Singularity container workflow.png
Usage workflow for Singularity containers

History

Singularity began as an open-source project in 2015, when a team of researchers at Lawrence Berkeley National Laboratory, led by Gregory Kurtzer, developed the initial version written in the C programming language and released it [8] under the BSD license. [9]

By the end of 2016, many developers from different research facilities joined forces with the team at Lawrence Berkeley National Laboratory to further the development of Singularity. [10]

Singularity quickly attracted the attention of computing-heavy scientific institutions worldwide: [11]

For two years in a row, in 2016 and 2017, Singularity was recognized by HPCwire editors as "One of five new technologies to watch". [19] [20] In 2017 Singularity also won the first place for the category "Best HPC Programming Tool or Technology". [20]

As of 2018, based on the data entered on a voluntary basis in a public registry, Singularity user base was estimated to be greater than 25,000 installations [21] and included users at academic institutions such as Ohio State University and Michigan State University, as well as top HPC centers like Texas Advanced Computing Center, San Diego Supercomputer Center, and Oak Ridge National Laboratory.

In February 2018 the Sylabs [22] company, founded by the Singularity author, was announced [23] to provide commercial support for Singularity. In October of that year Sylabs released version 3.0.0 [24] which was a rewrite in the Go programming language.

Apptainer / Singularity split

In May 2020 Gregory Kurtzer left Sylabs but retained leadership of the Singularity open source project. [25] In May 2021 Sylabs made a fork of the project [26] and called it SingularityCE. [27] [28] In November 2021 the Singularity open source project joined the Linux Foundation [29] and was renamed to Apptainer. [30]

Features

Singularity is able to support natively high-performance interconnects, such as InfiniBand [31] and Intel Omni-Path Architecture (OPA). [32]

Similar to the support for InfiniBand and Intel OPA devices, Singularity can support any PCIe-attached device within the compute node, such as graphic accelerators. [33]

Singularity also has native support for Open MPI library by utilizing a hybrid MPI container approach where OpenMPI exists both inside and outside the container. [31]

These features make Singularity increasingly useful in areas such as machine learning, deep learning and most data-intensive workloads where the applications benefit from the high bandwidth and low latency characteristics of these technologies. [34]

Integration

HPC systems traditionally already have resource management and job scheduling systems in place, so the container runtime environments must be integrated into the existing system resource manager.

Using other enterprise container solutions like Docker in HPC systems would require modifications to the software. [35] Docker containers can be automatically converted to stand-alone singularity files which can then be submitted to HPC resource managers. [36]

Singularity seamlessly integrates with many resource managers [37] including:

See also

Related Research Articles

<span class="mw-page-title-main">Beowulf cluster</span> Type of computing cluster

A Beowulf cluster is a computer cluster of what are normally identical, commodity-grade computers networked into a small local area network with libraries and programs installed which allow processing to be shared among them. The result is a high-performance parallel computing cluster from inexpensive personal computer hardware.

Checkpointing is a technique that provides fault tolerance for computing systems. It basically consists of saving a snapshot of the application's state, so that applications can restart from that point in case of failure. This is particularly important for long running applications that are executed in failure-prone computing systems.

Oracle Grid Engine, previously known as Sun Grid Engine (SGE), CODINE or GRD, was a grid computing computer cluster software system, acquired as part of a purchase of Gridware, then improved and supported by Sun Microsystems and later Oracle. There have been open source versions and multiple commercial versions of this technology, initially from Sun, later from Oracle and then from Univa Corporation.

OS-level virtualization is an operating system (OS) virtualization paradigm in which the kernel allows the existence of multiple isolated user space instances, called containers, zones, virtual private servers (OpenVZ), partitions, virtual environments (VEs), virtual kernels, or jails. Such instances may look like real computers from the point of view of programs running in them. A computer program running on an ordinary operating system can see all resources of that computer. However, programs running inside of a container can only see the container's contents and devices assigned to the container.

The Terascale Open-source Resource and Queue Manager (TORQUE) is a distributed resource manager designed to oversee batch jobs and distributed compute nodes. It offers control and management capabilities for clusters, aiding in utilization, scheduling, and administration tasks.

<span class="mw-page-title-main">The Portland Group</span> American technology company

PGI was a company that produced a set of commercially available Fortran, C and C++ compilers for high-performance computing systems. On July 29, 2013, Nvidia acquired The Portland Group, Inc. As of August 5, 2020, the "PGI Compilers and Tools" technology is a part of the Nvidia HPC SDK product available as a free download from Nvidia.

In computer science, high-throughput computing (HTC) is the use of many computing resources over long periods of time to accomplish a computational task.

Intel Parallel Studio XE was a software development product developed by Intel that facilitated native code development on Windows, macOS and Linux in C++ and Fortran for parallel computing. Parallel programming enables software programs to take advantage of multi-core processors from Intel and other processor vendors.

Windows HPC Server 2008, released by Microsoft on 22 September 2008, is the successor product to Windows Compute Cluster Server 2003. Like WCCS, Windows HPC Server 2008 is designed for high-end applications that require high performance computing clusters. This version of the server software is claimed to efficiently scale to thousands of cores. It includes features unique to HPC workloads: a new high-speed NetworkDirect RDMA, highly efficient and scalable cluster management tools, a service-oriented architecture (SOA) job scheduler, an MPI library based on open-source MPICH2, and cluster interoperability through standards such as the High Performance Computing Basic Profile (HPCBP) specification produced by the Open Grid Forum (OGF).

<span class="mw-page-title-main">Slurm Workload Manager</span> Free and open-source job scheduler for Linux and similar computers

The Slurm Workload Manager, formerly known as Simple Linux Utility for Resource Management (SLURM), or simply Slurm, is a free and open-source job scheduler for Linux and Unix-like kernels, used by many of the world's supercomputers and computer clusters.

BeeGFS is a parallel file system, developed and optimized for high-performance computing. BeeGFS includes a distributed metadata architecture for scalability and flexibility reasons. Its most used and widely known aspect is data throughput.

<span class="mw-page-title-main">LXC</span> Operating system-level virtualization for Linux

Linux Containers (LXC) is an operating-system-level virtualization method for running multiple isolated Linux systems (containers) on a control host using a single Linux kernel.

cgroups is a Linux kernel feature that limits, accounts for, and isolates the resource usage of a collection of processes.

<span class="mw-page-title-main">OpenShift</span> Cloud computing software

OpenShift is a family of containerization software products developed by Red Hat. Its flagship product is the OpenShift Container Platform — a hybrid cloud platform as a service built around Linux containers orchestrated and managed by Kubernetes on a foundation of Red Hat Enterprise Linux. The family's other products provide this platform through different environments: OKD serves as the community-driven upstream, Several deployment methods are available including self-managed, cloud native under ROSA, ARO and RHOIC on AWS, Azure, and IBM Cloud respectively, OpenShift Online as software as a service, and OpenShift Dedicated as a managed service.

Docker is a set of platform as a service (PaaS) products that use OS-level virtualization to deliver software in packages called containers. The service has both free and premium tiers. The software that hosts the containers is called Docker Engine. It was first released in 2013 and is developed by Docker, Inc.

Kubernetes is an open-source container orchestration system for automating software deployment, scaling, and management. Originally designed by Google, the project is now maintained by a worldwide community of contributors, and the trademark is held by the Cloud Native Computing Foundation.

<span class="mw-page-title-main">Mirantis</span> Cloud computing software and services company

Mirantis Inc. is a Campbell, California, based B2B open source cloud computing software and services company. Its primary container and cloud management products, part of the Mirantis Cloud Native Platform suite of products, are Mirantis Container Cloud and Mirantis Kubernetes Engine. The company focuses on the development and support of container and cloud infrastructure management platforms based on Kubernetes and OpenStack. The company was founded in 1999 by Alex Freedland and Boris Renski. It was one of the founding members of the OpenStack Foundation, a non-profit corporate entity established in September, 2012 to promote OpenStack software and its community. Mirantis has been an active member of the Cloud Native Computing Foundation since 2016.

Omni-Path Architecture (OPA) is a high-performance communication architecture developed by Intel. It aims for low communication latency, low power consumption and a high throughput. It directly competes with InfiniBand. Intel planned to develop technology based on this architecture for exascale computing. The current owner of Omni-Path is Cornelis Networks.

<span class="mw-page-title-main">ROCm</span> Parallel computing platform: GPGPU libraries and application programming interface

ROCm is an Advanced Micro Devices (AMD) software stack for graphics processing unit (GPU) programming. ROCm spans several domains: general-purpose computing on graphics processing units (GPGPU), high performance computing (HPC), heterogeneous computing. It offers several programming models: HIP, OpenMP/Message Passing Interface (MPI), and OpenCL.

The Cloud Native Computing Foundation (CNCF) is a Linux Foundation project that was started in 2015 to help advance container technology and align the tech industry around its evolution.

References

  1. ↑ "Releases · apptainer/singularity". github.com. Retrieved 29 June 2022.
  2. ↑ "Singularity+GoLang". GitHub . Retrieved 3 December 2021.
  3. ↑ "Singularity License". Apptainer.org. Singularity Team. Retrieved 3 December 2021.
  4. ↑ "Singularity presentation at FOSDEM 17". archive.fosdem.org.
  5. ↑ Kurtzer, Gregory M.; Sochat, Vanessa; Bauer, Michael W. (2017). "Singularity: Scientific Containers for Mobility of Compute". PLOS ONE. 12 (5): e0177459. Bibcode:2017PLoSO..1277459K. doi: 10.1371/journal.pone.0177459 . PMC   5426675 . PMID   28494014.
  6. ↑ "Singularity, a container for HPC". admin-magazine.com. 24 April 2016.
  7. ↑ "Singularity Manual: Mobility of Compute". Singularity User Guide - Version 2.5.2.
  8. ↑ "Sylabs Brings Singularity Containers into Commercial HPC". top500.org.
  9. ↑ "Singularity License". singularity.lbl.gov. Singularity Team. 19 March 2018. Retrieved 19 March 2018.
  10. ↑ "Changes to the AUTHORS.md file in Singularity source code made in April 2017". GitHub .
  11. ↑ "Berkeley Lab's Open-Source Spinoff Serves Science". 7 June 2017.
  12. ↑ "XStream online user manual, section on Singularity". xstream.stanford.edu.
  13. ↑ "XStream cluster overview". Archived from the original on 24 October 2020. Retrieved 10 April 2018.
  14. ↑ "Sherlock: What's New, Containers and Deep Learning Tools". Stanford Research Computing Center.
  15. ↑ "NIH HPC online user manual, section on Singularity". hpc.nih.gov.
  16. ↑ "NIH HPC Systems". hpc.nih.gov.
  17. ↑ "Singularity on the OSG".
  18. ↑ "Singularity in CMS: Over a million containers served" (PDF).
  19. ↑ "HPCwire Reveals Winners of the 2016 Readers' and Editors' Choice Awards at SC16 Conference in Salt Lake City". HPCwire.
  20. 1 2 "HPCwire Reveals Winners of the 2017 Readers' and Editors' Choice Awards at SC17 Conference in Denver". HPCwire.
  21. ↑ "Voluntary registry of Singularity installations".
  22. ↑ "Sylabs home page" . Retrieved 29 June 2022.
  23. ↑ "Sylabs Emerges from Stealth to Bring Singularity Container Technology to Enterprise Performance Computing" (Press release). 8 February 2018. Retrieved 29 June 2022.
  24. ↑ "Singularity 3.0.0". GitHub.
  25. ↑ "Singularity repository move and company updates" . Retrieved 29 June 2022.
  26. ↑ "Sylabs fork of Singularity" . Retrieved 29 June 2022.
  27. ↑ "SingularityCE" . Retrieved 30 June 2022.
  28. ↑ "SingularityCE". 28 October 2022 – via GitHub.
  29. ↑ "Singularity has joined the Linux Foundation!" . Retrieved 29 June 2022.
  30. ↑ "Apptainer website" . Retrieved 15 February 2023.
  31. 1 2 "Intel Advanced Tutorial: HPC Containers & Singularity – Advanced Tutorial – Intel" (PDF).
  32. ↑ "Intel Application Note: Building Containers for Intel Omni-Path Fabrics using Docker and Singularity" (PDF).
  33. ↑ "Singularity Manual: A GPU example".
  34. ↑ Tallent, Nathan R.; Gawande, Nitin; Siegel, Charles; Vishnu, Abhinav; Hoisie, Adolfy (2018). "Evaluating On-Node GPU Interconnects for Deep Learning Workloads". High Performance Computing Systems. Performance Modeling, Benchmarking, and Simulation. Lecture Notes in Computer Science. Vol. 10724. pp. 3–21. doi:10.1007/978-3-319-72971-8_1. ISBN   978-3-319-72970-1. S2CID   1674152.
  35. ↑ Jonathan Sparks, Cray Inc. (2017). "HPC Containers in use" (PDF).
  36. ↑ "Singularity and Docker" . Retrieved 3 December 2021.
  37. ↑ "Support on existing traditional HPC".
  38. ↑ "HTCondor Stable Release Manual : Singularity Support". Archived from the original on 4 February 2020. Retrieved 4 February 2020.

Further reading

This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.