 |
A software repository, or repo for short, is a storage location for software packages. Often a table of contents is also stored, along with metadata. A software repository is typically managed by source or version control, or repository managers. Package managers allow automatically installing and updating repositories, sometimes called "packages".
Many software publishers and other organizations maintain servers on the Internet for this purpose, either free of charge or for a subscription fee. Repositories may be solely for particular programs, such as CPAN for the Perl programming language, or for an entire operating system. Operators of such repositories typically provide a package management system, tools intended to search for, install and otherwise manipulate software packages from the repositories. For example, many Linux distributions use Advanced Packaging Tool (APT), commonly found in Debian based distributions, or Yellowdog Updater, Modified (yum) found in Red Hat based distributions. There are also multiple independent package management systems, such as pacman, used in Arch Linux and equo, found in Sabayon Linux.
As software repositories are designed to include useful packages, major repositories are designed to be malware free. If a computer is configured to use a digitally signed repository from a reputable vendor, and is coupled with an appropriate permissions system, this significantly reduces the threat of malware to these systems. As a side effect, many systems that have these abilities do not need anti-malware software such as antivirus software. [1]
Most major Linux distributions have many repositories around the world that mirror the main repository.
In an enterprise environment, a software repository is usually used to store artifacts, or to mirror external repositories which may be inaccessible due to security restrictions. Such repositories may provide additional functionality, like access control, versioning, security checks for uploaded software, cluster functionality etc. and typically support a variety of formats in one package, so as to cater for all the needs in an enterprise, and thus aiming to provide a single point of truth. Popular examples are JFrog Artifactory, [2] [3] Nexus repository [4] and Cloudsmith, [5] a cloud-based product.
At client side, a package manager helps installing from and updating the repositories.
At server side, a software repository is typically managed by source control or repository managers. Some of the repository managers allow to aggregate other repository location into one URL and provide a caching proxy. When doing continuous builds many artifacts are produced and often centrally stored, so automatically deleting the ones which are not released is important.
A package management system is different from a package development process.
A typical use of a package management system is to facilitate the integration of code from possibly different sources into a coherent stand-alone operating unit. Thus, a package management system might be used to produce a distribution of Linux, possibly a distribution tailored to a specific restricted application.
A package development process, by contrast, is used to manage the co-development of code and documentation of a collection of functions or routines with a common theme, producing thereby a package of software functions that typically will not be complete and usable by themselves. A good package development process will help users conform to good documentation and coding practices, integrating some level of unit testing.
The following table lists a few languages with repositories for contributed software. The "Autochecks" column describes the routine checks done.
Very few people have the ability to test their software under multiple operating systems with different versions of the core code and with other contributed packages they may use. For the R programming language, the Comprehensive R Archive Network (CRAN) runs tests routinely.
To understand how this is valuable, imagine a situation with two developers, Sally and John. Sally contributes a package A. Sally only runs the current version of the software under one version of Microsoft Windows, and has only tested it in that environment. At more or less regular intervals, CRAN tests Sally's contribution under a dozen combinations of operating systems and versions of the core R language software. If one of them generates an error, she gets that error message. With luck, that error message details may provide enough input to allow enable a fix for the error, even if she cannot replicate it with her current hardware and software. Next, suppose John contributes to the repository a package B that uses a package A. Package B passes all the tests and is made available to users. Later, Sally submits an improved version of A, which unfortunately, breaks B. The autochecks make it possible to provide information to John so he can fix the problem.
This example exposes both a strength and a weakness in the R contributed-package system: CRAN supports this kind of automated testing of contributed packages, but packages contributed to CRAN need not specify the versions of other contributed packages that they use. Procedures for requesting specific versions of packages exist, but contributors might not use those procedures.
Beyond this, a repository such as CRAN running regular checks of contributed packages actually provides an extensive if ad hoc test suite for development versions of the core language. If Sally (in the example above) gets an error message she does not understand or thinks is inappropriate, especially from a development version of the language, she can (and often does with R) ask the core development-team for the language for help. In this way, the repository can contribute to improving the quality of the core language software.
| Language, purpose | Package development process | Repository | Install methods | Collaborative development platform | Autochecks |
|---|---|---|---|---|---|
| Haskell | Common Architecture for Building Applications and Libraries [6] | Hackage | cabal (software) | ||
| Java | Maven [7] | ||||
| Julia [8] | |||||
| Common Lisp | Quicklisp [9] | ||||
| .NET | NuGet | NuGet [10] | dotnet add package <package> | ||
| Node.js | node | npm, [11] yarn, bower | npm install <package> yarn add <package> bower install <package> | ||
| Perl | CPAN | PPM [12] | ActiveState | ||
| PHP | PEAR, Composer | PECL, Packagist | composer require <package> pear install <package> | ||
| Python | Setuptools | PyPI | pip, EasyInstall, PyPM, Anaconda | ||
| R | R CMD check process [13] [14] | CRAN [15] | install.packages [16] remotes [17] | GitHub [18] | Often on 12 platforms or combinations of different versions of R (devel, prerel, patched, release) on different operating systems (different versions of Linux, Windows, macOS, and Solaris). |
| Ruby | RubyGems | RubyGems [19] | RubyGems, [19] Bundler [20] | ||
| Rust | Cargo [21] | crates.io [22] | Cargo [21] | ||
| Go | go | pkg.go.dev | go get <package> | GitHub [18] | |
| Dart | Flutter | pub.dev | flutter pub get <package> | ||
| D | DUB | dlang.org | dub add <package> | ||
| TeX, LaTeX | CTAN |
(Parts of this table were copied from a "List of Top Repositories by Programming Language" on Stack Overflow [23] )
Many other programming languages, among them C, C++, and Fortran, do not possess a central software repository with universal scope. Notable repositories with limited scope include:
Package managers help manage repositories and the distribution of them. If a repository is updated, a package manager will typically allow the user to update that repository through the package manager. They also help with managing things such as dependencies between other software repositories. Some examples of Package Managers include:
| Package Manager | Description |
|---|---|
| npm | A package manager for Node.js [24] |
| pip | A package installer for Python [25] |
| apt | For managing Debian Packages [26] |
| Homebrew | A package installer for MacOS that allows one to install packages Apple didn't [27] |
| vcpkg | A package manager for C and C++ [28] [29] |
| yum and dnf | Package manager for Fedora and Red Hat Enterprise Linux [30] |
| pacman | Package manager for Arch Linux [31] |
As part of the development lifecycle, source code is continuously being built into binary artifacts using continuous integration. This may interact with a binary repository manager much like a developer would by getting artifacts from the repositories and pushing builds there. Tight integration with CI servers enables the storage of important metadata such as:
Artifacts and packages inherently mean different things. Artifacts are simply an output or collection of files (ex. JAR, WAR, DLLS, RPM etc.) and one of those files may contain metadata (e.g. POM file). Whereas packages are a single archive file in a well-defined format (ex. NuGet) that contain files appropriate for the package type (ex. DLL, PDB). [32] Many artifacts result from builds but other types are crucial as well. Packages are essentially one of two things: a library or an application. [33]
Compared to source files, binary artifacts are often larger by orders of magnitude, they are rarely deleted or overwritten (except for rare cases such as snapshots or nightly builds), and they are usually accompanied by much metadata such as id, package name, version, license and more.
Metadata describes a binary artifact, is stored and specified separately from the artifact itself, and can have several additional uses. The following table shows some common metadata types and their uses:
| Metadata type | Used for |
|---|---|
| Versions available | Upgrading and downgrading automatically |
| Dependencies | Specify other artifacts that the current artifact depends on |
| Downstream dependencies | Specify other artifacts that depend on the current artifact |
| License | Legal compliance |
| Build date and time | Traceability |
| Documentation | Provide offline availability for contextual documentation in IDEs |
| Approval information | Traceability |
| Metrics | Code coverage, compliance to rules, test results |
| User-created metadata | Custom reports and processes |
The Comprehensive Perl Archive Network (CPAN) is a repository of over 250,000 software modules and accompanying documentation for 39,000 distributions, written in the Perl programming language by over 12,000 contributors. CPAN can denote either the archive network or the Perl program that acts as an interface to the network and as an automated software installer. Most software on CPAN is free and open source software.
Debian, also known as Debian GNU/Linux, is a Linux distribution composed of free and open-source software, developed by the community-supported Debian Project, which was established by Ian Murdock on August 16, 1993. The first version of Debian (0.01) was released on September 15, 1993, and its first stable version (1.1) was released on June 17, 1996. The Debian Stable branch is the most popular edition for personal computers and servers. Debian is also the basis for many other distributions, like PureOS, Ubuntu, Pardus, and Linux Mint.
Slackware is a Linux distribution created by Patrick Volkerding in 1993. Originally based on Softlanding Linux System (SLS), Slackware has been the basis for many other Linux distributions, most notably the first versions of SUSE Linux distributions, and is the oldest distribution that is still maintained.
A package manager or package-management system is a collection of software tools that automates the process of installing, upgrading, configuring, and removing computer programs for a computer in a consistent manner.
The Yellowdog Updater Modified (YUM) is a free and open-source command-line package-management utility for computers running the Linux operating system using the RPM Package Manager. Though YUM has a command-line interface, several other tools provide graphical user interfaces to YUM functionality.
R is a programming language for statistical computing and graphics supported by the R Core Team and the R Foundation for Statistical Computing. Created by statisticians Ross Ihaka and Robert Gentleman, R is used among data miners, bioinformaticians and statisticians for data analysis and developing statistical software. The core R language is augmented by a large number of extension packages containing reusable code and documentation.
Arch Linux is an independently developed, x86-64 general-purpose Linux distribution that strives to provide the latest stable versions of most software by following a rolling-release model. The default installation is a minimal base system, configured by the user to only add what is purposely required.
PCLinuxOS, often shortened to PCLOS, is a rolling release Linux distribution for x86-64 computers, with KDE Plasma, MATE, and XFCE as its default user interfaces. It is a primarily FOSS operating system for personal computers aimed at ease of use.
Puppy Linux is an operating system and family of light-weight Linux distributions that focus on ease of use and minimal memory footprint. The entire system can be run from random-access memory (RAM) with current versions generally taking up about 600 MB (64-bit), 300 MB (32-bit), allowing the boot medium to be removed after the operating system has started. Applications such as AbiWord, Gnumeric and MPlayer are included, along with a choice of lightweight web browsers and a utility for downloading other packages. The distribution was originally developed by Barry Kauler and other members of the community, until Kauler retired in 2013. The tool Woof can build a Puppy Linux distribution from the binary packages of other Linux distributions.
Linux Mint is a community-driven Linux distribution based on Ubuntu, bundled with a variety of free and open-source applications. It can provide full out-of-the-box multimedia support for those who choose to include proprietary software such as multimedia codecs. Compared to Ubuntu, it uses the Cinnamon interface by default, using a different, more traditional layout that can be customized by dragging the applets and creating panels. New applets can also be downloaded.
Nix is a cross-platform package manager that uses a deployment model where software is installed into unique directories generated through cryptographic hashes. It is also the name of the tool's programming language. A package's hash takes into account the dependencies, which is claimed to eliminate dependency hell, as an alternative to the typical solution of installing multiple versions of dependencies at the same time. This package management model advertises more reliable, reproducible, and portable packages.
Readahead is a system call of the Linux kernel that loads a file's contents into the page cache. This prefetches the file so that when it is subsequently accessed, its contents are read from the main memory (RAM) rather than from a hard disk drive (HDD), resulting in much lower file access latencies.
RPM Package Manager (RPM) is a free and open-source package management system. The name RPM refers to the .rpm file format and the package manager program itself. RPM was intended primarily for Linux distributions; the file format is the baseline package format of the Linux Standard Base.
The Python Package Index, abbreviated as PyPI and also known as the Cheese Shop, is the official third-party software repository for Python. It is analogous to the CPAN repository for Perl and to the CRAN repository for R. PyPI is run by the Python Software Foundation, a charity. Some package managers, including pip, use PyPI as the default source for packages and their dependencies.
Anaconda is a distribution of the Python and R programming languages for scientific computing, that aims to simplify package management and deployment. The distribution includes data-science packages suitable for Windows, Linux, and macOS. It is developed and maintained by Anaconda, Inc., which was founded by Peter Wang and Travis Oliphant in 2012. As an Anaconda, Inc. product, it is also known as Anaconda Distribution or Anaconda Individual Edition, while other products from the company are Anaconda Team Edition and Anaconda Enterprise Edition, neither of which are free.
DNF or Dandified YUM is the next-generation version of the Yellowdog Updater, Modified (yum), a package manager for .rpm-based Linux distributions. DNF was introduced in Fedora 18 in 2013; it has been the default package manager since Fedora 22 in 2015, Red Hat Enterprise Linux 8, and OpenMandriva, and is also an alternative package manager for Mageia.
Void Linux is an independent Linux distribution that uses the X Binary Package System (XBPS) package manager, which was designed and implemented from scratch, and the runit init system. Excluding binary kernel blobs, a base install is composed entirely of free software.
R packages are extensions to the R statistical programming language. R packages contain code, data, and documentation in a standardised collection format that can be installed by users of R, typically via a centralised software repository such as CRAN. The large number of packages available for R, and the ease of installing and using them, has been cited as a major factor driving the widespread adoption of the language in data science.
