Monorepo

Last updated

In version-control systems, a monorepo ("mono" meaning 'single' and "repo" being short for 'repository') is a software-development strategy in which the code for a number of projects is stored in the same repository. [1] This practice dates back to at least the early 2000s, [2] when it was commonly called a shared codebase. [2] Google, [3] Meta, [4] Microsoft, [5] Uber, [6] Airbnb, and Twitter [7] all employ very large monorepos with varying strategies to scale build systems and version control software with a large volume of code and daily changes.

Contents

A related concept is a monolithic application, but whereas a monolith combines its sub-projects into one large project, a monorepo may contain multiple independent projects. [8] [9] [10]

Advantages

There are a number of potential advantages to a monorepo over individual repositories: [3] [11]

Ease of code reuse
Similar functionality or communication protocols can be abstracted into shared libraries and directly included by projects, without the need of a dependency package manager.
Simplified dependency management
In a multiple repository environment where multiple projects depend on a third-party dependency, that dependency might be downloaded or built multiple times. In a monorepo the build can be easily optimized, as referenced dependencies all exist in the same codebase.
Atomic commits
When projects that work together are contained in separate repositories, releases need to sync which versions of one project work with the other. And in large enough projects, managing compatible versions between dependencies can become dependency hell. [6] In a monorepo this problem can be negated, since developers may change multiple projects atomically. [12]
Large-scale code refactoring
Since developers have access to the entire project, refactors can ensure that every piece of the project continues to function after a refactor.
Collaboration across teams
In a monorepo that uses source dependencies (dependencies that are compiled from source), [7] teams can improve projects being worked on by other teams. This leads to flexible code ownership.

Limitations and disadvantages

Loss of version information
Although not required, some monorepo builds use one version number across all projects in the repository. This leads to a loss of per-project semantic versioning. [13]
Lack of per-project access control
With split repositories, access to a repository can be granted based upon need. A monorepo allows read access to all software in the project, possibly presenting new security issues. [14] Note that there are versioning systems in which this limitation is not an issue. For example, when Subversion is used, it's possible to download any part of the repo (even a single directory), and path-based authorization can be used to restrict access to certain parts of a repository.
More storage needed by default
With split repositories, you fetch only the project you are interested in by default. With a monorepo, you check out all projects by default. This can take up a significant amount of storage space. While some versioning systems have a mechanism to do a partial checkout, [15] [16] [17] doing so defeats some of the advantages of a monorepo.

Scalability challenges

Companies with large projects have come across hurdles with monorepos, specifically concerning build tools and version control systems. [4] Google's monorepo, speculated to be the largest in the world, meets the classification of an ultra-large-scale system [3] and must handle tens of thousands of contributions every day in a repository over 80 terabytes in size. [18]

Scaling version control software

Companies using or switching to existing version control software found that software could not efficiently handle the amount of data required for a large monorepo. Facebook and Microsoft chose to contribute to or fork existing version control software Mercurial and Git respectively, while Google eventually created their own version control system.

For more than ten years, Google had relied on Perforce hosted on a single machine. In 2005 Google's build servers could get locked up to 10 minutes at a time. Google improved this to 30 seconds–1 minute in 2010. [19] Due to scaling issues, Google eventually developed its own in-house distributed version control system dubbed Piper. [3]

Facebook ran into performance issues with the version control system Mercurial and made upstream contributions to the client, [20] and in January 2014 made it faster than a competing solution in Git. [21]

In May 2017 Microsoft announced that virtually all of its Windows engineers use a Git monorepo. [5] In the transition, Microsoft made substantial upstream contributions to the Git client to remove unnecessary file access and improve handling of large files with Virtual File System for Git. [22]

Scaling build software

Few build tools work well in a monorepo, [7] and flows where builds and continuous integration testing of the entire repository are performed upon check-in will cause performance problems. [13] [14] A build system that processes dependencies as a directed graph (such as Buck, Bazel, Please, or Pants) solves this by compartmentalizing each build or test to the active area of development. [23]

Twitter began development of Pants in 2011, as both Facebook's Buck and Google's Bazel were closed-source at the time. [24] Twitter open-sourced Pants in 2012 under the Apache 2.0 License. [25]

Please is a Go-based build system; it was developed in 2016 by Thought Machine, whose developers were both inspired by Google's Bazel and dissatisfied with Facebook's Buck. [26]

Related Research Articles

Version control is the software engineering practice of controlling, organizing, and tracking different versions in history of computer files; primarily source code text files, but generally any type of file.

<span class="mw-page-title-main">Git</span> Distributed version control software system

Git is a distributed version control system that tracks versions of files. It is often used to control source code by programmers who are developing software collaboratively.

In software development, distributed version control is a form of version control in which the complete codebase, including its full history, is mirrored on every developer's computer. Compared to centralized version control, this enables automatic management branching and merging, speeds up most operations, improves the ability to work offline, and does not rely on a single location for backups. Git, the world's most popular version control system, is a distributed version control system.

<span class="mw-page-title-main">Mantis Bug Tracker</span> Bug tracking system

Mantis Bug Tracker is a free and open source, web-based bug tracking system. The most common use of MantisBT is to track software defects. However, MantisBT is often configured by users to serve as a more generic issue tracking system and project management tool.

In software development, a codebase is a collection of source code used to build a particular software system, application, or software component. Typically, a codebase includes only human-written source code system files; thus, a codebase usually does not include source code files generated by tools or binary library files, as they can be built from the human-written source code. However, it generally does include configuration and property files, as they are the data necessary for the build.

A source-code-hosting facility is a file archive and web hosting facility for source code of software, documentation, web pages, and other works, accessible either publicly or privately. They are often used by open-source software projects and other multi-developer projects to maintain revision and version history, or version control. Many repositories provide a bug tracking system, and offer release management, mailing lists, and wiki-based project documentation. Software authors generally retain their copyright when software is posted to a code hosting facilities.

<span class="mw-page-title-main">Mercurial</span> Distributed revision-control tool for software developers

Mercurial is a distributed revision control tool for software developers. It is supported on Microsoft Windows, Linux, and other Unix-like systems, such as FreeBSD and macOS.

The following tables describe attributes of notable version control and software configuration management (SCM) software systems that can be used to compare and contrast the various systems.

Assembla is a web-based version control and project management software as a service provider for enterprises. It was founded in 2005 and acquired by Idera, Inc. in 2018. It offers Git, Perforce Helix Core and Apache Subversion repository management, integrations with other collaboration tools such as Trello, Slack, GitHub and JIRA. Assembla also offers integrations with customer's managed private clouds.

<span class="mw-page-title-main">TortoiseHg</span>

TortoiseHg is a GUI front-end for Mercurial that runs on Microsoft Windows, Mac OS X, and Linux.

Unity Version Control is a cross-platform commercial distributed version control tool developed by Códice Software for Microsoft Windows, Mac OS X, Linux, and other operating systems. It includes a command-line tool, native GUIs, diff and merge tool and integration with a number of IDEs. It is a full version control stack not based on Git.

<span class="mw-page-title-main">Jenkins (software)</span> Open source automation server

Jenkins is an open source automation server. It helps automate the parts of software development related to building, testing, and deploying, facilitating continuous integration, and continuous delivery. It is a server-based system that runs in servlet containers such as Apache Tomcat. It supports version control tools, including AccuRev, CVS, Subversion, Git, Mercurial, Perforce, ClearCase, and RTC, and can execute Apache Ant, Apache Maven, and sbt based projects as well as arbitrary shell scripts and Windows batch commands.

In version control systems, a repository is a data structure that stores metadata for a set of files or directory structure. Depending on whether the version control system in use is distributed, like Git or Mercurial, or centralized, like Subversion, CVS, or Perforce, the whole set of information in the repository may be duplicated on every user's system or may be maintained on a single server. Some of the metadata that a repository contains includes, among other things, a historical record of changes in the repository, a set of commit objects, and a set of references to commit objects, called heads.

Perforce Software, Inc. is an American developer of software used for developing and running applications, including version control software, web-based repository management, developer collaboration, application lifecycle management, web application servers, debugging tools, platform automation, and agile planning software.

Buck is a multi-language build system developed and used by Meta Platforms, Inc. It was designed for building small, reusable modules consisting of code and resources within a monorepo. It supports many programming languages, including C++, Swift, Unix Shell, Java, Kotlin, Python, Lua, OCaml, Rust and Go. It can produce binary outputs for a variety of target platforms including iOS, Android, .NET, and Java virtual machine (VM) runtime systems. Licensing for Buck1 is under Apache License 2.0, while Buck2 is under either MIT or Apache 2.0.

<span class="mw-page-title-main">Bazel (software)</span> Software tool that automates software builds and tests

Bazel is a free and open-source software tool used for the automation of building and testing software. Google uses the build tool Blaze internally and released an open-source port of the Blaze tool as Bazel, named as an anagram of Blaze. Bazel was first released in March 2015 and entered beta by September 2015. Version 1.0 was released in October 2019.

yarn (package manager) JavaScript package manager

Yarn is one of the main JavaScript package managers, developed in 2016 by Sebastian McKenzie of Meta for the Node.js JavaScript runtime environment. An alternative to the npm package manager, Yarn was created as a collaboration of Facebook, Exponent, Google, and Tilde to solve consistency, security, and performance problems with large codebases.

Piper is a centralized version control system used by Google for its internal software development. Originally designed for Linux, it supports Microsoft Windows and macOS since October 2012.

References

  1. ↑ "Infrastructure as Code | Second edition". Thoughtworks. 2021-02-25. Retrieved 2022-12-01.
  2. 1 2 Mark "Nurgle." Collins (2001). Linux Game Programming. Prima Tech. ISBN   978-0-7615-3255-2. OCLC   1044194694.
  3. 1 2 3 4 Potvin, Rachel; Levenberg, Josh (July 2016). "Why Google Stores Billions of Lines of Code in a Single Repository". Communications of the ACM . 59 (7): 78–87. doi: 10.1145/2854146 . Retrieved 20 July 2018.
  4. 1 2 Goode, Durham; Rain (7 January 2014). "Scaling Mercurial at Facebook – Facebook Code". fb code. Retrieved 24 July 2018.
  5. 1 2 Lardinois, Frederic (24 March 2017). "Microsoft now uses Git and GVFS to develop Windows". TechCrunch. Retrieved 20 July 2018.
  6. 1 2 Aimee Lucido (7 April 2017). Uber Technology Day: Monorepo to Multirepo and Back Again . Retrieved 24 July 2018.
  7. 1 2 3 Dorothy Ordogh (5 April 2018). Pants and Monorepos . Retrieved 24 July 2018.
  8. ↑ Reece, Brock (November 7, 2017). "From Monolith to Monorepo". From Monolith to Monorepo. Since starting at Croud. Retrieved March 19, 2019.
  9. ↑ Savkin, Victor (August 14, 2019). "Misconceptions about Monorepos: Monorepo != Monolith". blog.nrwl.io. Retrieved June 16, 2020.
  10. ↑ Oberlehner, Markus (Jun 12, 2017). "Monorepos in the Wild" . Retrieved July 25, 2018.
  11. ↑ Brousse, Nicolas (2019). "The issue of monorepo and polyrepo in large enterprises". Proceedings of the Conference Companion of the 3rd International Conference on Art, Science, and Engineering of Programming. pp. 1–4. doi:10.1145/3328433.3328435. ISBN   9781450362573. S2CID   201670751 . Retrieved 7 September 2019.
  12. ↑ Santacroce, Ferdinando; Olsson, Aske; Voss, Rasmus; Narebski, Jakub (2016). Git: Mastering Version Control. Packt Publishing Ltd. p. 756. ISBN   9781787122796.
  13. 1 2 Farina, Matt. "Dangers of Monorepo Projects - DZone DevOps". DZone. Retrieved 20 July 2018.
  14. 1 2 点融黑帮 (16 August 2017). "浅谈monorepo" [Talking about monorepo]. Sohu (in Chinese). Retrieved 20 July 2018.
  15. ↑ git partial clone
  16. ↑ Svn Book: Sparse Directories
  17. ↑ Perforce: Clone
  18. ↑ Metz, Cade (16 September 2015). "Google Is 2 Billion Lines of Code—And It's All in One Place". WIRED. Retrieved 20 July 2018.
  19. ↑ Bloch, Dan. "Still All on One Server: Perforce at Scale" (PDF). Retrieved 23 July 2018.
  20. ↑ Claburn, Thomas. "Facebook is writing a Mercurial server in Rust. This is not a drill". The Register. Retrieved 20 July 2018.
  21. ↑ Blewitt, Alex (9 Jan 2014). "Facebook makes Mercurial faster than Git". InfoQ. Retrieved 24 July 2018.
  22. ↑ Bright, Peter (24 May 2017). "Windows switch to Git almost complete: 8,500 commits and 1,760 builds each day". Ars Technica. Retrieved 20 July 2018.
  23. ↑ Hammant, Paul; Smith, Steve. "Trunk Based Development". trunkbaseddevelopment. Retrieved 24 July 2018.
  24. ↑ Mohilo, Dominik (10 June 2016). "8 Build-Tools im Vergleich: Ant – Buildr – Maven – Bazel – Buck – Gradle – Pants – sbt - JAXenter" [8 build tools compared: Ant - Buildr - Maven - Bazel - Buck - Gradle - Pants - sbt]. JAXenter (in German). Retrieved 20 July 2018.
  25. ↑ Moore, Madison (3 May 2016). "GitLab releases security fixes, Pants 1.0, and Sauce Labs integration for JIRA—SD Times news digest: May 3, 2016 - SD Times". SD Times. Retrieved 20 July 2018.
  26. ↑ Ebden, Peter (December 2017). "Please - the Thought Machine Build System". Blog. Thought Machine. Archived from the original on 2019-12-28.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.