Apache Mesos

Last updated
Apache Mesos
Developer(s) Apache Software Foundation
Stable release
1.11.0 / November 24, 2020;3 years ago (2020-11-24) [1]
Repository Mesos Repository
Written in C++
Type Cluster management software
License Apache License 2.0
Website mesos.apache.org   OOjs UI icon edit-ltr-progressive.svg

Apache Mesos is an open-source project to manage computer clusters. It was developed at the University of California, Berkeley.

Contents

History

Mesos began as a research project in the UC Berkeley RAD Lab by then PhD students Benjamin Hindman, Andy Konwinski, and Matei Zaharia, as well as professor Ion Stoica. The students started working on the project as part of a course taught by David Culler. It was originally named Nexus but due to a conflict with another university's project, was renamed to Mesos. [2]

Mesos was first presented in 2009 (while still named Nexus) by Andy Konwinski at HotCloud '09 in a talk accompanying the first paper published about the project. [3] Later in 2011 it was presented in a more mature state in a talk by Zaharia at the Usenix Symposium on Networked Systems Design and Implementation conference about the paper "Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center" by Benjamin Hindman, Andy Konwinski, Zaharia, Ali Ghodsi, Anthony D. Joseph, Randy Katz, Scott Shenker, Ion Stoica. [4]

On July 27, 2016, the Apache Software Foundation announced version 1. [5] It added the ability to centrally supply Docker, rkt and appc instances. [6]

On April 5, 2021, it was voted to move Mesos to the Apache Attic, [7] however the vote was cancelled two days later due to increased interest. [8]

Technology

Mesos uses Linux cgroups to provide isolation for CPU, memory, I/O and file system. [9] Mesos is comparable to Google's Borg scheduler, a platform used internally to manage and distribute Google's services. [10]

Apache Aurora
Developer(s) Apache Software Foundation
Final release
0.22.0 / December 12, 2019;4 years ago (2019-12-12) [11]
Repository Aurora Repository
Written in Java, Python
Type Mesos Framework
License Apache License 2.0
Website aurora.apache.org

Apache Aurora

Apache Aurora is a Mesos framework for both long-running services and cron jobs, originally developed by Twitter starting in 2010 and open sourced in late 2013. [12] It can scale to tens of thousands of servers, and holds many similarities to Borg [13] [14] including its rich domain-specific language (DSL) for configuring services. As of February 2020 the project was retired to the Attic. [15] A fork of the project was maintained by former members, hosted on GitHub under the name Aurora Scheduler. [16]

Chronos

Chronos is a distributed cron-like system which is elastic and capable of expressing dependencies between jobs. [17]

Marathon

Marathon is promoted for platform as a service or container orchestration system scaling to thousands of physical servers. It is fully REST-based and allows canary-style deployments and deployment topologies. It is written in the programming language Scala. [18]

Users

The social networking site Twitter began using Mesos and Apache Aurora in 2010, after Hindman gave a presentation to a group of Twitter engineers. [10]

Airbnb said in July 2013 that it uses Mesos to run data processing systems like Apache Hadoop and Apache Spark. [19]

The Internet auction website eBay stated in April 2014 that it used Mesos to run continuous integration on a per-developer basis. They accomplish this by using a custom Mesos plugin that allows developers to launch their own private Jenkins instance. [20]

In April 2015, it was announced that Apple service Siri is using its own Mesos framework called Jarvis. [21]

In August 2015, it was announced that Verizon selected Mesosphere's DC/OS, which is based on open source Apache Mesos, for data center service orchestration. [22]

In November 2015, Yelp announced they had been using Mesos and Marathon for a year and a half for production services. [23]

Commercial support

Software startup Mesosphere, Inc. sells the Datacenter Operating System, a distributed operating system, based on Apache Mesos. [24] In September 2015, Microsoft announced a commercial partnership with Mesosphere to build container scheduling and orchestration services for Microsoft Azure. [25] In October 2015, Oracle announced support for Mesos through Oracle Container Cloud Service. [26]

See also

Related Research Articles

Operating systems use lock managers to organise and serialise the access to resources. A distributed lock manager (DLM) runs in every machine in a cluster, with an identical copy of a cluster-wide lock database. In this way a DLM provides software applications which are distributed across a cluster on multiple machines with a means to synchronize their accesses to shared resources.

<span class="mw-page-title-main">LXC</span> Operating system-level virtualization for Linux

Linux Containers (LXC) is an operating-system-level virtualization method for running multiple isolated Linux systems (containers) on a control host using a single Linux kernel.

cgroups is a Linux kernel feature that limits, accounts for, and isolates the resource usage of a collection of processes.

<span class="mw-page-title-main">Ion Stoica</span> Romanian–American computer scientist

Ion Stoica is a Romanian–American computer scientist specializing in distributed systems, cloud computing and computer networking. He is a professor of computer science at the University of California, Berkeley and co-director of AMPLab. He co-founded Conviva and Databricks with other original developers of Apache Spark.

<span class="mw-page-title-main">OpenShift</span> Cloud computing software

OpenShift is a family of containerization software products developed by Red Hat. Its flagship product is the OpenShift Container Platform — a hybrid cloud platform as a service built around Linux containers orchestrated and managed by Kubernetes on a foundation of Red Hat Enterprise Linux. The family's other products provide this platform through different environments: OKD serves as the community-driven upstream, Several deployment methods are available including self-managed, cloud native under ROSA, ARO and RHOIC on AWS, Azure, and IBM Cloud respectively, OpenShift Online as software as a service, and OpenShift Dedicated as a managed service.

Docker is a set of platform as a service (PaaS) products that use OS-level virtualization to deliver software in packages called containers. The service has both free and premium tiers. The software that hosts the containers is called Docker Engine. It was first released in 2013 and is developed by Docker, Inc.

Matei Zaharia is a Romanian-Canadian computer scientist, educator and the creator of Apache Spark.

<span class="mw-page-title-main">Apache Spark</span> Open-source data analytics cluster computing framework

Apache Spark is an open-source unified analytics engine for large-scale data processing. Spark provides an interface for programming clusters with implicit data parallelism and fault tolerance. Originally developed at the University of California, Berkeley's AMPLab, the Spark codebase was later donated to the Apache Software Foundation, which has maintained it since.

Kubernetes is an open-source container orchestration system for automating software deployment, scaling, and management. Originally designed by Google, the project is now maintained by a worldwide community of contributors, and the trademark is held by the Cloud Native Computing Foundation.

<span class="mw-page-title-main">Borg (backup software)</span> Deduplicating backup program

Borg is deduplicating backup software for various Unix-like operating systems.

PerfKit Benchmarker is an open source benchmarking tool used to measure and compare cloud offerings. PerfKit Benchmarker is licensed under the Apache 2 license terms. PerfKit Benchmarker is a community effort involving over 500 participants including researchers, academic institutions and companies together with the originator, Google.

<span class="mw-page-title-main">Ali Ghodsi</span> Swedish computer scientist

Ali Ghodsi is a Persian computer scientist and entrepreneur specializing in distributed systems and big data. He is a co-founder and CEO of Databricks and an adjunct professor at UC Berkeley. He coauthored several influential papers, including Apache Mesos and Apache Spark SQL.

Wercker is a Docker-based continuous delivery platform that helps software developers build and deploy their applications and microservices. Using its command-line interface, developers can create Docker containers on their desktop, automate their build and deploy processes, testing them on their desktop, and then deploy them to various cloud platforms, ranging from Heroku to AWS and Rackspace. The command-line interface to Wercker has been open-sourced.

Reynold Xin is a computer scientist and engineer specializing in big data, distributed systems, and cloud computing. He is a co-founder and Chief Architect of Databricks. He is best known for his work on Apache Spark, a leading open-source Big Data project. He was designer and lead developer of the GraphX, Project Tungsten, and Structured Streaming components and he co-designed DataFrames, all of which are part of the core Apache Spark distribution; he also served as the release manager for Spark's 2.0 release.

<span class="mw-page-title-main">Singularity (software)</span> Free, cross-platform and open-source computer program

Singularity is a free and open-source computer program that performs operating-system-level virtualization also known as containerization.

Borg is a cluster manager used by Google. It led to widespread use of similar approaches, such as Docker and Kubernetes.

Haoyuan (H.Y.) Li is a computer scientist and entrepreneur specializing in distributed systems, big data, and cloud computing. He is best known for proposing Virtual Distributed File System (VDFS), and creating an open-source data orchestration system, Alluxio. He is the Founder, Chairman, and CEO of Alluxio, Inc, a company commercializing the Alluxio Data Orchestration Technology. He is also an adjunct professor at Peking University. He is a frequent speaker on the topic of AI, Big Data, Cloud Computing, and Open Source at conferences.

HammerDB is an open source database benchmarking application developed by Steve Shaw. HammerDB supports databases such as Oracle, SQL Server, Db2, MySQL and MariaDB. HammerDB is written in TCL and C, and is licensed under the GPL v3.

Dominant resource fairness (DRF) is a rule for fair division. It is particularly useful for dividing computing resources in among users in cloud computing environments, where each user may require a different combination of resources. DRF was presented by Ali Ghodsi, Matei Zaharia, Benjamin Hindman, Andy Konwinski, Scott Shenker and Ion Stoica in 2011.

References

  1. "ASF Git Repos - mesos.git/tag" . Retrieved 27 September 2022.
  2. Zaharia, Matei. "HUG Meetup August 2010: Mesos: A Flexible Cluster Resource manager - Part 1". youtube.com. Retrieved 13 January 2015.
  3. "A Common Substrate for Cluster Computing" (PDF).
  4. Hindman, Benjamin; Konwinski, Andy; Zaharia, Matei; Ghodsi, Ali; Joseph, Anthony; Katz, Randy; Shenker, Scott; Stoica, Ion (2011). "Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center" (PDF). NSDI. 11: 22-22. Retrieved 12 January 2015.
  5. "The Apache Software Foundation Announces Apache Mesos v1.0". Press release. July 27, 2016. Retrieved February 24, 2017.
  6. "Mesos 1.0 brings a new container runtime and more third party integrations". July 27, 2016.
  7. "[VOTE] Move Apache Mesos to Attic". lists.apache.org. Archived from the original on 2021-04-06. Retrieved 2021-04-07.
  8. "Re: [VOTE] Move Apache Mesos to Attic". lists.apache.org. Archived from the original on 2021-04-09. Retrieved 2021-04-09.
  9. Bappalige, Sachin P. (2014-09-15). "Open-Source Datacenter Computing with Apache Mesos". OpenSource.com . Red Hat . Retrieved 2016-12-10.
  10. 1 2 Metz, Cade. "Return of the Borg: How Twitter Rebuilt Google's Secret Weapon". Wired. Retrieved 12 January 2015.
  11. "Apache Aurora Blog" . Retrieved 16 March 2021.
  12. "All about Apache Aurora". Twitter . Retrieved 20 May 2015.
  13. "Large-scale cluster management at Google with Borg" (PDF). Retrieved 20 May 2015.
  14. "Twitter's Aurora and How It Relates to Google's Borg". 18 February 2015. Retrieved 20 May 2015.
  15. "Apache Aurora - Apache Attic". attic.apache.org. Retrieved 2021-02-18.
  16. "Aurora Scheduler". GitHub. Retrieved 2023-04-02.
  17. "Chronos". GitHub.com. GitHub . Retrieved 30 March 2015.
  18. "Marathon". Mesosphere.GitHub.io. Mesosphere. 2014. Retrieved 30 March 2015.
  19. Harris, Derrick. "Airbnb is engineering itself into a data-driven company". gigaom.com. Retrieved 12 January 2015.
  20. The eBay PAAS Team (4 April 2014). "Delivering eBay's CI Solution with Apache Mesos - Part I". EbayTechBlog.com. eBay. Retrieved 12 January 2015.
  21. Harris, Derrick (2015-04-23). "Apple Details How It Rebuilt Siri on Mesos". Mesosphere.com. Mesosphere. Archived from the original on 2015-04-29. Retrieved 2015-04-27.
  22. "Verizon selects Mesosphere DCOS as nationwide platform for data center service orchestration". Verizon. 21 August 2015. Retrieved 21 August 2015.
  23. "Introducing PaaSTA: An Open, Distributed, Platform as a Service". engineeringblog.yelp.com. Retrieved 2016-07-12.
  24. "The Mesosphere DCOS". mesosphere.com. Retrieved 13 January 2015.
  25. Mary Jo Foley (September 29, 2015). "New Azure Container Service to bring together Mesos, Docker and Azure cloud". ZDNet .
  26. "Oracle Updates Oracle Cloud Infrastructure Services". oracle.com. Retrieved 2018-02-06.