PerfKit Benchmarker is an open source benchmarking tool used to measure and compare cloud offerings. PerfKit Benchmarker is licensed under the Apache 2 license terms. PerfKit Benchmarker is a community effort involving over 500 participants including researchers, academic institutions and companies together with the originator, Google.
PerfKit Benchmarker (PKB) is a community effort to deliver a repeatable, consistent, and open way of measuring Cloud Performance. It supports a growing list of cloud providers including: Alibaba Cloud, Amazon Web Services, CloudStack, DigitalOcean, Google Cloud Platform, Kubernetes, Microsoft Azure, OpenStack, Rackspace, IBM Bluemix (Softlayer). In addition to Cloud Providers to supports container orchestration including Kubernetes and Mesos and local "static" workstations and clusters of computers .
The goal is to create an open source living benchmark [framework] that represents how Cloud developers are building applications, evaluating Cloud alternatives, learning how to architect applications for each cloud. Living because it will change and morph quickly as developers change.
PerfKit Benchmarker measures the end to end time to provision resources in the cloud, in addition to reporting on the most standard metrics of peak performance, e.g.: latency, throughput, time-to-complete, IOPS. PerfKit Benchmarker reduces the complexity in running benchmarks on supported cloud providers by unified and simple commands. It's designed to operate via vendor provided command line tools.
PerfKit Benchmarker contains a canonical set of public benchmarks. All benchmarks are running with default/initial state and configuration (Not tuned to in favor of any providers). This provides a way to benchmark across cloud platforms, while getting a transparent view of application throughput, latency, variance, and overhead. [1]
PerfKit Benchmarker (PKB) was started by Anthony F. Voellm, Alain Hamel, and Eric Hankland at Google in 2014. Once an initial "alpha" was in place Anthony F. Voellm and Ivan Santa Maria Filho built a community including ARM, Broadcom, Canonical, CenturyLink, Cisco, CloudHarmony, CloudSpectator, EcoCloud@EPFL, Intel, Mellanox, Microsoft, Qualcomm Technologies, Inc., Rackspace, Red Hat, Tradeworx Inc., and Thesys Technologies LLC.
This community worked together behind the scenes in a private GitHub project to create an open way to measure cloud performance. This community released the first public "beta" was released on February 11, 2015, and announced in a blog post at which point the GitHub project was open to everyone. After almost a year and with large adaption (600+ participants on GitHub) the V1.0.0 was released along with a detailed architectural design on December 10, 2015.
A list of available benchmarks from PerfKitBenchmarker: (The latest set of benchmarks can be found at GitHub readme file. [2] )
Big Data / IoT | High Performance Computing Scientific Computing | Simulation | Web benchmarks | |
---|---|---|---|---|
Workloads | - Aerospike YCSB - Cassandra YCSB - Hadoop Terasort - HBase YCSB - MongoDB YCSB - Redis YCSB | - HPCC - Scimark2 | - OLDIsim - etcd | - EPFL CS Web Search - Tomcat |
Storage benchmarks | CPU benchmarks | Network benchmarks | System | |
---|---|---|---|---|
micro-benchmarks | - Bonnie - File Copy - Fio - Google Cloud BigTable - Object Storage - Synthetic Storage - Sysbench OLTP | - Coremark | - Iperf - Mesh Network - Netperf - Ping | - Cluster Boot |
Since Google open sourced the PerfKitBenchmarker, it became a community effort from over 30 leading researchers, academic schools and industry companies. Those organizations include: ARM, Broadcom, Canonical, CenturyLink, Cisco, CloudHarmony, Cloud Spectator, EcoCloud@EPFL, Intel, Mellanox, Microsoft, Qualcomm Technologies, Rackspace, Red Hat, and Thesys Technologies. In addition, Stanford and MIT are leading quarterly discussions on default benchmarks and settings proposed by the community. EcoCloud@EPFL is integrating CloudSuite into PerfKit Benchmarker.
Example run on Google Cloud Platform
$ ./pkb.py --cloud=GCP --project=<GCP project ID> --benchmarks=iperf --machine_type=f1-micro
Example run on AWS
$ ./pkb.py --cloud=AWS --benchmarks=iperf --machine_type=t1.micro
Example run on Azure
$ ./pkb.py --cloud=Azure --machine_type=ExtraSmall --benchmarks=iperf
Example run on Rackspace
$ ./pkb.py --cloud=Rackspace --machine_type=standard1 --benchmarks=iperf
Example run on a local machine
$ ./pkb.py --stack_vm_file=local_config.json --benchmarks=iperf
RightScale was a company that sold software as a service for cloud computing management for multiple providers. The company was based in Santa Barbara, California. It was acquired by Flexera Software in 2018.
Microsoft Azure, often referred to as Azure, is a cloud computing platform run by Microsoft. It offers access, management, and the development of applications and services through global data centers. It also provides a range of capabilities, including software as a service (SaaS), platform as a service (PaaS), and infrastructure as a service (IaaS). Microsoft Azure supports many programming languages, tools, and frameworks, including Microsoft-specific and third-party software and systems.
Cloud Foundry is an open source, multi-cloud application platform as a service (PaaS) governed by the Cloud Foundry Foundation, a 501(c)(6) organization.
A cloud database is a database that typically runs on a cloud computing platform and access to the database is provided as-a-service. There are two common deployment models: users can run databases on the cloud independently, using a virtual machine image, or they can purchase access to a database service, maintained by a cloud database provider. Of the databases available on the cloud, some are SQL-based and some use a NoSQL data model.
OpenShift is a family of containerization software products developed by Red Hat. Its flagship product is the OpenShift Container Platform — a hybrid cloud platform as a service built around Linux containers orchestrated and managed by Kubernetes on a foundation of Red Hat Enterprise Linux. The family's other products provide this platform through different environments: OKD serves as the community-driven upstream, Several deployment methods are available including self-managed, cloud native under ROSA, ARO and RHOIC on AWS, Azure, and IBM Cloud respectively, OpenShift Online as software as a service, and OpenShift Dedicated as a managed service.
Plotly is a technical computing company headquartered in Montreal, Quebec, that develops online data analytics and visualization tools. Plotly provides online graphing, analytics, and statistics tools for individuals and collaboration, as well as scientific graphing libraries for Python, R, MATLAB, Perl, Julia, Arduino, JavaScript and REST.
Google Cloud Platform (GCP), offered by Google, is a suite of cloud computing services that provides a series of modular cloud services including computing, data storage, data analytics and machine learning, alongside a set of management tools. It runs on the same infrastructure that Google uses internally for its end-user products, such as Google Search, Gmail, and Google Docs, according to Verma, et.al. Registration requires a credit card or bank account details.
ZeroVM is an open source light-weight virtualization and sandboxing technology. It virtualizes a single process using the Google Native Client platform. Since only a single process is virtualized, the startup overhead is in the order of 5 ms.
DigitalOcean Holdings, Inc. is an American multinational technology company and cloud service provider. The company is headquartered in New York City, New York, US, with 15 globally distributed data centers. DigitalOcean provides developers, startups, and SMBs with cloud infrastructure-as-a-service platforms.
Kubernetes is an open-source container orchestration system for automating software deployment, scaling, and management. Originally designed by Google, the project is now maintained by the Cloud Native Computing Foundation.
Mirantis Inc. is a Campbell, California, based B2B open source cloud computing software and services company. Its primary container and cloud management products, part of the Mirantis Cloud Native Platform suite of products, are Mirantis Container Cloud and Mirantis Kubernetes Engine. The company focuses on the development and support of container and cloud infrastructure management platforms based on Kubernetes and OpenStack. The company was founded in 1999 by Alex Freedland and Boris Renski. It was one of the founding members of the OpenStack Foundation, a non-profit corporate entity established in September, 2012 to promote OpenStack software and its community. Mirantis has been an active member of the Cloud Native Computing Foundation since 2016.
The Serverless Framework is a free and open-source web framework written using Node.js. Serverless is the first framework developed for building applications on AWS Lambda, a serverless computing platform provided by Amazon as a part of Amazon Web Services. Currently, applications developed with Serverless can be deployed to other function as a service providers, including Microsoft Azure with Azure Functions, IBM Bluemix with IBM Cloud Functions based on Apache OpenWhisk, Google Cloud using Google Cloud Functions, Oracle Cloud using Oracle Fn, Kubeless based on Kubernetes, Spotinst and Webtask by Auth0.
The OpenBMC project is a Linux Foundation collaborative open-source project that produces an open source implementation of the baseboard management controllers (BMC) firmware stack. OpenBMC is a Linux distribution for BMCs meant to work across heterogeneous systems that include enterprise, high-performance computing (HPC), telecommunications, and cloud-scale data centers.
Netlify is a remote-first cloud computing company that offers a development platform that includes build, deploy, and serverless backend services for web applications and dynamic websites. The platform is built on open web standards, making it possible to integrate build tools, web frameworks, APIs, and various web technologies into a unified developer workflow.
CloudBolt is a hybrid cloud management platform developed by CloudBolt Software for deploying and managing virtual machines (VMs), applications, and other IT resources, both in public clouds and in private data centers.
Kubeflow is an open-source platform for machine learning and MLOps on Kubernetes introduced by Google. The different stages in a typical machine learning lifecycle are represented with different software components in Kubeflow, including model development (Kubeflow Notebooks), model training (Kubeflow Pipelines, Kubeflow Training Operator), model serving (KServe), and automated machine learning (Katib).
IBM Cloud is a set of cloud computing services for business offered by the information technology company IBM.
Teleport is an open-source tool for providing zero trust access to servers and cloud applications using SSH, Kubernetes and HTTPS. It can eliminate the need for VPNs by providing a single gateway to access computing infrastructure via SSH, Kubernetes clusters, and cloud applications via a built-in proxy.
The Software for Open Networking in the Cloud or alternatively abbreviated and stylized as SONiC, is a free and open source network operating system based on Linux. It was originally developed by Microsoft and the Open Compute Project. In 2022, Microsoft ceded oversight of the project to the Linux Foundation, who will continue to work with the Open Compute Project for continued ecosystem and developer growth. SONiC includes the networking software components necessary for a fully functional L3 device and was designed to meet the requirements of a cloud data center. It allows cloud operators to share the same software stack across hardware from different switch vendors and works on over 100 different platforms. There are multiple companies offering enterprise service and support for SONiC,.
CBL-Mariner is a free and open-source Linux distribution that Microsoft has developed. It is the base container OS for Microsoft Azure services and the graphical component of WSL 2.