DevOps Research and Assessment

Last updated
The DORA team are part of Google Cloud Kubernetes talk at Google Cloud Summit.jpg
The DORA team are part of Google Cloud

DevOps Research and Assessment (abbreviated to DORA) is a team that is part of Google Cloud that engages in opinion polling of software engineers to conduct research for the DevOps movement. [1]

Contents

The DORA team was founded by Nicole Forsgren, Jez Humble and Gene Kim. [2] [3] and conducted research for the DevOps company Puppet and later became an independent team (with Puppet continuing to produce reports by a new team). [4] [5]

Whilst the founding members have departed, the DORA team continue to publish research in the form of annual State of DevOps Reports. [6]

State of DevOps Reports

The DORA team began publishing State of DevOps Reports in 2013. [7] [8] The latest DORA State of DevOps Report published in 2023 found culture and a customer centric focus key to success, whilst AI was providing limited benefits. [9] [10]

DORA Four Key Metrics

For the purposes of their research, Four Key Metrics, sometimes referred to as DORA Metrics, are used to assess the performance of teams. [11] [12]

The four metrics are as follows: [13] [14]

Using these performance measures, the team are able to assess how practices (like outsourcing) and risk factors impact performance metrics for an engineering team. [15] [16] These metrics can be crudely measured using psychometrics [17] or using commercial services. [18]

Limitations

These metrics have been used by organisations to evaluate team-by-team performance, a use-case which the DORA team issued a warning against in October 2023. [1] [19]

Some professionals have argued that using the DORA Four Key Metrics as a target within engineering teams encourages focus on wrong incentives. [20] [21] For example; James Walker, CEO at Curiosity Software, has argued the "metrics aren’t a definitive route to DevOps success" and challenges in using them for team comparisons. [22]

Research conducted by the computer scientist Junade Ali and the British polling firm Survation found that both software engineers (when building software systems) and public perception (when using software systems) found other factors mattered significantly more than the outcome measures which were treated as the "Four Key Metrics" (which ultimately measure the speed of resolving issues and the speed of fixing bugs, and are used to create the findings in the book), and risk and reward appetite varies from sector-to-sector. [23] [24] [25] [26] [27]

Ali has also criticised the research on the basis that reputable opinion polling firms who comply with the rules of organisations like the British Polling Council should publish their full results and raw data tables, which the DORA team did not do - and additionally that the sponsors of the polling (Google Cloud and previously Puppet) create products which have a vested interest in having software engineers deliver faster (despite research indicating high levels of burnout amongst software engineers), which the results of the research ultimately supported. Despite the authors arguing that speed of delivery and software quality go hand-in-hand, Ali has offered several counter-examples; including the comparatively high quality of aviation software despite infrequent changes, contrasted with rapid application development being pioneered in the software that resulted in the British Post Office scandal and agile software development being used in the software responsible for the 2009–2011 Toyota vehicle recalls. [28] [29] [30]

The software developer Bryan Finster has also discussed how, as correlation does not imply causation, organisations who are considered "high performing" in the research are not high performing because they focussed on the DORA metrics, but instead focussed on delivering value to users and arguing the metrics should be used as "trailing indicators for poor health, not indicators everything is going well". [31] [32]

Accelerate (book)

Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations
Accelerate (book).jpg
Author Nicole Forsgren, Jez Humble, Gene Kim
LanguageEnglish
Subject DevOps
PublisherIT Revolution
Publication date
2018
Publication placeUnited States
Pages156 (paperback)
Awards Shingo Prize [33] [34]
ISBN 9781942788331
OCLC 1035781633

Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations is a software engineering book co-authored by Nicole Forsgren, Jez Humble and Gene Kim from their time in the DORA team. [35] The book explores how software development teams using Lean Software and DevOps can measure their performance and the performance of software engineering teams impacts the overall performance of an organization. [36] [14]

The book discusses their research conducted as part of the DORA team for the annual State of DevOps Reports. In total, the authors considered 23,000 data points from a variety of companies of various different sizes (from start-up to enterprises), for-profit and not-for-profit and both those with legacy systems and those with modern systems. [37] [38] [39]

24 Key Capabilities

The authors outline 24 practices to improve software delivery which they refer to as "key capabilities" and group them into five categories. [40]

Continuous Delivery

  • Use Version Control for all Production Artifacts
  • Automate Your Deployment Process
  • Implement Continuous Integration
  • Use Trunk-Based Development Methods
  • Implement Test Automation
  • Support Test Data Management
  • Shift Left on Security
  • Implement Continuous Delivery (CD)

Architecture

  • Use a Loosely Coupled Architecture
  • Architect for Empowered Teams

Product and Process

  • Gather and Implement Customer Feedback
  • Make the Flow of Work Visible through the Value Stream
  • Work in Small Batches
  • Foster and Enable Team Experimentation

Lean Management and Monitoring

  • Have a Lightweight Change Approval Processes
  • Monitor across Application and Infrastructure to Inform Business Decisions
  • Check System Health Proactively
  • Improve Processes and Manage Work with Work-In-Process (WIP) Limits
  • Visualize Work to Monitor Quality and Communicate throughout the Team

Cultural

  • Support a Generative Culture
  • Encourage and Support Learning
  • Support and Facilitate Collaboration among Teams
  • Provide Resources and Tools that Make Work Meaningful
  • Support or Embody Transformational Leadership

Related Research Articles

Release engineering, frequently abbreviated as RE or as the clipped compound Releng, is a sub-discipline in software engineering concerned with the compilation, assembly, and delivery of source code into finished products or other software components. Associated with the software release life cycle, it was said by Boris Debic of Google Inc. that release engineering is to software engineering as manufacturing is to an industrial process:

Release engineering is the difference between manufacturing software in small teams or startups and manufacturing software in an industrial way that is repeatable, gives predictable results, and scales well. These industrial style practices not only contribute to the growth of a company but also are key factors in enabling growth.

A change-advisory board (CAB) delivers support to a change-management team by advising on requested changes, assisting in the assessment and prioritization of changes. This body is generally made up of IT and Business representatives that include: a change manager, user managers and groups, product owners, technical experts, and possible third parties and customers.

Progress Chef is a configuration management tool written in Ruby and Erlang. It uses a pure-Ruby, domain-specific language (DSL) for writing system configuration "recipes". Chef is used to streamline the task of configuring and maintaining a company's servers, and can integrate with cloud-based platforms such as Amazon EC2, Google Cloud Platform, Oracle Cloud, OpenStack, IBM Cloud, Microsoft Azure, and Rackspace to automatically provision and configure new machines. Chef contains solutions for both small and large scale systems.

<span class="mw-page-title-main">Release management</span> Process of software building

Release management is the process of managing, planning, scheduling and controlling a software build through different stages and environments; it includes testing and deploying software releases.

DevOps is a methodology in the software development and IT industry. Used as a set of practices and tools, DevOps integrates and automates the work of software development (Dev) and IT operations (Ops) as a means for improving and shortening the systems development life cycle. DevOps is complementary to agile software development; several DevOps aspects came from the agile way of working.

Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate. Continuous testing was originally proposed as a way of reducing waiting time for feedback to developers by introducing development environment-triggered tests as well as more traditional developer/tester-triggered tests.

New Relic, Inc. is an American web tracking and analytics company based in San Francisco. The company's cloud-based software allows websites and mobile apps to track user interactions and service operators' software and hardware performance.

Continuous delivery (CD) is a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time and following a pipeline through a "production-like environment", without doing so manually. It aims at building, testing, and releasing software with greater speed and frequency. The approach helps reduce the cost, time, and risk of delivering changes by allowing for more incremental updates to applications in production. A straightforward and repeatable deployment process is important for continuous delivery.

<span class="mw-page-title-main">Jesse Robbins</span> American entrepreneur

Jesse Robbins is an American technology entrepreneur, investor, and firefighter notable for his pioneering work in Cloud computing, role in creating DevOps/Chaos Engineering, and efforts to improve emergency management.

Application-release automation (ARA) refers to the process of packaging and deploying an application or update of an application from development, across various environments, and ultimately to production. ARA solutions must combine the capabilities of deployment automation, environment management and modeling, and release coordination.

<span class="mw-page-title-main">Dynatrace</span> American technology company

Dynatrace, Inc. is a global technology company that provides a software observability platform based on artificial intelligence (AI) and automation. Dynatrace technologies are used to monitor, analyze, and optimize application performance, software development and security practices, IT infrastructure, and user experience for businesses and government agencies throughout the world.

Infrastructure as code (IaC) is the process of managing and provisioning computer data center resources through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools. The IT infrastructure managed by this process comprises both physical equipment, such as bare-metal servers, as well as virtual machines, and associated configuration resources. The definitions may be in a version control system, rather than maintaining the code through manual processes. The code in the definition files may use either scripts or declarative definitions, but IaC more often employs declarative approaches.

<span class="mw-page-title-main">DevOps toolchain</span> DevOps toolchain release package.

A DevOps toolchain is a set or combination of tools that aid in the delivery, development, and management of software applications throughout the systems development life cycle, as coordinated by an organisation that uses DevOps practices.

Site reliability engineering (SRE) is a set of principles and practices that applies aspects of software engineering to IT infrastructure and operations. SRE claims to create highly reliable and scalable software systems. Although they are closely related, SRE is slightly different from DevOps.

Continuous configuration automation (CCA) is the methodology or process of automating the deployment and configuration of settings and software for both physical and virtual data center equipment.

<span class="mw-page-title-main">Grafana</span> Platform for data analytics and monitoring

Grafana is a multi-platform open source analytics and interactive visualization web application. It can produce charts, graphs, and alerts for the web when connected to supported data sources.

Nicole Forsgren is an American technology executive, IT impact expert, and author. In 2020 she was named vice-president of Research & Strategy at Microsoft's GitHub and more recently Partner at Microsoft Research. She is coauthor of Accelerate: The Science of Lean Software and DevOps which won the Shingo Research and Professional Publication Award in 2019.

<span class="mw-page-title-main">Junade Ali</span> British computer scientist and cybersecurity researcher (born 1996)

Junade Ali is a British computer scientist known for research in cybersecurity.

In software engineering, more specifically in distributed computing, observability is the ability to collect data about programs' execution, modules' internal states, and the communication among components. To improve observability, software engineers use a wide range of logging and tracing techniques to gather telemetry information, and tools to analyze and use it. Observability is foundational to site reliability engineering, as it is the first step in triaging a service outage. One of the goals of observability is to minimize the amount of prior knowledge needed to debug an issue.

Buildkite is a continuous integration (CI) and continuous delivery (CD) platform used in DevOps and company was founded in September 2013. Companies using Buildkite include Wayfair, Shopify, Slack, Twilo, Canva, Pinterest, Envato, MYOB and Lyft.

References

  1. 1 2 Pariseau, Beth. "Google's DORA DevOps report warns against metrics misuse - TechTarget". Software Quality. TechTarget. Retrieved 20 May 2024.
  2. Chan, Rosalie. "GitHub just hired a top DevOps researcher away from Google Cloud". Business Insider. Retrieved 20 May 2024.
  3. Humble, Jez (2 February 2019). "DORA's Journey: An Exploration". Medium. Retrieved 20 May 2024.
  4. "2023 State of DevOps". Splunk-Blogs. Retrieved 5 January 2024.
  5. Anderson, Tim (6 October 2022). "State of DevOps report 2022: for secure software, team culture counts more than technology". Devclass. Retrieved 5 January 2024.
  6. Vizard, Mike (5 October 2023). "Latest DORA Report Highlights Importance of DevOps Culture". DevOps.com. Retrieved 20 May 2024.
  7. "History of DevOps Reports - Puppet by Perforce". www.puppet.com. Retrieved 20 May 2024.
  8. McLellan, Charles. "DevOps: What is it, and how can it help your business?". ZDNET. Retrieved 20 May 2024.
  9. Kerner, Sean Michael (6 October 2023). "Google DORA 2023 State of DevOps Report Finds Culture, User Focus Key to Success". ITPro Today: IT News, How-Tos, Trends, Case Studies, Career Tips, More. Retrieved 20 May 2024.
  10. Anderson, Tim (10 October 2023). "Google-sponsored DevOps survey shows limited benefits of AI, return of elite teams". Devclass. Retrieved 20 May 2024.
  11. Mellifera, Nočnica (12 February 2024). "The Wrong Way to Use DORA Metrics". The New Stack. Retrieved 20 May 2024.
  12. Etkin, Dylan (1 August 2023). "Why software developers prefer DORA metrics". InfoWorld. Retrieved 20 May 2024.
  13. "Are you an Elite DevOps performer? Find out with the Four Keys Project". Google Cloud Blog. Retrieved 5 April 2021.
  14. 1 2 Drugeot, Constance (30 March 2021). "Why is a DevOps assessment important, and how to do it right?". DevOps Online. Retrieved 5 April 2021.
  15. Johnson, Emily. "Study: DevOps Outsourcing Can Cost You". InformationWeek. Retrieved 5 April 2021.
  16. Doddi, Lakshmi (1 April 2021). "Book Summary: Accelerate — by Nicole Forsgren, Gene Kim, Jez Humble". Medium. Retrieved 5 April 2021.
  17. Linders, Ben; Humble, Jez. "Q&A on the Book Accelerate: Building and Scaling High Performance Technology Organizations". InfoQ. Retrieved 5 April 2021.
  18. "What are DORA Metrics and How Do They Improve Dev Teams? | Dev Interrupted Powered by LinearB". linearb.io. 17 June 2021. Retrieved 20 May 2024.
  19. McMurray, Alex (8 September 2023). "Ex-Finance developers mock McKinsey's monitoring metrics". eFinancialCareers. Retrieved 20 May 2024.
  20. "Perverse incentives and DORA metrics". The Disco Blog. 19 February 2024. Retrieved 20 May 2024.
  21. "Want productive engineers? Start with trust, not metrics - Multitudes". www.multitudes.co. Retrieved 20 May 2024.
  22. "Everything wrong with DORA Metrics - Aviator Blog". 18 January 2023. Retrieved 20 May 2024.
  23. "75% of software engineers faced retaliation the last time they reported wrongdoing - ETHRWorldSEA". ETHRWorld.com. Retrieved 5 January 2024.
  24. Saran, Cliff. "Software engineers worry about speaking out". ComputerWeekly.com. Retrieved 5 January 2024.
  25. Woollacott, Emma (21 November 2023). "Workplace retaliation is stopping software engineers from speaking out over malpractice". ITPro. Retrieved 5 January 2024.
  26. Mitchell, Sean. "Unethical activities found prevalent in UK software engineering industry". IT Brief UK. Retrieved 5 January 2024.
  27. Turner, Graham (20 November 2023). "Report: Software Engineers Face Backlash for Reporting Wrongdoing". Digit.
  28. "Overview of the Rules of Disclosure". British Polling Council. Retrieved 5 January 2024.
  29. Hughes, Owen. "Developers are exhausted. Here's what needs to change". zdnet.com. Retrieved 5 January 2024.
  30. Ali, Junade (8 April 2024). How to Protect Yourself from Killer Computers: From the Post Office Scandal to Artificial Intelligence. Engprax Ltd. ISBN   978-1068605710.
  31. Finster, Bryan (29 March 2021). "5 Minute DevOps: The Metrics are Lying". Medium. Retrieved 5 January 2024.
  32. Cummins, Holly. "Holly Cummins on X". X.com. Retrieved 5 January 2024.
  33. "IT Revolution Announces 'Accelerate' Book Wins Shingo Publication Award". markets.businessinsider.com. Retrieved 5 April 2021.
  34. "Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performance Technology by Nicole Forsgren, PhD, Jez Humble, Gene Kim, Steve Bell, and Karen Whitley Bell Receives Shingo Publication Award". Shingo Institute. 4 February 2020. Retrieved 5 April 2021.
  35. "The 2018 InfoQ Editors' Recommended Reading List: Part One". InfoQ. Retrieved 5 April 2021.
  36. Bloomberg, Jason. "Digital Influencer Nicole Forsgren: Bringing Science to DevOps". Forbes. Retrieved 5 April 2021.
  37. Overby, Stephanie. "7 takeaways to "Accelerate" your DevOps". TechBeacon. Retrieved 5 April 2021.
  38. Linders, Ben; Humble, Jez. "Q&A on the Book Accelerate: Building and Scaling High Performance Technology Organizations". InfoQ. Retrieved 5 April 2021.
  39. Mangot, Dave (17 March 2021). "Get good at delivering software". CIO. Retrieved 5 April 2021.
  40. "24 Key Capabilities to Drive Improvement in Software Delivery".