Continuous integration

Last updated

Sketch of flow diagram for continuous integration Continuous Integration.jpg
Sketch of flow diagram for continuous integration

Continuous integration (CI) is the practice of integrating source code changes frequently and ensuring that the integrated codebase is in a workable state.

Contents

Typically, developers merge changes to an integration branch, and an automated system builds and tests the software system. [1] Often, the automated process runs on each commit or runs on a schedule such as once a day.

Grady Booch first proposed the term CI in 1991, [2] although he did not advocate integrating multiple times a day, but later, CI came to include that aspect. [3]

History

The earliest known work (1989) on continuous integration was the Infuse environment developed by G. E. Kaiser, D. E. Perry, and W. M. Schell. [4]

In 1994, Grady Booch used the phrase continuous integration in Object-Oriented Analysis and Design with Applications (2nd edition) [5] to explain how, when developing using micro processes, "internal releases represent a sort of continuous integration of the system, and exist to force closure of the micro process".

In 1997, Kent Beck and Ron Jeffries invented extreme programming (XP) while on the Chrysler Comprehensive Compensation System project, including continuous integration. [1] [ self-published source ] Beck published about continuous integration in 1998, emphasising the importance of face-to-face communication over technological support. [6] In 1999, Beck elaborated more in his first full book on Extreme Programming. [7] CruiseControl, one of the first open-source CI tools, [8] [ self-published source ] was released in 2001.

In 2010, Timothy Fitz published an article detailing how IMVU's engineering team had built and been using the first practical CI system. While his post was originally met with skepticism, it quickly caught on and found widespread adoption [9] as part of the lean software development methodology, also based on IMVU.

Practices

The core activities of CI are developers co-locate code changes in a shared, integration area frequently and that the resulting integrated codebase is verified for correctness. The first part generally involves merging changes to a common version control branch. The second part generally involves automated processes including: building, testing and many other processes.

Typically, a server builds from the integration area frequently; i.e. after each commit or periodically like once a day. The server may perform quality control checks such as running unit tests [10] and collect software quality metrics via processes such as static analysis and performance testing.

This section lists best practices from practitioners for other practices that enhance CI.

Build automation

Build automation is a best practice. [11] [12]

Atomic commits

CI requires the version control system to support atomic commits; i.e., all of a developer's changes are handled as a single commit.

Committing changes

When making a code change, a developer creates a branch that is a copy of the current codebase. As other changes are committed to the repository, this copy diverges from the latest version.

The longer development continues on a branch without merging to the integration branch, the greater the risk of multiple integration conflicts [13] and failures when the developer branch is eventually merged back. When developers submit code to the repository they must first update their code to reflect the changes in the repository since they took their copy. The more changes the repository contains, the more work developers must do before submitting their own changes.

Eventually, the repository may become so different from the developers' baselines that they enter what is sometimes referred to as "merge hell", or "integration hell", [14] where the time it takes to integrate exceeds the time it took to make their original changes. [15]

Testing locally

Proponents of CI suggest that developers should use test-driven development and to ensure that all unit tests pass locally before committing to the integration branch so that one developer's work does not break another developer's copy.

Incomplete features can be disabled before committing, using feature toggles.

Continuous delivery and continuous deployment

Continuous delivery ensures the software checked in on an integration branch is always in a state that can be deployed to users, and continuous deployment automates the deployment process.

Continuous delivery and continuous deployment are often performed in conjunction with CI and together form a CI/CD pipeline.

Version control

Proponents of CI recommend storing all files and information needed for building in version control, (for git a repository); that the system should be buildable from a fresh checkout and not require additional dependencies.

Martin Fowler recommends that all developers commit to the same integration branch. [16]

Automate the build

Build automation tools automate building.

Proponents of CI recommend that a single command should have the capability of building the system.

Automation often includes automating the integration, which often includes deployment into a production-like environment. In many cases, the build script not only compiles binaries but also generates documentation, website pages, statistics and distribution media (such as Debian DEB, Red Hat RPM or Windows MSI files).

Commit frequently

Developers can reduce the effort of resolving conflicting changes by synchronizing changes with each other frequently; at least daily. Checking in a week's worth of work risks conflict both in likelihood of occurrence and complexity to resolve. Relatively small conflicts are significantly easier to resolve than larger ones. Integrating (committing) changes at least once a day is considered good practice, and more often better. [17]

Daily build

Building daily, if not more often, is generally recommended.[ citation needed ]

Every commit should be built

The system should build commits to the current working version to verify that they integrate correctly. A common practice is to use Automated Continuous Integration, although this may be done manually. Automated Continuous Integration employs a continuous integration server or daemon to monitor the revision control system for changes, then automatically run the build process.

Every bug-fix commit should come with a test case

When fixing a bug, it is a good practice to push a test case that reproduces the bug. This avoids the fix to be reverted, and the bug to reappear, which is known as a regression.

Keep the build fast

The build needs to complete rapidly so that if there is a problem with integration, it is quickly identified.

Test in a clone of the production environment

Having a test environment can lead to failures in tested systems when they deploy in the production environment because the production environment may differ from the test environment in a significant way. However, building a replica of a production environment is cost-prohibitive. Instead, the test environment or a separate pre-production environment ("staging") should be built to be a scalable version of the production environment to alleviate costs while maintaining technology stack composition and nuances. Within these test environments, service virtualisation is commonly used to obtain on-demand access to dependencies (e.g., APIs, third-party applications, services, mainframes, etc.) that are beyond the team's control, still evolving, or too complex to configure in a virtual test lab.

Make it easy to get the latest deliverables

Making builds readily available to stakeholders and testers can reduce the amount of rework necessary when rebuilding a feature that doesn't meet requirements. Additionally, early testing reduces the chances that defects survive until deployment. Finding errors earlier can reduce the amount of work necessary to resolve them.

All programmers should start the day by updating the project from the repository. That way, they will all stay up to date.

Everyone can see the results of the latest build

It should be easy to find out whether the build breaks and, if so, who made the relevant change and what that change was.

Automate deployment

Most CI systems allow the running of scripts after a build finishes. In most situations, it is possible to write a script to deploy the application to a live test server that everyone can look at. A further advance in this way of thinking is continuous deployment, which calls for the software to be deployed directly into production, often with additional automation to prevent defects or regressions. [18] [19]

Benefits

CI benefits include:

Risks

Risks of CI include:

See also

Related Research Articles

Version control is the software engineering practice of controlling, organizing, and tracking different versions in history of computer files; primarily source code text files, but generally any type of file.

Test-driven development (TDD) is a way of writing code that involves writing an automated unit-level test case that fails, then writing just enough code to make the test pass, then refactoring both the test code and the production code, then repeating with another new test case.

In software testing, test automation is the use of software separate from the software being tested to control the execution of tests and the comparison of actual outcomes with predicted outcomes. Test automation can automate some repetitive but necessary tasks in a formalized testing process already in place, or perform additional testing that would be difficult to do manually. Test automation is critical for continuous delivery and continuous testing.

In software development, distributed version control is a form of version control in which the complete codebase, including its full history, is mirrored on every developer's computer. Compared to centralized version control, this enables automatic management branching and merging, speeds up most operations, improves the ability to work offline, and does not rely on a single location for backups. Git, the world's most popular version control system, is a distributed version control system.

Release engineering, frequently abbreviated as RE or as the clipped compound Releng, is a sub-discipline in software engineering concerned with the compilation, assembly, and delivery of source code into finished products or other software components. Associated with the software release life cycle, it was said by Boris Debic of Google Inc. that release engineering is to software engineering as manufacturing is to an industrial process:

Release engineering is the difference between manufacturing software in small teams or startups and manufacturing software in an industrial way that is repeatable, gives predictable results, and scales well. These industrial style practices not only contribute to the growth of a company but also are key factors in enabling growth.

Build automation is the practice of building software systems in a relatively unattended fashion. The build is configured to run with minimized or no software developer interaction and without using a developer's personal computer. Build automation encompasses the act of configuring the build system as well the resulting system itself.

AnthillPro is a software tool originally developed and released as one of the first continuous integration servers. AnthillPro automates the process of building code into software projects and testing it to verify that project quality has been maintained. Software developers are able to identify bugs and errors earlier by using AnthillPro to track, collate, and test changes in real time to a collectively maintained body of computer code.

Multi-stage continuous integration is a software development technique intended to achieve highly integrated parallel development activity while reducing the scope of integration problems.

<span class="mw-page-title-main">Release management</span> Process of software building

Release management is the process of managing, planning, scheduling and controlling a software build through different stages and environments; it includes testing and deploying software releases.

DevOps is a methodology in the software development and IT industry. Used as a set of practices and tools, DevOps integrates and automates the work of software development (Dev) and IT operations (Ops) as a means for improving and shortening the systems development life cycle. DevOps is complementary to agile software development; several DevOps aspects came from the agile way of working.

Appcircle Inc. is a mobile technology company focusing on Mobile DevOps and mobile Continuous Integration and Continuous Delivery for mobile application development, primarily in enterprise settings. Appcircle is headquartered in Delaware with offices in New Jersey, Palo Alto and Istanbul.

Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate. Continuous testing was originally proposed as a way of reducing waiting time for feedback to developers by introducing development environment-triggered tests as well as more traditional developer/tester-triggered tests.

Continuous delivery (CD) is a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time. It aims at building, testing, and releasing software with greater speed and frequency. The approach helps reduce the cost, time, and risk of delivering changes by allowing for more incremental updates to applications in production. A straightforward and repeatable deployment process is important for continuous delivery.

<span class="mw-page-title-main">Travis CI</span> Service to build and test software projects

Travis CI is a hosted continuous integration service used to build and test software projects hosted on GitHub, Bitbucket, GitLab, Perforce, Apache Subversion and Assembla.

<span class="mw-page-title-main">BuildMaster</span>

BuildMaster is an application release automation tool, designed by the software development team Inedo. It combines build management and ARA capabilities to manage and automate processes primarily related to continuous integration, database change scripts, and production deployments, overall releasing applications reliably. The tool is browser-based and able to be used "out-of-the-box". Its feature set and scope puts it in line with the DevOps movement, and is marketed as "more than a release automatigs together the people, processes, and practices that allow teams to deliver software rapidly, reliably, and responsibly.” It's a tool that embodies incremental DevOps adoption.

In software development, a neutral build is a software build that reflects the current state of the source code checked into the source code version control system by the developers, and done in a neutral environment.

In software deployment, an environment or tier is a computer system or set of systems in which a computer program or software component is deployed and executed. In simple cases, such as developing and immediately executing a program on the same machine, there may be a single environment, but in industrial use, the development environment and production environment are separated, often with several stages in between. This structured release management process allows phased deployment (rollout), testing, and rollback in case of problems.

Probo is a user automated testing tool that provides continuous integration, workflow organization and quality assurance using the same tool for project managers and developers.

Wercker is a Docker-based continuous delivery platform that helps software developers build and deploy their applications and microservices. Using its command-line interface, developers can create Docker containers on their desktop, automate their build and deploy processes, testing them on their desktop, and then deploy them to various cloud platforms, ranging from Heroku to AWS and Rackspace. The command-line interface to Wercker has been open-sourced.

In software engineering, CI/CD or CICD is the combined practices of continuous integration (CI) and continuous delivery (CD) or, less often, continuous deployment. They are sometimes referred to collectively as continuous development or continuous software development.

References

  1. 1 2 3 Fowler, Martin (1 May 2006). "Continuous Integration" . Retrieved 9 January 2014.
  2. Booch, Grady (1991). Object Oriented Design: With Applications. Benjamin Cummings. p. 209. ISBN   9780805300918 . Retrieved 18 August 2014.
  3. Beck, K. (1999). "Embracing change with extreme programming". Computer. 32 (10): 70–77. doi:10.1109/2.796139. ISSN   0018-9162.
  4. Kaiser, G. E.; Perry, D. E.; Schell, W. M. (1989). Infuse: fusing integration test management with change management. Proceedings of the Thirteenth Annual International Computer Software & Applications Conference. Orlando, Florida. pp. 552–558. CiteSeerX   10.1.1.101.3770 . doi:10.1109/CMPSAC.1989.65147.
  5. Booch, Grady (December 1998). Object-Oriented Analysis and Design with applications (PDF) (2nd ed.). Archived from the original (PDF) on 19 August 2019. Retrieved 2 December 2014.
  6. Beck, Kent (28 March 1998). "Extreme Programming: A Humanistic Discipline of Software Development". Fundamental Approaches to Software Engineering: First International Conference. Vol. 1. Lisbon, Portugal: Springer. p. 4. ISBN   9783540643036.
  7. Beck, Kent (1999). Extreme Programming Explained . Addison-Wesley Professional. p.  97. ISBN   978-0-201-61641-5.
  8. "A Brief History of DevOps, Part III: Automated Testing and Continuous Integration". CircleCI. 1 February 2018. Retrieved 19 May 2018.
  9. Sane, Parth (2021), "A Brief Survey of Current Software Engineering Practices in Continuous Integration and Automated Accessibility Testing", 2021 Sixth International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET), pp. 130–134, arXiv: 2103.00097 , doi:10.1109/WiSPNET51692.2021.9419464, ISBN   978-1-6654-4086-8, S2CID   232076320
  10. Radigan, Dan. "Continuous integration". Atlassian Agile Coach.
  11. Brauneis, David (1 January 2010). "[OSLC] Possible new Working Group – Automation". open-services.net Community (Mailing list). Archived from the original on 1 September 2018. Retrieved 16 February 2010.
  12. Taylor, Bradley. "Rails Deployment and Automation with ShadowPuppet and Capistrano". Rails machine (blog). Archived from the original on 2 December 2012. Retrieved 16 February 2010.
  13. Duvall, Paul M. (2007). Continuous Integration. Improving Software Quality and Reducing Risk. Addison-Wesley. ISBN   978-0-321-33638-5.
  14. Cunningham, Ward (5 August 2009). "Integration Hell". WikiWikiWeb. Retrieved 19 September 2009.
  15. "What is Continuous Integration?". Amazon Web Services.
  16. Fowler, Martin. "Practices". Continuous Integration (article). Retrieved 29 November 2015.
  17. Paul M. Duvall; Steve Matyas; Andrew Glover (2007). Continuous Integration: Improving Software Quality and Reducing Risk. Addison-Wesley Professional. ISBN   978-0-321-33638-5.
  18. Ries, Eric (30 March 2009). "Continuous deployment in 5 easy steps". Radar. O’Reilly. Retrieved 10 January 2013.
  19. Fitz, Timothy (10 February 2009). "Continuous Deployment at IMVU: Doing the impossible fifty times a day". Wordpress. Retrieved 10 January 2013.
  20. Junpeng, Jiang; Zhu, Can; Zhang, Xiaofang (July 2020). "An Empirical Study on the Impact of Code Contributor on Code Smell" (PDF). International Journal of Performability Engineering. 16 (7): 1067–1077. doi:10.23940/ijpe.20.07.p9.10671077. S2CID   222588815.
  21. Laukkanen, Eero (2016). "Problems, causes and solutions when adopting continuous delivery—A systematic literature review". Information and Software Technology. 82: 55–79. doi: 10.1016/j.infsof.2016.10.001 .
  22. 1 2 3 Debbiche, Adam. "Assessing challenges of continuous integration in the context of software requirements breakdown: a case study" (PDF).