Regression testing

Last updated

Regression testing (rarely, non-regression testing [1] ) is re-running functional and non-functional tests to ensure that previously developed and tested software still performs as expected after a change. [2] If not, that would be called a regression .

Contents

Changes that may require regression testing include bug fixes, software enhancements, configuration changes, and even substitution of electronic components (hardware). [3] As regression test suites tend to grow with each found defect, test automation is frequently involved. The evident exception is the GUIs regression testing, which normally must be executed manually. Sometimes a change impact analysis is performed to determine an appropriate subset of tests (non-regression analysis [4] ).

Background

As software is updated or changed, or reused on a modified target, emergence of new faults and/or re-emergence of old faults is quite common.

Sometimes re-emergence occurs because a fix gets lost through poor revision control practices (or simple human error in revision control). Often, a fix for a problem will be "fragile" in that it fixes the problem in the narrow case where it was first observed but not in more general cases which may arise over the lifetime of the software. Frequently, a fix for a problem in one area inadvertently causes a software bug in another area.

It may happen that when a feature is redesigned some of the same mistakes that were made in the original implementation of the feature also occur in the redesign. In most software development situations, it is considered good coding practice, when a bug is located and fixed, to record a test that exposes the bug and re-run that test regularly after subsequent changes to the program. [5]

Although this may be done through manual testing procedures using programming techniques, it is often done using automated testing tools. [6] Such a test suite contains software tools that allow the testing environment to execute all the regression test cases automatically; some projects even set up automated systems to re-run all regression tests at specified intervals and report any failures (which could imply a regression or an out-of-date test). [7]

Common strategies are to run such a system after every successful compile (for small projects), every night, or once a week. Those strategies can be automated by an external tool.

Regression testing is an integral part of the extreme programming software development method. In this method, design documents are replaced by extensive, repeatable, and automated testing of the entire software package throughout each stage of the software development process. Regression testing is done after functional testing has concluded, to verify that the other functionalities are working.

In the corporate world, regression testing has traditionally been performed by a software quality assurance team after the development team has completed work. However, defects found at this stage are the most costly to fix. This problem is being addressed by the rise of unit testing. Although developers have always written test cases as part of the development cycle, these test cases have generally been either functional tests or unit tests that verify only intended outcomes. Developer testing compels a developer to focus on unit testing and to include both positive and negative test cases. [8]

Techniques

The various regression testing techniques are:

Retest all

This technique checks all the test cases on the current program to check its integrity. Though it is expensive as it needs to re-run all the cases, it ensures that there are no errors because of the modified code. [9]

Regression test selection

Unlike Retest all, this technique runs a part of the test suite (owing to the cost of retest all) if the cost of selecting the part of the test suite is less than the Retest all technique. [9]

Test case prioritization

Prioritize the test cases so as to increase a test suite's rate of fault detection. Test case prioritization techniques schedule test cases so that the test cases that are higher in priority are executed before the test cases that have a lower priority. [9]

Types of test case prioritization

  • General prioritization – Prioritize test cases that will be beneficial on subsequent versions
  • Version-specific prioritization – Prioritize test cases with respect to a particular version of the software.

Hybrid

This technique is a hybrid of regression test selection and test case prioritization. [9]

Benefits and drawbacks

Regression testing is performed when changes are made to the existing functionality of the software or if there is a bug fix in the software. Regression testing can be achieved through multiple approaches; if a test all approach is followed, it provides certainty that the changes made to the software have not affected the existing functionalities, which are unaltered. [10]

In agile software development—where the software development life cycles are very short, resources are scarce, and changes to the software are very frequent—regression testing might introduce a lot of unnecessary overhead. [10]

In a software development environment which tends to use black box components from a third party, performing regression testing can be tricky, as any change in the third-party component may interfere with the rest of the system (and performing regression testing on a third-party component is difficult, because it is an unknown entity). [10]

Uses

Regression testing can be used not only for testing the correctness of a program but often also for tracking the quality of its output. [11] For instance, in the design of a compiler, regression testing could track the code size and the time it takes to compile and execute the test suite cases.

Also as a consequence of the introduction of new bugs, program maintenance requires far more system testing per statement written than any other programming. Theoretically, after each fix, one must run the entire batch of test cases previously run against the system to ensure that it has not been damaged in an obscure way. In practice, such regression testing must indeed approximate this theoretical idea, and it is very costly.

—  Fred Brooks, The Mythical Man Month , p. 122

Regression tests can be broadly categorized as functional tests or unit tests. Functional tests exercise the complete program with various inputs. Unit tests exercise individual functions, subroutines, or object methods. Both functional testing tools and unit-testing tools tend to be automated and are often third-party products that are not part of the compiler suite. A functional test may be a scripted series of program inputs, possibly even involving an automated mechanism for controlling mouse movements and clicks. A unit test may be a set of separate functions within the code itself or a driver layer that links to the code without altering the code being tested.

See also

Related Research Articles

Software testing is the act of examining the artifacts and the behavior of the software under test by verification and validation. Software testing can also provide an objective, independent view of the software to allow the business to appreciate and understand the risks of software implementation. Test techniques include, but are not limited to:

In computer programming, unit testing is a software testing method by which individual units of source code—sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures—are tested to determine whether they are fit for use. It is a standard step in development and implementation approaches such as Agile.

Test-driven development (TDD) is a software development process relying on software requirements being converted to test cases before software is fully developed, and tracking all software development by repeatedly testing the software against all test cases. This is as opposed to software being developed first and test cases created later.

A programming tool or software development tool is a computer program that software developers use to create, debug, maintain, or otherwise support other programs and applications. The term usually refers to relatively simple programs, that can be combined to accomplish a task, much as one might use multiple hands to fix a physical object. The most basic tools are a source code editor and a compiler or interpreter, which are used ubiquitously and continuously. Other tools are used more or less depending on the language, development methodology, and individual engineer, often used for a discrete task, like a debugger or profiler. Tools may be discrete programs, executed separately – often from the command line – or may be parts of a single large program, called an integrated development environment (IDE). In many cases, particularly for simpler use, simple ad hoc techniques are used instead of a tool, such as print debugging instead of using a debugger, manual timing instead of a profiler, or tracking bugs in a text file or spreadsheet instead of a bug tracking system.

In software testing, test automation is the use of software separate from the software being tested to control the execution of tests and the comparison of actual outcomes with predicted outcomes. Test automation can automate some repetitive but necessary tasks in a formalized testing process already in place, or perform additional testing that would be difficult to do manually. Test automation is critical for continuous delivery and continuous testing.

<span class="mw-page-title-main">Continuous integration</span> Software development practice based on frequent submission of granular changes

In software engineering, continuous integration (CI) is the practice of merging all developers' working copies to a shared mainline several times a day. Nowadays it is typically implemented in such a way that it triggers an automated build with testing. Grady Booch first proposed the term CI in his 1991 method, although he did not advocate integrating several times a day. Extreme programming (XP) adopted the concept of CI and did advocate integrating more than once per day – perhaps as many as tens of times per day.

Dynamic program analysis is analysis of computer software that involves executing the program in question. Dynamic program analysis includes familiar techniques from software engineering such as unit testing, debugging, and measuring code coverage, but also includes lesser-known techniques like program slicing and invariant inference. Dynamic program analysis is widely applied in security in the form of runtime memory error detection, fuzzing, dynamic symbolic execution, and taint tracking.

Dynamic testing is a term used in software engineering to describe the testing of the dynamic behavior of code.

A software regression is a type of software bug where a feature that has worked before stops working. This may happen after changes are applied to the software's source code, including the addition of new features and bug fixes. They may also be introduced by changes to the environment in which the software is running, such as system upgrades, system patching or a change to daylight saving time. A software performance regression is a situation where the software still functions correctly, but performs more slowly or uses more memory or resources than before. Various types of software regressions have been identified in practice, including the following:

A test strategy is an outline that describes the testing approach of the software development cycle. The purpose of a test strategy is to provide a rational deduction from organizational, high-level objectives to actual test activities to meet those objectives from a quality assurance perspective. The creation and documentation of a test strategy should be done in a systematic way to ensure that all objectives are fully covered and understood by all stakeholders. It should also frequently be reviewed, challenged and updated as the organization and the product evolve over time. Furthermore, a test strategy should also aim to align different stakeholders of quality assurance in terms of terminology, test and integration levels, roles and responsibilities, traceability, planning of resources, etc.

<span class="mw-page-title-main">Parasoft</span> Software testing framework

Parasoft is an independent software vendor specializing in automated software testing and application security with headquarters in Monrovia, California. It was founded in 1987 by four graduates of the California Institute of Technology who planned to commercialize the parallel computing software tools they had been working on for the Caltech Cosmic Cube, which was the first working hypercube computer built.

In computer programming and software development, debugging is the process of finding and resolving bugs within computer programs, software, or systems.

Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate. Continuous testing was originally proposed as a way of reducing waiting time for feedback to developers by introducing development environment-triggered tests as well as more traditional developer/tester-triggered tests.

Bisection is a method used in software development to identify change sets that result in a specific behavior change. It is mostly employed for finding the patch that introduced a bug. Another application area is finding the patch that indirectly fixed a bug.

<span class="mw-page-title-main">Parasoft C/C++test</span> Integrated set of tools

Parasoft C/C++test is an integrated set of tools for testing C and C++ source code that software developers use to analyze, test, find defects, and measure the quality and security of their applications. It supports software development practices that are part of development testing, including static code analysis, dynamic code analysis, unit test case generation and execution, code coverage analysis, regression testing, runtime error detection, requirements traceability, and code review. It's a commercial tool that supports operation on Linux, Windows, and Solaris platforms as well as support for on-target embedded testing and cross compilers.

Development testing is a software development process that involves synchronized application of a broad spectrum of defect prevention and detection strategies in order to reduce software development risks, time, and costs.

In computer programming and software testing, smoke testing is preliminary testing or sanity testing to reveal simple failures severe enough to, for example, reject a prospective software release. Smoke tests are a subset of test cases that cover the most important functionality of a component or system, used to aid assessment of whether main functions of the software appear to work correctly. When used to determine if a computer program should be subjected to further, more fine-grained testing, a smoke test may be called a pretest or an intake test. Alternatively, it is a set of tests run on each new build of a product to verify that the build is testable before the build is released into the hands of the test team. In the DevOps paradigm, use of a build verification test step is one hallmark of the continuous integration maturity stage.

In software development, a neutral build is a software build that reflects the current state of the source code checked into the source code version control system by the developers, and done in a neutral environment.

Automatic bug-fixing is the automatic repair of software bugs without the intervention of a human programmer. It is also commonly referred to as automatic patch generation, automatic bug repair, or automatic program repair. The typical goal of such techniques is to automatically generate correct patches to eliminate bugs in software programs without causing software regression.

This article discusses a set of tactics useful in software testing. It is intended as a comprehensive list of tactical approaches to Software Quality Assurance (more widely colloquially known as Quality Assurance and general application of the test method.

References

  1. ↑ Pezzè, Mauro; Young, Michal (2008). Software testing and analysis: process, principles, and techniques. Wiley. Testing activities that focus on regression problems are called (non) regression testing. Usually "non" is omitted
  2. ↑ Basu, Anirban (2015). Software Quality Assurance, Testing and Metrics. PHI Learning. ISBN   978-81-203-5068-7.
  3. ↑ National Research Council Committee on Aging Avionics in Military Aircraft: Aging Avionics in Military Aircraft. The National Academies Press, 2001, page 2: ″Each technology-refresh cycle requires regression testing.″
  4. ↑ Boulanger, Jean-Louis (2015). CENELEC 50128 and IEC 62279 Standards. Wiley. ISBN   978-1119122487.
  5. ↑ Kolawa, Adam; Huizinga, Dorota (2007). Automated Defect Prevention: Best Practices in Software Management. Wiley-IEEE Computer Society Press. p. 73. ISBN   978-0-470-04212-0.
  6. ↑ Automate Regression Tests When Feasible, Automated Testing: Selected Best Practices, Elfriede Dustin, Safari Books Online
  7. ↑ daVeiga, Nada (2008-02-06). "Change Code Without Fear: Utilize a Regression Safety Net". Dr. Dobb's Journal .
  8. ↑ Dudney, Bill (2004-12-08). "Developer Testing Is 'In': An interview with Alberto Savoia and Kent Beck" . Retrieved 2007-11-29.
  9. 1 2 3 4 Duggal, Gaurav; Suri, Bharti (2008-03-29). Understanding Regression Testing Techniques. National Conference on Challenges and Opportunities. Mandi Gobindgarh, Punjab, India. CiteSeerX   10.1.1.460.5875 .
  10. 1 2 3 Yoo, S.; Harman, M. (2010). "Regression testing minimization, selection and prioritization: a survey". Software Testing, Verification and Reliability. 22 (2): 67–120. doi:10.1002/stvr.430.
  11. ↑ Kolawa, Adam. "Regression Testing, Programmer to Programmer". Wrox.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.