Software verification and validation

Last updated March 02, 2024

In software project management, software testing, and software engineering, verification and validation (V&V) is the process of checking that a software system meets specifications and requirements so that it fulfills its intended purpose. It may also be referred to as software quality control. It is normally the responsibility of software testers as part of the software development lifecycle. In simple terms, software verification is: "Assuming we should build X, does our software achieve its goals without any bugs or gaps?" On the other hand, software validation is: "Was X what we should have built? Does X meet the high-level requirements?"

Definitions

Verification and validation are not the same thing, although they are often confused. Boehm succinctly expressed the difference as^[1]

Verification: Are we building the product right?
Validation: Are we building the right product?

"Building the product right" checks that the specifications are correctly implemented by the system while "building the right product" refers back to the user's needs. In some contexts, it is required to have written requirements for both as well as formal procedures or protocols for determining compliance. Ideally, formal methods provide a mathematical guarantee that software meets its specifications.

Building the product right implies the use of the Requirements Specification as input for the next phase of the development process, the design process, the output of which is the Design Specification. Then, it also implies the use of the Design Specification to feed the construction process. Every time the output of a process correctly implements its input specification, the software product is one step closer to final verification. If the output of a process is incorrect, the developers are not correctly building the product the stakeholders want. This kind of verification is called "artifact or specification verification".

software product is released;^{[ clarification needed ]} though, it need not always be the case.^{[lower-alpha 1]}

Software verification

It would imply to verify if the specifications are met by running the software but this is not possible (e. g., how can anyone know if the architecture/design/etc. are correctly implemented by running the software?). Only by reviewing its associated artifacts, can someone conclude whether or not the specifications are met.

Artifact or specification verification

The output of each software development process stage can also be subject to verification when checked against its input specification (see the definition by CMMI below).

Examples of artifact verification:

Of the design specification against the requirement specification: Do the architectural design, detailed design and database logical model specifications correctly implement the functional and non-functional requirements specifications?
Of the construction artifacts against the design specification: Do the source code, user interfaces and database physical model correctly implement the design specification?

Software validation

Software validation checks that the software product satisfies or fits the intended use (high-level checking), i.e., the software meets the user requirements, not as specification artifacts or as needs of those who will operate the software only; but, as the needs of all the stakeholders (such as users, operators, administrators, managers, investors, etc.). There are two ways to perform software validation: internal and external. During internal software validation, it is assumed that the goals of the stakeholders were correctly understood and that they were expressed in the requirement artifacts precisely and comprehensively. If the software meets the requirement specification, it has been internally validated. External validation happens when it is performed by asking the stakeholders if the software meets their needs. Different software development methodologies call for different levels of user and stakeholder involvement and feedback; so, external validation can be a discrete or a continuous event. Successful final external validation occurs when all the stakeholders accept the software product and express that it satisfies their needs. Such final external validation requires the use of an acceptance test which is a dynamic test.

However, it is also possible to perform internal static tests to find out if the software meets the requirements specification but that falls into the scope of static verification because the software is not running.

Artifact or specification validation

Requirements should be validated before the software product as a whole is ready (the waterfall development process requires them to be perfectly defined before design starts; but iterative development processes do not require this to be so and allow their continual improvement).

Examples of artifact validation:

User Requirements Specification validation: User requirements as stated in a document called User Requirements Specification are validated by checking if they indeed represent the will and goals of the stakeholders. This can be done by interviewing the stakeholders and asking them directly (static testing) or even by releasing prototypes and having the users and stakeholders to assess them (dynamic testing).
User input validation: User input (gathered by any peripheral such as keyboard, bio-metric sensor, etc.) is validated by checking if the input provided by the software operators or users meets the domain rules and constraints (such as data type, range, and format).

Validation vs. verification

According to the Capability Maturity Model (CMMI-SW v1.1),^[3]

Software Validation: The process of evaluating software during or at the end of the development process to determine whether it satisfies specified requirements. [IEEE-STD-610]
Software Verification: The process of evaluating software to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase. [IEEE-STD-610]

Validation during the software development process can be seen as a form of User Requirements Specification validation; and, that at the end of the development process is equivalent to Internal and/or External Software validation. Verification, from CMMI's point of view, is evidently of the artifact kind.

In other words, software verification ensures that the output of each phase of the software development process effectively carry out what its corresponding input artifact specifies (requirement -> design -> software product), while software validation ensures that the software product meets the needs of all the stakeholders (therefore, the requirement specification was correctly and accurately expressed in the first place). Software verification ensures that "you built it right" and confirms that the product, as provided, fulfills the plans of the developers. Software validation ensures that "you built the right thing" and confirms that the product, as provided, fulfills the intended use and goals of the stakeholders.

This article has used the strict or narrow definition of verification.

From a testing perspective:

Fault – wrong or missing function in the code.
Failure – the manifestation of a fault during execution. The software was not effective. It does not do "what" it is supposed to do.
Malfunction – according to its specification the system does not meet its specified functionality. The software was not efficient (it took too many resources such as CPU cycles, it used too much memory, performed too many I/O operations, etc.), it was not usable, it was not reliable, etc. It does not do something "how" it is supposed to do it.

Related concepts

Both verification and validation are related to the concepts of quality and of software quality assurance. By themselves, verification and validation do not guarantee software quality; planning, traceability, configuration management and other aspects of software engineering are required.

Within the modeling and simulation (M&S) community, the definitions of verification, validation and accreditation are similar:

M&S Verification is the process of determining that a computer model, simulation, or federation of models and simulations implementations and their associated data accurately represent the developer's conceptual description and specifications.^[4]
M&S Validation is the process of determining the degree to which a model, simulation, or federation of models and simulations, and their associated data are accurate representations of the real world from the perspective of the intended use(s).^[4]
Accreditation is the formal certification that a model or simulation is acceptable to be used for a specific purpose.^[4]

The definition of M&S validation focuses on the accuracy with which the M&S represents the real-world intended use(s). Determining the degree of M&S accuracy is required because all M&S are approximations of reality, and it is usually critical to determine if the degree of approximation is acceptable for the intended use(s). This stands in contrast to software validation.

V&V methods

Formal

In mission-critical software systems, formal methods may be used to ensure the correct operation of a system. These formal methods can prove costly, however, representing as much as 80 percent of total software design cost.

Independent

Independent Software Verification and Validation (ISVV) is targeted at safety-critical software systems and aims to increase the quality of software products, thereby reducing risks and costs through the operational life of the software. The goal of ISVV is to provide assurance that software performs to the specified level of confidence and within its designed parameters and defined requirements.^[5]^[6]

ISVV activities are performed by independent engineering teams, not involved in the software development process, to assess the processes and the resulting products. The ISVV team independency is performed at three different levels: financial, managerial and technical.

ISVV goes beyond "traditional" verification and validation techniques, applied by development teams. While the latter aim to ensure that the software performs well against the nominal requirements, ISVV is focused on non-functional requirements such as robustness and reliability, and on conditions that can lead the software to fail.

ISVV results and findings are fed back to the development teams for correction and improvement.

History

ISVV derives from the application of IV&V (Independent Verification and Validation) to the software. Early ISVV application (as known today) dates back to the early 1970s when the U.S. Army sponsored the first significant program related to IV&V for the Safeguard Anti-Ballistic Missile System.^[7] Another example is NASA's IV&V Program, which was established in 1993.^[8]

By the end of the 1970s IV&V was rapidly becoming popular. The constant increase in complexity, size and importance of the software led to an increasing demand on IV&V applied to software.

Meanwhile, IV&V (and ISVV for software systems) consolidated and is now widely used by organizations such as the DoD, FAA,^[9] NASA ^[8] and ESA.^[10] IV&V is mentioned in DO-178B, ISO/IEC 12207 and formalized in IEEE 1012.

At ESA

Initially in 2004-2005, a European consortium led by the European Space Agency, and composed by DNV, Critical Software SA, Terma and CODA SciSys plc created the first version of a guide devoted to ISVV, called "ESA Guide for Independent Verification and Validation" with support from other organizations.^[11] This guide covers the methodologies applicable to all the software engineering phases in what concerns ISVV.

In 2008 the European Space Agency released a second version, having received inputs from many different European Space ISVV stakeholders.^[11]

Methodology

ISVV is usually composed by five principal phases, these phases can be executed sequentially or as results of a tailoring process.

Planning

Planning of ISVV activities
System criticality analysis: Identification of critical components through a set of RAMS activities (Value for Money)
Selection of the appropriate methods and tools

Requirements verification

Verification for: completeness, correctness, testability

Design verification

Design adequacy and conformance to software requirements and interfaces
Internal and external consistency
Verification of feasibility and maintenance

Code verification

Verification for: completeness, correctness, consistency
Code metrics analysis
Coding standards compliance verification

Validation

Identification of unstable components/functionalities
Validation focused on error-handling: complementary (not concurrent) validation regarding the one performed by the development team
Compliance with software and system requirements
Black box testing and White box testing techniques
Experience based techniques

Regulatory environment

Software often must meet the compliance requirements of legally regulated industries, which is often guided by government agencies^[12]^[13] or industrial administrative authorities. For instance, the FDA requires software versions and patches to be validated.^[14]

External links

Chapter on Software quality (including VnV) Archived 2014-12-08 at the Wayback Machine in SWEBOK

Related Research Articles

In engineering and its various subdisciplines, acceptance testing is a test conducted to determine if the requirements of a specification or contract are met. It may involve chemical tests, physical tests, or performance tests.

Software testing is the act of examining the artifacts and the behavior of the software under test by verification and validation. Software testing can also provide an objective, independent view of the software to allow the business to appreciate and understand the risks of software implementation. Test techniques include, but are not limited to:

The spiral model is a risk-driven software development process model. Based on the unique risk patterns of a given project, the spiral model guides a team to adopt elements of one or more process models, such as incremental, waterfall, or evolutionary prototyping.

In the context of hardware and software systems, formal verification is the act of proving or disproving the correctness of a system with respect to a certain formal specification or property, using formal methods of mathematics. Formal verification is a key incentive for formal specification of systems, and is at the core of formal methods. It represents an important dimension of analysis and verification in electronic design automation and is one approach to software verification. The use of formal verification enables the highest Evaluation Assurance Level (EAL7) in the framework of common criteria for computer security certification.

In systems engineering and software engineering, requirements analysis focuses on the tasks that determine the needs or conditions to meet the new or altered product or project, taking account of the possibly conflicting requirements of the various stakeholders, analyzing, documenting, validating and managing software or system requirements.

In product development and process optimization, a requirement is a singular documented physical or functional need that a particular design, product or process aims to satisfy. It is commonly used in a formal sense in engineering design, including for example in systems engineering, software engineering, or enterprise engineering. It is a broad concept that could speak to any necessary function, attribute, capability, characteristic, or quality of a system for it to have value and utility to a customer, organization, internal user, or other stakeholder. Requirements can come with different levels of specificity; for example, a requirement specification or requirement "spec" refers to an explicit, highly objective/clear requirement to be satisfied by a material, design, product, or service.

<span class="mw-page-title-main">Systems development life cycle</span> Systems engineering terms

In systems engineering, information systems and software engineering, the systems development life cycle (SDLC), also referred to as the application development life cycle, is a process for planning, creating, testing, and deploying an information system. The SDLC concept applies to a range of hardware and software configurations, as a system can be composed of hardware only, software only, or a combination of both. There are usually six stages in this cycle: requirement analysis, design, development and testing, implementation, documentation, and evaluation.

Software verification is a discipline of software engineering, programming languages, and theory of computation whose goal is to assure that software satisfies the expected requirements.

System testing is testing conducted on a complete integrated system to evaluate the system's compliance with its specified requirements.

The V-model is a graphical representation of a systems development lifecycle. It is used to produce rigorous development lifecycle models and project management models. The V-model falls into three broad categories, the German V-Modell, a general testing model, and the US government standard.

Requirements management is the process of documenting, analyzing, tracing, prioritizing and agreeing on requirements and then controlling change and communicating to relevant stakeholders. It is a continuous process throughout a project. A requirement is a capability to which a project outcome should conform.

<span class="mw-page-title-main">Integrated circuit design</span> Engineering process for electronic hardware

Integrated circuit design, or IC design, is a sub-field of electronics engineering, encompassing the particular logic and circuit design techniques required to design integrated circuits, or ICs. ICs consist of miniaturized electronic components built into an electrical network on a monolithic semiconductor substrate by photolithography.

The process of establishing documentary evidence demonstrating that a procedure, process, or activity carried out in testing and then production maintains the desired level of compliance at all stages. In the pharmaceutical industry, it is very important that in addition to final testing and compliance of products, it is also assured that the process will consistently produce the expected results. The desired results are established in terms of specifications for outcome of the process. Qualification of systems and equipment is therefore a part of the process of validation. Validation is a requirement of food, drug and pharmaceutical regulating agencies such as the US FDA and their good manufacturing practices guidelines. Since a wide variety of procedures, processes, and activities need to be validated, the field of validation is divided into a number of subsections including the following:

A design history file is a compilation of documentation that describes the design history of a finished medical device. The design history file, or DHF, is part of regulation introduced in 1990 when the U.S. Congress passed the Safe Medical Devices Act, which established new standards for medical devices that can cause or contribute to the death, serious illness, or injury of a patient. Prior to this legislation, U.S. Food and Drug Administration (FDA) auditors were limited to examining the production and quality control records of the device.

A functional specification in systems engineering and software development is a document that specifies the functions that a system or component must perform.

In software development, the V-model represents a development process that may be considered an extension of the waterfall model, and is an example of the more general V-model. Instead of moving down in a linear way, the process steps are bent upwards after the coding phase, to form the typical V shape. The V-Model demonstrates the relationships between each phase of the development life cycle and its associated phase of testing. The horizontal and vertical axes represent time or project completeness (left-to-right) and level of abstraction, respectively.

Verification and validation are independent procedures that are used together for checking that a product, service, or system meets requirements and specifications and that it fulfills its intended purpose. These are critical components of a quality management system such as ISO 9000. The words "verification" and "validation" are sometimes preceded with "independent", indicating that the verification and validation is to be performed by a disinterested third party. "Independent verification and validation" can be abbreviated as "IV&V".

Software quality control is the set of procedures used by organizations to ensure that a software product will meet its quality goals at the best value to the customer, and to continually improve the organization’s ability to produce software products in the future.

Software requirements for a system are the description of what the system should do, the service or services that it provides and the constraints on its operation. The IEEE Standard Glossary of Software Engineering Terminology defines a requirement as:

A condition or capability needed by a user to solve a problem or achieve an objective
A condition or capability that must be met or possessed by a system or system component to satisfy a contract, standard, specification, or other formally imposed document
A documented representation of a condition or capability as in 1 or 2

Verification and validation of computer simulation models is conducted during the development of a simulation model with the ultimate goal of producing an accurate and credible model. "Simulation models are increasingly being used to solve problems and to aid in decision-making. The developers and users of these models, the decision makers using information obtained from the results of these models, and the individuals affected by decisions based on such models are all rightly concerned with whether a model and its results are "correct". This concern is addressed through verification and validation of the simulation model.

References

↑ Pham, H. (1999). Software Reliability. John Wiley & Sons, Inc. p. 567. ISBN 9813083840. Software Validation. The process of ensuring that the software is performing the right process. Software Verification. The process of ensuring that the software is performing the process right." Likewise and also there: "In short, Boehm (3) expressed the difference between the software verification and software validation as follows: Verification: Are we building the product right? Validation: Are we building the right product?.
↑ Snoderly, John; Faisandier, Alan. "System Verification". ISO/IEC/IEEE 15288 . Archived from the original on 26 June 2023. Retrieved 18 May 2022.
↑ "CMMI for Software Engineering, Version 1.1, Staged Representation (CMMI-SW, V1.1, Staged)". resources.sei.cmu.edu. Retrieved 2021-03-20.
1 2 3 Department of Defense Documentation of Verification, Validation & Accreditation (VV&A) for Models and Simulations, Missile Defense Agency, 2008
↑ Rogers, R. (1981-10-26). "Planning for independent software verification and validation". 3rd Computers in Aerospace Conference. Computers in Aerospace Conference. San Diego, CA, U.S.: American Institute of Aeronautics and Astronautics. doi:10.2514/6.1981-2100.
↑ Ambrosio, Ana; Mattiello-Francisco, Fátima; Martins, Eliane (2008-05-12). "A Independent Software Verification and Validation Process for Space Applications". SpaceOps 2008 Conference. Heidelberg, Germany: American Institute of Aeronautics and Astronautics. doi: 10.2514/6.2008-3517 . ISBN 978-1-62410-167-0.
↑ Lewis, Robert O. (1992). Independent verification and validation : a life cycle engineering process for quality software. New York: Wiley. ISBN 0-471-57011-7. OCLC 74908695.
1 2 Asbury, Michael (2015-03-09). "About NASA's IV&V Program". NASA. Retrieved 2021-03-20.
↑ Balci, O. (2010). "Golden Rules of Verification, Validation, Testing, and Certification of Modeling and Simulation Applications". S2CID 61476570.{{cite web}}: Missing or empty |url= (help)
↑ "Flight Software Systems Section (TEC-SWF)". www.esa.int. Retrieved 2021-03-20.
1 2 lavva.pt. "New ISVV Guide for Space in the Works". www.criticalsoftware.com. Retrieved 2021-03-20.
↑ "General Principles of Software validation; Final Guidance for Industry and FDA Staff" (PDF). Food and Drug Administration. 11 January 2002. Retrieved 12 July 2009.
↑ "Guidance for Industry: Part 11, Electronic Records; Electronic Signatures — Scope and Application" (PDF). Food and Drug Administration. August 2003. Retrieved 12 July 2009.
↑ "Guidance for Industry: Cybersecurity for Networked Medical Devices Containing Off-the Shelf (OTS) Software" (PDF). Food and Drug Administration. 14 January 2005. Retrieved 12 July 2009.

Notes

↑ The term verification is often associated with the term validation and understood as a single concept of V&V. Validation is used to ensure that one is working the right problem, whereas verification is used to ensure that one has solved the problem right (Martin 1997). From an actual and etymological meaning, the term verification comes from the Latin verus, which means truth, and facere, which means to make/perform. Thus, verification means to prove that something is true or correct (a property, a characteristic, etc.). The term validation comes from the Latin valere, which means to become strong, and has the same etymological root as the word value. Thus, validation means to prove that something has the right features to produce the expected effects. (Adapted from "Verification and Validation in plain English" (Lake INCOSE 1999).^[2]

This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.

[PhamSoft99-1] Pham, H. (1999). Software Reliability. John Wiley & Sons, Inc. p. 567. ISBN 9813083840. Software Validation. The process of ensuring that the software is performing the right process. Software Verification. The process of ensuring that the software is performing the process right." Likewise and also there: "In short, Boehm (3) expressed the difference between the software verification and software validation as follows: Verification: Are we building the product right? Validation: Are we building the right product?.

[2] Snoderly, John; Faisandier, Alan. "System Verification". ISO/IEC/IEEE 15288 . Archived from the original on 26 June 2023. Retrieved 18 May 2022.

[4] "CMMI for Software Engineering, Version 1.1, Staged Representation (CMMI-SW, V1.1, Staged)". resources.sei.cmu.edu. Retrieved 2021-03-20.

[Missile_Defense_Agency-5] 1 2 3 Department of Defense Documentation of Verification, Validation & Accreditation (VV&A) for Models and Simulations, Missile Defense Agency, 2008

[6] Rogers, R. (1981-10-26). "Planning for independent software verification and validation". 3rd Computers in Aerospace Conference. Computers in Aerospace Conference. San Diego, CA, U.S.: American Institute of Aeronautics and Astronautics. doi:10.2514/6.1981-2100.

[7] Ambrosio, Ana; Mattiello-Francisco, Fátima; Martins, Eliane (2008-05-12). "A Independent Software Verification and Validation Process for Space Applications". SpaceOps 2008 Conference. Heidelberg, Germany: American Institute of Aeronautics and Astronautics. doi: 10.2514/6.2008-3517 . ISBN 978-1-62410-167-0.

[8] Lewis, Robert O. (1992). Independent verification and validation : a life cycle engineering process for quality software. New York: Wiley. ISBN 0-471-57011-7. OCLC 74908695.

[:0-9] 1 2 Asbury, Michael (2015-03-09). "About NASA's IV&V Program". NASA. Retrieved 2021-03-20.

[10] Balci, O. (2010). "Golden Rules of Verification, Validation, Testing, and Certification of Modeling and Simulation Applications". S2CID 61476570.{{cite web}}: Missing or empty |url= (help)

[11] "Flight Software Systems Section (TEC-SWF)". www.esa.int. Retrieved 2021-03-20.

[:1-12] 1 2 lavva.pt. "New ISVV Guide for Space in the Works". www.criticalsoftware.com. Retrieved 2021-03-20.

[gpsv-13] "General Principles of Software validation; Final Guidance for Industry and FDA Staff" (PDF). Food and Drug Administration. 11 January 2002. Retrieved 12 July 2009.

[dgi21-14] "Guidance for Industry: Part 11, Electronic Records; Electronic Signatures — Scope and Application" (PDF). Food and Drug Administration. August 2003. Retrieved 12 July 2009.

[gicn-15] "Guidance for Industry: Cybersecurity for Networked Medical Devices Containing Off-the Shelf (OTS) Software" (PDF). Food and Drug Administration. 14 January 2005. Retrieved 12 July 2009.

[3] The term verification is often associated with the term validation and understood as a single concept of V&V. Validation is used to ensure that one is working the right problem, whereas verification is used to ensure that one has solved the problem right (Martin 1997). From an actual and etymological meaning, the term verification comes from the Latin verus, which means truth, and facere, which means to make/perform. Thus, verification means to prove that something is true or correct (a property, a characteristic, etc.). The term validation comes from the Latin valere, which means to become strong, and has the same etymological root as the word value. Thus, validation means to prove that something has the right features to produce the expected effects. (Adapted from "Verification and Validation in plain English" (Lake INCOSE 1999).^[2]

[1]

[lower-alpha 1]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]

[14]

[2]