Specification by example

Last updated

Specification by example (SBE) is a collaborative approach to defining requirements and business-oriented functional tests for software products based on capturing and illustrating requirements using realistic examples instead of abstract statements. It is applied in the context of agile software development methods, in particular behavior-driven development. This approach is particularly successful for managing requirements and functional tests on large-scale projects of significant domain and organisational complexity. [1]

Contents

Specification by example is also known as example-driven development, executable requirements, acceptance test–driven development (ATDD [2] or A-TDD [3] ), Agile Acceptance Testing, [4] Test-Driven Requirements (TDR).

Advantages

Highly abstract or novel new concepts can be difficult to understand without concrete examples.[ citation needed ] Specification by example is intended to construct an accurate understanding, and significantly reduces feedback loops in software development, leading to less rework, higher product quality, faster turnaround time for software changes and better alignment of activities of various roles involved in software development such as testers, analysts and developers. [1]

Examples as a single source of truth

A key aspect of specification by example is creating a single source of truth about required changes from all perspectives. When business analysts work on their own documents, software developers maintain their own documentation and testers maintain a separate set of functional tests, software delivery effectiveness is significantly reduced by the need to constantly coordinate and synchronise those different versions of truth. With short iterative cycles, such coordination is often required on weekly or biweekly basis. With Specification by example, different roles participate in creating a single source of truth that captures everyone's understanding. Examples are used to provide clarity and precision, so that the same information can be used both as a specification and a business-oriented functional test. Any additional information discovered during development or delivery, such as clarification of functional gaps, missing or incomplete requirements or additional tests, is added to this single source of truth. As there is only one source of truth about the functionality, there is no need for coordination, translation and interpretation of knowledge inside the delivery cycle.

When applied to required changes, a refined set of examples is effectively a specification and a business-oriented test for acceptance of software functionality. After the change is implemented, specification with examples becomes a document explaining existing functionality. As the validation of such documents is automated, when they are validated frequently, such documents are a reliable source of information on business functionality of underlying software. To distinguish between such documents and typical printed documentation, which quickly gets outdated, [4] a complete set of specifications with examples is called Living Documentation. [1]

Key practices

Teams that apply Specification by example successfully commonly apply the following process patterns: [1]

Software teams that apply specification by example within a Scrum framework typically spend 5%-10% of their time in refining the product backlog, including specifying collaboratively, illustrating requirements using examples and refining examples. [3]

Example Mapping

Example Mapping is a simple technique that can steer the conversation and derive Acceptance criteria in a short time. The process breaks stories into Rules and Examples documented in the form of specification by example, and is a widely used technique in the BDD sphere. [5]

Applicability

Specification by example applies to projects with sufficient organisational and domain complexity to cause problems in understanding or communicating requirements from a business domain perspective. It does not apply to purely technical problems or where the key complexity is not in understanding or communicating knowledge. There are documented usages of this approach in domains including investment banking, financial trading, insurance, airline ticket reservation, online gaming and price comparison. [1] A similar approach is documented also in a nuclear-power plant simulation project. [3]

Tests based on shared examples fit best in the category of tests designed to support a team while delivering software from a business perspective (see Agile Testing Quadrants [6] ) - ensuring that the right product is built. They do not replace tests that look at a software system from a purely technical perspective (those that evaluate whether a product is built the right way, such as unit tests, component or technical integration tests) or tests that evaluate a product after it was developed (such as security penetration tests).

History

The earliest documented usage of realistic examples as a single source of truth, requirements and automated tests, on software projects is the WyCash+ project, described by Ward Cunningham in the paper A Pattern Language of Competitive Development [7] [8] in 1996. The name Specification by Example was coined by Martin Fowler in 2004. [9]

Specification by Example is an evolution of the Customer Test [10] practice of Extreme Programming proposed around 1997 and Ubiquitous Language [11] idea from Domain-driven design from 2004, using the idea of black-box tests as requirements described by Weinberg and Gause [12] in 1989.

Automation

Successful application of Specification by example on large scale projects requires frequent validation of software functionality against a large set of examples (tests). In practice, this requires tests based on examples to be automated. A common approach is to automate the tests but keep examples in a form readable and accessible to non-technical and technical team members, keeping the examples as a single source of truth. This process is supported by a class of test automation tools which work with tests divided into two aspects - the specification and the automation layer. The specification of a test which is often in a plain text or HTML form and contains the examples and auxiliary descriptions. The automation layer connects the example to a software system under test. Examples of such tools are:

Related Research Articles

Acceptance testing Test to determine if the requirements of a specification or contract are met

In engineering and its various subdisciplines, acceptance testing is a test conducted to determine if the requirements of a specification or contract are met. It may involve chemical tests, physical tests, or performance tests.

Software testing is an investigation conducted to provide stakeholders with information about the quality of the software product or service under test. Software testing can also provide an objective, independent view of the software to allow the business to appreciate and understand the risks of software implementation. Test techniques include the process of executing a program or application with the intent of finding failures, and verifying that the software product is fit for use.

Software development is the process of conceiving, specifying, designing, programming, documenting, testing, and bug fixing involved in creating and maintaining applications, frameworks, or other software components. Software development is a process of writing and maintaining the source code, but in a broader sense, it includes all that is involved between the conception of the desired software through to the final manifestation of the software, sometimes in a planned and structured process. Therefore, software development may include research, new development, prototyping, modification, reuse, re-engineering, maintenance, or any other activities that result in software products.

In software and systems engineering, the phrase use case is a polyseme with two senses:

  1. A usage scenario for a piece of software; often used in the plural to suggest situations where a piece of software may be useful.
  2. A potential scenario in which a system receives an external request and responds to it.

Test-driven development (TDD) is a software development process relying on software requirements being converted to test cases before software is fully developed, and tracking all software development by repeatedly testing the software against all test cases. This is as opposed to software being developed first and test cases created later.

In software development, agile is a set of practices intended to improve the effectiveness of software development professionals, teams, and organizations. It involves discovering requirements and developing solutions through the collaborative effort of self-organizing and cross-functional teams and their customer(s)/end user(s). It advocates adaptive planning, evolutionary development, early delivery, and continual improvement, and it encourages flexible responses to changes in requirements, resource availability, and understanding of the problems to be solved.

White-box testing is a method of software testing that tests internal structures or workings of an application, as opposed to its functionality. In white-box testing an internal perspective of the system, as well as programming skills, are used to design test cases. The tester chooses inputs to exercise paths through the code and determine the expected outputs. This is analogous to testing nodes in a circuit, e.g. in-circuit testing (ICT). White-box testing can be applied at the unit, integration and system levels of the software testing process. Although traditional testers tended to think of white-box testing as being done at the unit level, it is used for integration and system testing more frequently today. It can test paths within a unit, paths between units during integration, and between subsystems during a system–level test. Though this method of test design can uncover many errors or problems, it has the potential to miss unimplemented parts of the specification or missing requirements. Where white-box testing is design-driven, that is, driven exclusively by agreed specifications of how each component of software is required to behave then white-box test techniques can accomplish assessment for unimplemented or missing requirements.

In software testing, test automation is the use of software separate from the software being tested to control the execution of tests and the comparison of actual outcomes with predicted outcomes. Test automation can automate some repetitive but necessary tasks in a formalized testing process already in place, or perform additional testing that would be difficult to do manually. Test automation is critical for continuous delivery and continuous testing.

In software engineering, a test case is a specification of the inputs, execution conditions, testing procedure, and expected results that define a single test to be executed to achieve a particular software testing objective, such as to exercise a particular program path or to verify compliance with a specific requirement. Test cases underlie testing that is methodical rather than haphazard. A battery of test cases can be built to produce the desired coverage of the software being tested. Formally defined test cases allow the same tests to be run repeatedly against successive versions of the software, allowing for effective and consistent regression testing.

A test plan is a document detailing the objectives, resources, and processes for a specific test for a software or hardware product. The plan typically contains a detailed understanding of the eventual workflow.

In software engineering, behavior-driven development (BDD) is an agile software development process that encourages collaboration among developers, quality assurance testers, and customer representatives in a software project. It encourages teams to use conversation and concrete examples to formalize a shared understanding of how the application should behave. It emerged from test-driven development (TDD). Behavior-driven development combines the general techniques and principles of TDD with ideas from domain-driven design and object-oriented analysis and design to provide software development and management teams with shared tools and a shared process to collaborate on software development.

Agile testing is a software testing practice that follows the principles of agile software development. Agile testing involves all members of a cross-functional agile team, with special expertise contributed by testers, to ensure delivering the business value desired by the customer at frequent intervals, working at a sustainable pace. Specification by example is used to capture examples of desired and undesired behavior and guide coding.

V-Model (software development)

In software development, the V-model represents a development process that may be considered an extension of the waterfall model, and is an example of the more general V-model. Instead of moving down in a linear way, the process steps are bent upwards after the coding phase, to form the typical V shape. The V-Model demonstrates the relationships between each phase of the development life cycle and its associated phase of testing. The horizontal and vertical axes represent time or project completeness (left-to-right) and level of abstraction, respectively.

There is considerable variety among software testing writers and consultants about what constitutes responsible software testing. Prominent members of the Context-Driven School of Testing consider much of the writing about software testing to be doctrine, mythology, and folklore. Some contend that this belief directly contradicts standards such as the IEEE 829 test documentation standard, and organizations such as the Food and Drug Administration who promote them. The Context-Driven School's retort is that Lessons Learned in Software Testing includes one lesson supporting the use IEEE 829 and another opposing it; that not all software testing occurs in a regulated environment and that practices appropriate for such environments would be ruinously expensive, unnecessary, and inappropriate for other contexts; and that in any case the FDA generally promotes the principle of the least burdensome approach.

The requirement for a system are the description of what the system should do, the service or services that it provides and the constraints on its operation. The IEEE Standard Glossary of Software Engineering Terminology defines a requirement as:

  1. A condition or capability needed by a user to solve a problem or achieve an objective.
  2. A condition or capability that must be met or possessed by a system or system component to satisfy a contract, standard, specification, or other formally imposed document.
  3. A documented representation of a condition or capability as in 1 or 2.

Concordion is a specification by example framework originally developed by David Peterson, and now maintained by a team of contributors, led by Nigel Charman.

Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate. Continuous testing was originally proposed as a way of reducing waiting time for feedback to developers by introducing development environment-triggered tests as well as more traditional developer/tester-triggered tests.

Extreme programming Software development methodology

Extreme programming (XP) is a software development methodology which is intended to improve software quality and responsiveness to changing customer requirements. As a type of agile software development, it advocates frequent "releases" in short development cycles, which is intended to improve productivity and introduce checkpoints at which new customer requirements can be adopted.

Acceptance test–driven development (ATDD) is a development methodology based on communication between the business customers, the developers, and the testers. ATDD encompasses many of the same practices as specification by example (SBE), behavior-driven development (BDD), example-driven development (EDD), and support-driven development also called story test–driven development (SDD). All these processes aid developers and testers in understanding the customer's needs prior to implementation and allow customers to be able to converse in their own domain language.

This article discusses a set of tactics useful in software testing. It is intended as a comprehensive list of tactical approaches to Software Quality Assurance (more widely colloquially known as Quality Assurance and general application of the test method.

References

  1. 1 2 3 4 5 Adzic, Gojko (2011). Specification by example: How successful teams deliver the right software. Manning. ISBN   9781617290084.
  2. ↑ Pugh, Ken (2011). Lean-Agile Acceptance Test Driven Development: Better Software Through Collaboration: A Tale of Lean-Agile Acceptance Test Driven Development. Addison Wesley. ISBN   978-0-321-71408-4.
  3. 1 2 3 Larman, Craig; Vodde, Bas (2010). Practices for Scaling Lean and Agile Development: Large, Multisite, and Offshore Product Development with Large-Scale Scrum. Pearson. ISBN   978-0-321-63640-9.
  4. 1 2 Adzic, Gojko (2009). Bridging the Communication Gap: Specification by Example and Agile Acceptance Testing. Neuri. ISBN   0-9556836-1-0.
  5. ↑ Wynne, Matt (8 December 2015). "Introducing Example Mapping". Cucumber Blog. Retrieved 10 May 2021.
  6. ↑ Crispin, Lisa; Gregory, Janet (2008). Agile Testing: A Practical Guide for Testers and Agile Teams. Addison Wesley. ISBN   978-0-321-53446-0.
  7. ↑ Pattern Languages of Program Design 2. Addison-Wesley. 1996. ISBN   978-0-201-89527-8.
  8. ↑ Ward Cunningham. "EPISODES: A Pattern Language of Competitive Development Part I". C2.com. Retrieved 2014-01-08.
  9. ↑ Martin Fowler 18 March 2004 (2004-03-18). "SpecificationByExample". Martinfowler.com. Retrieved 2014-01-08.
  10. ↑ Beck, K. (1999). Extreme Programming Explained: Embrace Change. Addison-Wesley. ISBN   978-0-321-27865-4.
  11. ↑ Evans, Eric (2004). Domain-Driven Design:Tackling Complexity in the Heart of Software. Addison-Wesley. ISBN   0-321-12521-5.
  12. ↑ Weinberg, Gerald; Gause, Donald (1989). Exploring Requirements: Quality Before Design. Dorset House. ISBN   0-932633-13-7.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.