Test and evaluation master plan

Last updated

Test and evaluation master plan (TEMP) is a critical aspect of project management involving complex systems that must satisfy specification requirements. The TEMP is used to support programmatic events called milestone decisions that separate the individual phases of a project. For military systems, the level of funding determines the Acquisition Category and the organization responsible for the milestone decision. [1] [2]

Contents

A traceability matrix is generally used to link items within the TEMP to items within specifications.

Definition

The Test and Evaluation Master Plan documents the overall structure and objectives of the Test & Evaluation for a program. [3] It covers activities over a program’s life-cycle and identifies evaluation criteria for the testers. [4]

The test and evaluation master plan consists of individual tests. Each test contains the following.

Test scenario

The test scenario establishes test conditions. This is typically associated with a specific mission profile. For military systems, this would be a combat scenario, and it may involve Live Fire Test and Evaluation (LFT&E). For commercial systems, this would involve a specific kind of situation involving the use of the item being developed.

For example, cold weather operation may require operation to be evaluated at temperatures below C using an environmental chamber. Evaluation of operation with a vehicle interface may require compatibility evaluation with a vibration test. Evaluation of an Internet store would require the system to take the user through a product purchase while the system is loaded with other traffic.

The test scenario identifies the following.

Data collection

Data collection identifies information that must be collected during the test. This involves preliminary setup before the test begins. This may involve preparation for any of the following.

Systems that incorporate a computer typically require the ability to extract and record specific kinds of data from the system while it is operating normally.

Electronic data collection may be started and stopped as one of the actions described in the test scenario.

When data access is restricted, so the transfer of data between organizations may require a Data Collection Plan. This can occur with classified military systems.

Data is analyzed after testing is complete. This analysis is called performance evaluation.

Performance evaluation

Measures of effectiveness are specific metrics that are used to measure results in the overall mission and execution of assigned tasks.

These may have flexible performance limits associated with the outcome of a specific event. For example, the first round fired from a gun aimed using a radar would not impact a specific location, but the position can be measured using the radar, so the system should be able to deliver a round within a specific radius after several rounds have been fired. The number of rounds required to land one inside the limit is the MOE. The radius would be a Measure Of Performance (MOP).

Measures of performance are specific metrics that have a pass or fail limit that must be satisfied. These are generally identified with the words shall or must in the specification.

One type of MOP is the distance that a vehicle with a specific load must travel at a specific speed before running out of fuel.

Another type of MOP is the distance that a radar can detect a 1 square meter reflector.

Measures of suitability evaluate the ability to be supported in its intended operational environment.

As an example, this may be an evaluation of the Mean Time Between Failure (MTBF) that is evaluated during other testing. A system with excessive failures may satisfy all other requirements and not be suitable for use. A gun that jams when dirty is not suitable for military use.

These requirements are associated with ilities.

Purpose

The results of the TEMP evaluation are used for multiple purposes.

Mission planning involves translation of MOEs, MOPs, and MOSs into the following.

ROC and POE are specific expectations evaluated by the TEMP that are used to determine how to deploy assets to satisfy a specific mission requirement. Diagnostic testing ensures these expectations are satisfied for the duration of the mission.

Other usages

The term 'test and evaluation master plan' as a distinct overall guide to the Test and Evaluation functions in a development has also been used by the Australian Department of Defence. [5] Others do use the term, or similar terms such as 'Master Test and Evaluation Plan'.

See also

Related Research Articles

Acceptance testing

In engineering and its various subdisciplines, acceptance testing is a test conducted to determine if the requirements of a specification or contract are met. It may involve chemical tests, physical tests, or performance tests.

Systems engineering interdisciplinary field of engineering and engineering management that focuses on how to design and manage complex systems over their life cycles

Systems engineering is an interdisciplinary field of engineering and engineering management that focuses on how to design and manage complex systems over their life cycles. At its core, systems engineering utilizes systems thinking principles to organize this body of knowledge. The individual outcome of such efforts, an engineered system, can be defined as a combination of components that work in synergy to collectively perform a useful function.

In software quality assurance, performance testing is in general a testing practice performed to determine how a system performs in terms of responsiveness and stability under a particular workload. It can also serve to investigate, measure, validate or verify other quality attributes of the system, such as scalability, reliability and resource usage.

Usability ease of use and learnability of a human-made object such as a tool or device; the degree to which a software can be used by consumers to achieve quantified objectives with effectiveness, efficiency, and satisfaction in a quantified context of use

Usability is the ease of use and learnability of a human-made object such as a tool or device. In software engineering, usability is the degree to which a software can be used by specified consumers to achieve quantified objectives with effectiveness, efficiency, and satisfaction in a quantified context of use.

Systems development life cycle Systems engineering term

In systems engineering, information systems and software engineering, the systems development life cycle (SDLC), also referred to as the application development life-cycle, is a process for planning, creating, testing, and deploying an information system. The systems development life cycle concept applies to a range of hardware and software configurations, as a system can be composed of hardware only, software only, or a combination of both. There are usually six stages in this cycle: requirement analysis, design, development and testing, implementation, documentation, and evaluation.

Department of Defense Architecture Framework

The Department of Defense Architecture Framework (DoDAF) is an architecture framework for the United States Department of Defense (DoD) that provides visualization infrastructure for specific stakeholders concerns through viewpoints organized by various views. These views are artifacts for visualizing, understanding, and assimilating the broad scope and complexities of an architecture description through tabular, structural, behavioral, ontological, pictorial, temporal, graphical, probabilistic, or alternative conceptual means.

Integrated logistic support (ILS) is an integrated and iterative process for developing materiel and a support strategy that optimizes functional support, leverages existing resources, and guides the system engineering process to quantify and lower life cycle cost and decrease the logistics footprint, making the system easier to support. Although originally developed for military purposes, it is also widely used in commercial product support or customer service organisations.

The Federal Acquisition Regulation (FAR) is the principal set of rules in the Federal Acquisition Regulations System regarding government procurement in the United States, and is codified at Chapter 1 of Title 48 of the Code of Federal Regulations, 48 C.F.R. 1.

The Capability Maturity Model Integration (CMMI) defines a Process Area as, "A cluster of related practices in an area that, when implemented collectively, satisfies a set of goals considered important for making improvement in that area." Both CMMI for Development v1.3 and CMMI for Acquisition v1.3 identify 22 process areas, whereas CMMI for Services v1.3 identifies 24 process areas. Many of the process areas are the same in these three models.

The processes of government procurement in the United States enable federal, state and local government bodies in the United States to acquire goods, services, and interests in real property.

Acquisition initiation (ISPL)

Acquisition Initiation is the initial process within the Information Services Procurement Library (ISPL) and is executed by a customer organization intending to procure Information Services. The process is composed of two main activities: the making of the acquisition goal definition and the making of the acquisition planning. During the acquisition initiation, an iterative process arises in which questions about the goal of the acquisition are usually asked. In response to these questions the Library provides details of the requirements, covering areas such as cost, feasibility and timelines. An example of such requirements is the "planning of the acquisition", a component that may also lead to more questions about the acquisition goal.

The Joint Capabilities Integration and Development System (JCIDS), is the formal United States Department of Defense (DoD) process which defines acquisition requirements and evaluation criteria for future defense programs. JCIDS was created to replace the previous service-specific requirements generation system that allowed redundancies in capabilities and failed to meet the combined needs of all US military services. In order to correct these problems, JCIDS is intended to guide the development of requirements for future acquisition systems to reflect the needs of all four services by focusing the requirements generation process on needed capabilities as requested or defined by one of the US combatant commanders. In an ideal implementation of the JCIDS process, regional and functional combatant commanders give early and continuous feedback into the acquisition and sustainment processes to ensure their current and evolving requirements are known and met.

Software security assurance is a process that helps design and implement software that protects the data and resources contained in and controlled by that software. Software is itself a resource and thus must be afforded appropriate security.

Integrated master plan important program management tool

In the United States Department of Defense, the Integrated Master Plan (IMP) and the Integrated Master Schedule (IMS) are important program management tools that provide significant assistance in the planning and scheduling of work efforts in large and complex materiel acquisitions. The IMP is an event-driven plan that documents the significant accomplishments necessary to complete the work and ties each accomplishment to a key program event. The IMP is expanded to a time-based IMS to produce a networked and multi-layered schedule showing all detailed tasks required to accomplish the work effort contained in the IMP. The IMS flows directly from the IMP and supplements it with additional levels of detail——both then form the foundations to implement an Earned Value Management System.

A specification often refers to a set of documented requirements to be satisfied by a material, design, product, or service. A specification is often a type of technical standard.

A strategic technology plan is a specific type of strategy plan that lets an organization know where they are now and where they want to be some time in the future with regard to the technology and infrastructure in their organisation. It often consists of the following sections.

In the United States military integrated acquisition lifecycle the Technical section has multiple acquisition "Technical Reviews". Technical reviews and audits assist the acquisition and the number and types are tailored to the acquisition. Overall guidance flows from the Defense Acquisition Guidebook chapter 4, with local details further defined by the review organizations. Typical topics examined include adequacy of program/contract metrics, proper staffing, risks, budget, and schedule.

An ERP system selection methodology is a formal process for selecting an enterprise resource planning (ERP) system. Existing methodologies include:

The Analysis of Alternatives (AoA) in the United States is a requirement of military acquisition policy, as controlled by the Office of Management and Budget (OMB) and the United States Department of Defense (DoD). It ensures that at least three feasible alternatives are analyzed prior to making costly investment decisions. The AoA establishes and benchmarks metrics for Cost, Schedule, Performance (CSP) and Risk (CSPR) depending on military "needs" derived from the Joint Capabilities Integration Development System process. It moves away from employing a single acquisition source to the exploration of multiple alternatives so agencies have a basis for funding the best possible projects in a rational, defensible manner considering risk and uncertainty.

Competence assessment is a process in which evidence is gathered by the assessor and evaluated against agreed criteria in order to make a judgement of competence. Skill assessment is the comparison of actual performance of a skill with the specified standard for performance of that skill under the circumstances specified by the standard, and evaluation of whether the performance meets or exceed the requirements. Assessment of a skill should comply with the four principles of validity, reliability, fairness and flexibility.

References

  1. "Incorporating Test and Evaluation into DoD Acquisition Contracts" (PDF). Defense Acquisition University.
  2. "Test and Evaluation Master Plan Procedures and Guidelines" (PDF). University of Idaho, Idaho Falls.
  3. "Test & Evaluation Master Plan (TEMP)" . Retrieved 6 Apr 2016.
  4. "Test & Evaluation, Test and Evaluation Master Plan" . Retrieved 6 Apr 2016.
  5. "Test and Evaluation of Major Defence Equipment Acquisitions" (PDF). Australian National Audit Office. 24 Jan 2002. ISBN   0642-80612-8 . Retrieved 6 Apr 2016.