Defect criticality

Defect criticality is a metric used in software quality assessment to quantify the potential impact of a software defect. It is defined as the product of three factors: severity, likelihood, and class.

Each defect must first be assigned a class (type of defect), and then its severity/impact and likelihood/visibility are scored within that context. The class identifies what kind of issue it is (e.g., security, performance, usability), while severity and likelihood describe how bad and how visible it is.

Severity/impact

Severity measures the degree to which the defect affects users or system operation. When determining severity, consider the applicable Class of defect (see below) and how strongly it affects the user experience or business outcome.

• 0 - Affects critical data or functionality and leaves users with no workaround
• 1 - Affects critical data or functionality and forces users to employ a workaround
• 2 - Affects non-critical data or functionality and forces users to employ a workaround
• 3 - Affects non-critical data or functionality and does not force users to employ a workaround
• 4 - Affects aesthetics, professional look and feel, “quality,” or “usability”

Likelihood/visibility

Likelihood measures how often or widely the defect is encountered by users. As with severity, consider the Class of defect when assessing this factor.

• 1 - Seen by all or almost all users who use the application (≥95% of users)
• 2 - Seen by more than two-thirds of the users (>67% and <95%)
• 3 - Seen by about half the users (>33% and <66%)
• 4 - Seen by about one-third or fewer users (>0% and <32%)

Class of defect

The class defines the nature or domain of the defect. It provides the context for evaluating both severity and likelihood. For example, a “Class 0” (stability or security) issue with a minor visual symptom should still be treated more seriously than a “Class 4” (cosmetic) issue with identical visibility.

Class 0

Critical system-level defects:

• Stability, reliability and availability
• Security
• Legal (liability, ADA, copyright)
• Testability
• Storage (data loss or corruption)

Class 1

High-performance or scalability defects:

• Performance and efficiency (use of memory, disk, CPU)
• Scalability

Class 2

Functional or logical correctness defects:

• Functionality
• Logic or calculation errors
• Compatibility
• Interoperability

Class 3

User experience and workflow defects:

• Usability
• Learnability
• Readability
• Documentation
• Consistency
• Workflow (“feel”)

Class 4

Presentation and cosmetic defects:

• Typographic or grammatical errors
• Aesthetics
• Appearance or visual cosmetic issues

Assessing the criticality score

The criticality score is computed as:

${\displaystyle Criticality=Severity\times Likelihood\times Class}$

Interpretation of results:

• 0–2 = Critical
• 3–9 = Major
• 10–20 = Medium
• 21–64 = Low

Example calculation

A defect causes data loss in a critical process, is visible to most users, and falls under Class 0 (stability):

• Class = 0 (stability)
• Severity = 0 (critical data loss, no workaround)
• Likelihood = 2 (seen by >67% of users)

${\displaystyle Criticality=0\times 2\times 0=0}$ → Critical defect.

Another example: A cosmetic typo on a rarely used screen:

• Class = 4 (cosmetic)
• Severity = 4 (minor aesthetic issue)
• Likelihood = 4 (seen by <32% of users)

${\displaystyle Criticality=4\times 4\times 4=64}$ → Low criticality defect.