Defect Leakage Analysis
QA Mentor has developed a Defect Leakage Analysis document that uses measurements
from an industry standard Test Efficiency Indicator (TEI) as a mechanism to measure defect leakage.
Certified QA Professionals
Quality can be difficult to measure. How do you know how effective testing has been without some manner of measurement and the means to interpret it? QA Mentor has developed a Defect Leakage Analysis document that uses measurements from an industry standard Test Efficiency Indicator (TEI) as a mechanism to measure defect leakage. Occasionally referred to as Defect Removal Efficiency, this measures how effective your testing cycle was, the extent of the coverage, and how many defects were missed and later identified in the UAT or production environment. This measurement is essentially the fruits of the QA Team labor.
QA Mentor’s SLA for defect leakage is 5% or less. This means that our TEI is 95%. In the simplest terms, our complete test coverage will reveal all defects 95% of the time. With our unique test processes and methodologies, we are continuously working towards quality process improvements and developing ways to avoid costly production mistakes. The other 5% go through a detailed internal review process to help us understand how and why the defects were missed. We determine the root cause for each defect and outline them in the Defect Leakage Analysis document. The causes are generally one of the following:
Missing test cases to cover appropriate condition
Poorly designed test case due to misunderstood requirements
Incorrect deployment in UAT or Production
Incorrect test data used
Calculating the TEI
The Test Efficiency Indicator is calculated using a defect severity point system combined with the total defects counts in System Testing vs. UAT. All defects logged are assigned a severity that translates into a point ranking
Critical = 4
Serious = 3
Medium = 2
All defects found during the System and UAT testing cycles are then counted and the corresponding point ranking calculated. Any defects found in UAT that cannot be replicated in the original test environment are eliminated from the equation. We then take the point total for the number of defects found in System Testing and divide it by the point total for defects found in both System and UAT testing. The resulting percentage is the representative efficiency of our testing process.
TEI equation: SIT Points / SIT + UAT Points = TEI
Generally, good testing processes have roughly a 90% TEI, with only 10-12% defect leakage. However, as stated above, QA Mentor aims higher than that for a 5% defect leakage.
Please complete the form and one of our QA Expert Specialists will be in contact within 24 hours.
Alternatively, drop us an email at firstname.lastname@example.org or give us a call at 212-960-3812