Quality can be difficult to measure. How do you know how effective testing has been without some manner of measurement and the means to interpret it? QA Mentor has developed a Defect Leakage Analysis document that uses measurements from an industry standard Test Efficiency Indicator (TEI) as a mechanism to measure defect leakage. Occasionally referred to as Defect Removal Efficiency, this measures how effective your testing cycle was, the extent of the coverage, and how many defects were missed and later identified in the UAT or production environment. This measurement is essentially the fruits of the QA Team labor.

QA Mentor’s SLA for defect leakage is 5% or less. This means that our TEI is 95%. In the simplest terms, our complete test coverage will reveal all defects 95% of the time. With our unique test processes and methodologies, we are continuously working towards quality process improvements and developing ways to avoid costly production mistakes. The other 5% go through a detailed internal review process to help us understand how and why the defects were missed. We determine the root cause for each defect and outline them in the Defect Leakage Analysis document. The causes are generally one of the following:

Calculating the TEI

The Test Efficiency Indicator is calculated using a defect severity point system combined with the total defects counts in System Testing vs. UAT. All defects logged are assigned a severity that translates into a point ranking

All defects found during the System and UAT testing cycles are then counted and the corresponding point ranking calculated. Any defects found in UAT that cannot be replicated in the original test environment are eliminated from the equation. We then take the point total for the number of defects found in System Testing and divide it by the point total for defects found in both System and UAT testing. The resulting percentage is the representative efficiency of our testing process.

TEI equation: SIT Points / SIT + UAT Points = TEI

Generally, good testing processes have roughly a 90% TEI, with only 10-12% defect leakage. However, as stated above, QA Mentor aims higher than that for a 5% defect leakage.