SHARE THIS POST

Yesterday I saw a very interesting post in one of the LinkedIn groups. Someone asked the question, “If you’re in an interview and asked ‘how do you know if your testing is effective’, what would your answer be? I think it is a very interesting and at the same time a very tricky question.  Along those same lines, the questions I ask in interviews are, “How do you measure your success in testing?”, and “How do you know you achieved good results?”.  My statistics show that 90% of the candidates struggle to answer these questions correctly. Here are some of my recommendations to answer this questions correctly.

In order to understand if something is effective you need to measure it against a baseline. You need metrics. The effectiveness of the testing can be measured by the following metrics (there more but the ones below are the ones I personally like the most):

1. Test Effectiveness %– this metrics shows the efficiency of removing defects by internal testing before delivering to customer. Internal testing means (static testing, SAT, SIT, UAT).
2. % of Defect Leakage– how many defects were missed during System Testing. For this you have to measure UAT defects or combined ST + UAT and measure leakage in production. For more information please refer to this link: https://www.qamentor.com/wisdom-center/sample-deliverables-showcase/defect-leakage-analysis
3. Defect Removal Efficiency %– comparison of internally reported defects with total defects, which includes those that customer reported from the production Environment.
4. Test Execution Productivity– productivity of executors, the number of test cases per tester per unit of time. If you are lazy you are not effective. Simple theory.
5. Defect Rejection Rate– this shows how well the testing team understood requirements and how many rejected defects are there due to misunderstanding of requirements.
6. Error Discovery Rate– It is defined as ratio of defects per test case. The higher the discovery rate the more structured and more complete your test case coverage is in general. Yes, it’s possible to argue that if the code is perfect then this metric wouldn’t make sense. I would agree, but it is still a better metric than no metric.

But one thing you must remember is that effectiveness needs to be measured and so we should have metrics that will help us to analyze and come up with decisions about our effectiveness.

About the Author

The author, Ruslan Desyatnikov, is the CEO & Founder of QA Mentor. He created QA Mentor to fill the gap he has witnessed in QA service providers during his near 20 years in QA. With Ruslan’s guidance, unique services and methodologies were developed at QA Mentor to aid clients in their QA difficulties while still offering a high ROI. Ruslan offers monthly seminars aimed at imparting his extensive testing knowledge that can be applied to start-ups as well as large companies. To learn more about QA Mentor and testing services please visit www.qamentor.com or contact Ruslan directly by sending email to rdesyatnikov@qamentor.com