What is Test Efficiency and Test Effectiveness ?

The Primary difference between Efficiency and Effectiveness is:
Efficiency - Productivity metric and Effectiveness - Quality metric.

Let us assume that there are two Test teams with equal number of resources with comparable skill set, functional expertise etc working on the same product, and If Test team "A" has prepared 500 Test cases with agreed quality, for the product in 10 days and for the same product if Test team "B" prepares 500 Test cases in 8 days, then which team can we say is efficient? Definitely Test team "B" is efficient, but there is no guarantee that Test team "B" is effective, we are not sure how many defect can Test team "B" uncover as compared to Test team "A".

Software Test Efficiency:

Test efficiency is not only about test execution alone, but all or most of test activities, like test planning, comprehension, test cases creation, review, execution, defect tracking and closure.
Test Efficiency helps to calculate the efficiency of testing i.e. how many defects were leaked to the customer as compared to number of defects reported by the testing team. Generally almost 10-15 % of defects will be leaked and is considered acceptable. In the recent years, Companies have stared spending huge amount of money for developing quality. Due to this defect leakage percentage has come down to less than 10%.

Here are some formulas to calculate Software Test Efficiency (for different factors):
Test efficiency = (Total number of defects found in Unit + Integration + System Testing) / (Total number of defects found in Unit + Integration + System + User acceptance Testing)
Test Case Efficiency = (Number of Test Cases Failed / Total Number of test cases executed)* 100

How to Measure Test Efficiency?
Following is the method to calculate the efficiency of a testing process based on the number of defects reported by Customer and to the number of defects identified by the Testing Team.

1. Provide Ranking to each Severity.
a. Critical - 4
b. Serious - 3
c. Moderate - 2
d. Minor - 1
2. Collect the list of defects reported by the Testing and Customer based on Severity.
For example:
- The Customer has Reported 1 Critical, 1 Serious, 2 Moderate and 5 Minor Defects. The Testing Team should have identified these defects.
- The Testing Team has Reported 10 Critical, 5 Serious, 10 Moderate and 10 Minor Defects.
- The Test Efficiency is calculated as follows:
Tester's Defects = 4*10+5*3+10*2+10*1 = 85
Customer's Defects = 1*4+1*3+2*2+5*1 = 16
So Test Efficiency is (85/85+16) * 100 = 84.16%

Suppose if the Customer had not identified any defects in above example then the Test Efficiency will be 100%. Consider a small project in which the testing team and customer did not find any defects (Assume you had a good programmer who did unit testing properly) then also the Test Efficiency will be 100%. If the Testing Team Failed to find any Defects and the Customer were finding them then the efficiency of the testing will be 0%.

Software Test Effectiveness :

Test effectiveness of a technique or a system or a team is the ability to find defects and isolate them, from a product or deliverable. Test effectiveness is to ensure quality and close the two quality gaps, namely producer’s quality gap and customer’s quality gap. As definition of quality goes, quality is both process and product quality which is meeting customer requirements and conformance to product specification. These metrics should be quantified, as they closely relate to quality, and for many people the term quality is relative.

Here are some formulas to calculate Software Test Effectiveness (for different factors):
- Test effectiveness = Number of defects found / Number of test cases executed.
- Test effectiveness = (total number of defects injected +total number of defect found) / (total number of defect escaped)* 100
- Test Effectiveness = Loss due to problems / Total resources processed by the system
Test effectiveness can be calculated for specific set of test activities. For e.g. Test preparation efficiency will be the time taken for "X" number of Test cases to be prepared, reviewed and reworked to finalize them. There is a catch here, the quality standards of the test cases should be predefined by using defining standards, as I have tried to state a few of them here.
- The Test cases are complete with respect to Use Cases on which they are based.
- A tester should be able to execute this test case using only this test case and any directly referenced items given the proper software and hardware configuration.
- Test data must be specific.  For example, don’t say “select any menu option to navigate out of current page.”  Say “Click the Back button.”  Don’t leave any test data to the imagination of the tester.
- Usually, each test step should contain a single action.  E.g. “Save” and “Search” functionalities should be split into 2 steps
- Test Case names follow the agreed upon naming convention
- No Grammatical mistakes

What is Test Efficiency and Test Effectiveness ?

Thanks and Regards,

Prashant Vadher | QC Engineer


Tushar said...

Awesome ! Mind blowing ! Crystal Clear explanation ! i have heard so far.

Sofia Hunt said...

Very informative post and it was quite helpful to me. I also wrote something similar lines on software testing effeciency - http://bit.ly/1Ta8Vfz

Post a Comment

Design by Prashant Vadher