Software Testing and Quality Assurance Rules of Thumb

Prashant Vadher - Software Testing and Quality Assurance Rules of Thumb

Rule of thumb - The earliest citation comes from Sir William Hope’s The Compleat Fencing-Master, second edition, 1692, page 157: What he doth, he doth by rule of thumb, and not by art." The term is thought to originate with wood workers who used the length of their thumbs rather than rulers for measuring things, cementing its modern use as an inaccurate, but reliable and convenient standard.

As testers, we often use rules of thumb throughout a project. For example, we sometimes use total number of expected defects during test planning and then compare actual defects per hour found versus what we would expect, during test execution. Each of these rules of thumb aids us in managing the information we deal with as testers and QA managers.

It would be nice (and useful) to have a collection of these rules of thumb in one place, each documented with examples. And that's my goal: to list as many software testing rules of thumbs in this article. I'll update it each week and re-post it. I look forward to reader comments and additions to this list and hope to make it a reference you each bookmark and refer to often.

THE NUMBER OF WHITE-BOX TESTS REQUIRED TO OBTAIN SUFFICIENT COVERAGE OF A MODULE IS ADEQUATELY DETERMINED BY ITS CYCLOMATIC COMPLEXITY. One common testing strategy, espoused for example by the NIST Structured Testing methodology, is to use the cyclomatic complexity of a module to determine the number of white-box tests that are required to obtain sufficient coverage of the module. In almost all cases, according to such a methodology, a module should have at least as many tests as its cyclomatic complexity; in most cases, this number of tests is adequate to exercise all the relevant paths of the function.

MORE THAN 40% OF ALL SOFTWARE APPLICATIONS ARE RELEASED WITH BETWEEN ONE AND 10 CRITICAL DEFECTS. And managers are fully aware. Three-quarters of deployed apps have between one and 10 critical errors, and again, bosses know about it.

80% OF DEVELOPMENT COSTS ARE CONSUMED BY SOFTWARE DEVELOPERS IDENTIFYING AND CORRECTING DEFECTS. A study commissioned by the United States Department of Commerce's National Institute of Standards and Technology (NIST) found that software defects cost the U.S. economy almost $60 billion annually. The study also found that about 80 percent of development costs are consumed by software developers identifying and correcting defects.

ABOUT 7% OF DEFECT REPAIRS WILL THEMSELVES ACCIDENTALLY INJECT A NEW DEFECT. A bad fix is a secondary defect accidentally injected in a bug repair. In other words, a bad fix is a failed attempt to repair a prior bug that accidentally contains a new bug. On average, about 7 percent of defect repairs will themselves accidentally inject a new defect, although the range is from less than 1 percent to more than 20 percent bad fix injections.

DEVELOPERS INJECT FEWER DEFECTS WHEN DESIGNING (2.0 DEFECTS PER HOUR) THAN WHEN THEY DESIGN WHILE CODING (4.6 DEFECTS PER HOUR). The practice of designing while coding is error prone. From data on 3,240 programs written in Personal Software Process (PSP)SM courses, the SEI has found that experienced developers inject fewer defects when designing (2.0 defects per hour) than when they design while coding (4.6 defects per hour). If you want low-defect designs, you must produce those designs, instead of just creating them while coding.

NONREPRODUCIBLE OR AD HOC TESTING IS OF LITTLE OR NO USE. One of several basic rules of testing compiled from Myers (1976), Priest (1988), and Pullum and Doyle (1998).

CHAOS WITH NO MEASURES TO TELL THE ORGANIZATION WHAT'S GOING ON LEADS TO … 3 IN 10 PROJECTS BEING CANCELED, 5 IN 10 OVERRUNNING SCHEDULE AND/OR BUDGET BY NEARLY DOUBLE, AND ONLY 1.6 IN 10 MAKING THEIR DEADLINES AND BUDGETS. Many organizations struggling with process chaos claim that they don't have time for software measures, which are sometimes perceived as a bureaucratic Dilbert exercise. But getting metrics in the face of deadlines, and using them to manage that pressure and reduce the chaos, is key to successful software development.

THE AVERAGE TIME TO FIND AND FIX DEFECTS IS 10 TO 20 HOURS. "One way to think about quality is to consider how the process would change as a function of defect fix times. For example, programmers generally think that it takes only a few minutes to fix defects in test. They base this on their experience with most of the defects they find in unit testing. In system test, however, the time to find and fix defects typically extends to many hours or even days. While most of these defects are fixed rather quickly, some take much longer. The average time to find and fix each defect is generally 10 to 20 or more hours."

ABOUT 80 PERCENT OF AVOIDABLE REWORK COMES FROM 20 PERCENT OF THE DEFECTS. That 80 percent value may be lower for smaller systems and higher for very large ones. Two major sources of avoidable rework involve hastily specified requirements and nominal-case design and development, in which late accommodation of off-nominal requirements causes major architecture, design, and code breakage.

EXPECT BETWEEN 6.5 AND 9 DEFECTS PER KLOC DURING SYSTEM TEST. Based on data from the Jet Propulsion Laboratory (JPL) from an internal report that listed all the defects found in the system testing of three spacecraft. These data were taken after the programs were developed, compiled, unit tested, and integration tested.

THE RELATIVE COST OF FIXING DEFECTS DURING TESTING IS 15 TIMES GREATER THAN DURING DESIGN. As reported by the Systems Sciences Institute at IBM. Additionally, the cost to fix an error found after product release was four to five times as much as one uncovered during design, and up to 100 times more than one identified in the maintenance phase.

REQUIREMENTS, DESIGN, AND CODE REWORK COSTS 40 TO 50 PERCENT OF THE TOTAL COST OF SOFTWARE DEVELOPMENT. "If you can prevent defects or detect and remove them early, you can realize a significant schedule benefit. Studies have found that reworking defective requirements, design, and code typically consumes 40 to 50 percent of the total cost of software development (Jones 1986). As a rule of thumb, every hour you spend on defect prevention will reduce your repair time from three to ten hours. In the worst case, reworking a software requirements problem once the software is in operation typically costs 50 to 200 times what it would take to rework the problem in the requirements stage (Boehm and Papaccio 1988). "

TESTERS WILL FIND, ON AVERAGE, 1 DEFECT EVERY 4 HOURS DURING BLACK BOX TESTING. Based on average defect detection rates for 6 different JPL projects as reported in "An Analysis of Defect Densities Found During Software Inspections," Proceedings of the Fifteenth Annual Software Engineering Workshop. More information can be found in Grady's Practical Software Metrics For Project Management And Process Improvement.


TESTING DONE WITHOUT MEASURING CODE COVERAGE TYPICALLY EXERCISES ONLY ABOUT 55 PERCENT OF THE CODE. Robert Grady of Hewlett-Packard reports that testing done without measuring code coverage typically exercises only about 55 percent of the code (1993). See page 615 of Code Complete.

Thanks & Regards ,

Prashant Vadher | QA Engineer


kalyani said...


In hyderabad we have different institutes for training of software testing but if you want to go for good institue i will suggest for reliance global services. it is located in ameerpet lane opp to chandana brothers. for more details call them on 040-66666165/156. they are authorised training partners of QAI. so they are providing software testing with certifications,

software test consulting said...

thank you for sharing with us your professional knowledge! it's very helpful explanations that will solve me unpleasantness. thanks again!

viji said...

Thank you for the info. It sounds pretty user friendly. I guess I’ll pick one up for fun. thank u

Amitysoft Software Testing Training Institutes in Chennai

sundara rami reddy said...

good info. mobile repairing course in hyderabad

Post a Comment

Design by Prashant Vadher