Any test coverage is based on a model of application under test. Without thinking about a model of application – you cannot have test coverage. What is test coverage or coverage in the first place? Coverage is a measure of “having visited/seen” or in the context of testing “how much of application” is tested. The phrase “how much” to an abstract entity like software can be confusing.
Michael Bolton defines Test coverage as “extent” to which we have travelled along/over “some” map of the application/system
RST : Coverage is how thoroughly we have examined the product with respect to some model.
Two keywords here
“Extent”
“Some Map”
Let us use an analogy. Let us say application is represented by a 2 dimensional shape such as rectangle or trapezium. Coverage in this analogy means % of area of the 2D shape that is … say painted with colour red and uncovered area is say in colour white. Here is the act of painting the shape is analogous to covering application through testing. Untested areas of application correspond to unpainted areas of the shape. I know this far more simplistic model to understand the idea of test coverage. But it might be an useful idea to consider when creating our model for test coverage.
Another analogy that might be handy here is considering an application as a forest or lake or an mountain or any geographical area. What would be equivalent of test coverage if your application is a mountain. Simplistically, walking on every inch of the mountain and leave your foot prints. What about considering an application as a forest? Test coverage here would mean walking and getting yourself familiarised with every bit of the area of forest, with all kinds of plants and animals in there and so on.
Real life, non-trivial applications are like wast areas of forest or mountain range.
Consider a popular type of test coverage –
Code Coverage: Here is application code is used as a model. Each line of code is similar to an unit area of say a forest (forest model). So entire application can be imagined to be a collection of lines of code. When we run a test case (or interact with application) – some line of code will be executed (covered or touched in the language of the model). Tools for measuring code coverage (aka test coverage surprisingly) – instrument the application code (add hooks) and report the lines of application code that were “touched” during a test. If we run, let us say, 100 test cases and managed to “touch” 5000 lines of code in a application that has 10000 lines of code – then we say 100 test cases provide about 50% of test coverage. Then we go and inspect those lines of code that were not “touched” by our 100 test cases and develop few more test cases to “increase” the coverage.
This process will be iterated several times until – say we have reached 100% test coverage or near 100% test coverage. At this point of time – the number of test cases that we have – is considered to be providing 100% test coverage – holy grail of testing !!!
There are some nuances of code coverage measured in this way. Let us consider a java class or function as representation of our code. The code will have lines – some are simple declarations of variables, some assignment statements, some conditional statements and some looping statements. Accordingly code coverage is further divided into line coverage, branch coverage, loop coverage and so on. This further level of break up is to get deeper insight into how lines/branches of code are executed and what should be considered as “touched” and how to avoid “multiple” counting of lines of code when same line of code is executed multiple times as test cases are executed on the code.
When a test case is executed against code – execution through a path or network of paths. Through code coverage – you can potentially determine “unreacheable” paths of code. Identifying such paths or potions of code helps the developer to redesign/refactor the code in such a way that unreachable and redundant portions of the code are eliminated.
Static (reviews, inspections and walkthroughs of code and requirements) and Dynamic code coverage:
Other forms of Test coverage:
Test coverage by feature/feature map/set, Use case, Requirement (Black box test coverage)
Test coverage by risks
Test coverage by GUI elements
Test coverage by browser/OS/Device combinations/Customer types/Geographical/Types of instruments/markets etc
Test coverage by End user side Infra details – type of device/memory/OS/Internet speed
What are we missing here?
Leave a Reply