Introduction
James Bach defined the “risk gap” as what we need to know before we ship a product. Testing seeks to close this gap, to find what we need to know. Well, how do you find out what this gap even is? How do you find out what you don’t even know you need to know? We can never quite know for certain: but we can gain a level of confidence through estimation and assessment, reasoning and evidence.
Summary of the Method
“The purpose of testing is to close the risk gap.”
In order to close the gap, we have to have some idea of its boundaries. Therefore, testing can be described in terms of 1) identifying the dimensions or boundaries of the risk-gap (also known as scope) and 2) determining if the gap is sufficiently closed.
Without defining boundaries for the risk gap we can’t determine when it’s closed. In a similar way, if I wanted to measure a garden, I’d need to know where the edges of the garden are. In the beginning those boundaries are rough estimates. We can make them more accurate as we gain new knowledge through testing.
To close the gap we need to determine the questions (tests) which will give us sufficient evidence and reason-based confidence that the gap we defined is accurate and actually closed to a sufficient degree (where “sufficient” depends on context, client, usage, etc.)
Steps
1) Estimate the dimensions of the risk gap (testing scope) given current knowledge.
Using our current knowledge we estimate a testing scope. What are we going to test and how? What features, data, and conditions do we think we need to check? This scope is the boundaries or dimensions of the risk-gap. The scope, or gap, becomes more accurate as we gather new information through testing.
Current knowledge comes in several primary buckets:
- Claims
- Code
- Historic knowledge, like domain expertise
- Testing skill
Using that knowledge, we estimate what dimensions of the product need to be tested. We know immediately we’ll need to check the claims, and probably will need to check them under different conditions. We might be able to expand or narrow the scope from analyzing the code change. Domain expertise includes knowledge of how the product is structured, causing us to estimate that if a certain feature is affected, another feature which involves the same data could be affected. Our testing skill will help us recognize the relevancy of different product dimensions and how they relate to claims and code. If you have little domain knowledge, you can start out with a playful product survey to get your bearings.
Examples of high-level dimensions which form the risk gap are:
- Area (parts of the product, different features)
- Functions given to the user by the application
- Data
- Conditions: users, permissions, time, etc.
- See HTSM for a list of many more dimensions
2) Design your test strategy to quickly shed light on the gap
What interactions with the product will help us rapidly assess the accuracy of our estimates? What interactions will reveal the knowledge we need to know before we ship?
The HTSM also lists many test strategies which, through experience and trial and error, a tester will know to apply to certain situations.
Keep in mind that testing has 3 purposes, all related to gathering knowledge:
- Shedding light on the accuracy of the estimated risk gap/scope
- Investigating and defining potential violations of pertinent oracles (finding bugs)
- Providing feedback on the efficacy of the test strategies being used (and testability of the product)
3) Reassess and Repeat with new knowledge
As your knowledge increases through testing you can make the risk-gap boundaries more accurate. (wider than you thought, smaller than you thought, more specific than you thought, etc.)
As you experience the product you will notice potential violations of pertinent oracles, which will indicate higher risk and require more testing.
Similarly your experience testing will indicate the effectiveness of your strategy and the testability of the product.
When combined, all 3 categories of knowledge gained through testing will make your subsequent test activities more effective (i.e. gathering more of more pertinent information)
You repeat this cycle until your reason demonstrates that the evidence gathered is sufficient to close the risk gap given the context and purpose of the testing.
4) Constantly assess your confidence on the dimensions and status of the risk-gap.
You need both evidence and reason to judge the dimensions and status.
My gut often gives me a feeling that more needs to be done or that I’ve tested enough.
You need to qualify that feeling by logically connecting the evidence gathered to the nature of the product/product-change or reason that you’re testing.
5) Testing skill improves this process
Over time as you get more experience testing, you become better at estimating scope, at recognizing test techniques to use, at using tools to help you, at using the smallest number of tests to extract the largest amount of information, at recognizing common problems or ways that you can be fooled.
Artifacts of Testing
Notes, mind maps, test cases, etc. are all means to document and track our journey to help us stay organized and focused. Early on when the amount of unknown is great, it may be a waste of time to create extensive documentation. A few early tests may reveal information which invalidates our documents or reveals whole other aspects that require us to start over anyway.