Tenets of Software Testing

1. A software product’s behavior is exhibited by interactions.
2. There is potentially an infinite number of possible behaviors in software.
3. Some of those behaviors are potentially negative, that is, would detract from the objectives of the software company or users.
4. The potentiality for that negative behavior is risk.
5. It’s impossible to guarantee a lack of risk as it’s impossible to experience an infinite number of behaviors.
6. Therefore a subset of behaviors must be sampled to represent the risk.
7. The ability to take an accurate sample, representative of the true risk, is a testing skill.
8. A code change to an existing product may also affect the product in an infinite number of ways.
9. It is possible to infer that some behaviors are more likely to be affected by that change than others.
10. The risk -of that change- is higher within the set of behaviors that are more likely to be affected by that change.
11. The ability to accurately estimate a scope of affected behavior is another testing skill.
12. The scope and sampling ideas alone are meaningless without empirical evidence.
13. Empirical evidence is gathered through interactions with the product, observation of resultant behavior, and assessment of those observations.
14. The accuracy and speed of scope estimation, behavior sampling, and gathering of evidence are key performance indicators for the tester.
15. Heuristics for the gathering of such evidence, the estimation of scope, and the sampling of behavior are defined in the Heuristic Test Strategy Model.

These tenets were inspired by James Bach’s “Risk Gap” and Doug Hubbard’s book “How to Measure Anything.” Both Bach and Hubbard discuss a very similar idea from different spaces. Hubbard suggests that by defining our uncertainty, we can communicate the value of reducing the uncertainty. Bach describes the “knowledge we need to know” as the “Risk Gap.” This Risk Gap is our uncertainty, and in defining it, we can compute the value of closing it. In testing, I realized we have three primary areas of uncertainty: 1) what is the “risk gap,” or knowledge we need to find out, 2) how can we know when we’ve acquired enough of that unknown knowledge, and 3) how can we design interactions with the program to efficiently reveal this knowledge.

In my own testing experience, I am in a constant cyclical process of defining and reducing uncertainty.