Premise
We can never know if there is no risk since we can never test all software behaviors. I wrote about this idea in the Tenets of Testing.
A huge part of testing is identifying what areas of the product pose significant risk vs which ones pose acceptable risk.
Within those areas, we must choose specific software behaviors (actions, data, conditions) to check, or “sample.” Those sampled behaviors/conditions represent the potential risk for the product.
To estimate the risk areas and create accurate samples we apply sensemaking, estimation, and our gut instinct to what knowledge we have available.
Interactions with the product continually add to this available knowledge, allowing continually more refined estimation and risk assessment.
Below I describe how a tester might test a new work item or “ticket.”
Define the Mission
- Goal: what is the goal of this ticket in terms meaningful to me, my understanding of the product, the business, the real world?
- Risk: what risks might there be as a result of the changes (i.e. business logic, code, infrastructure) necessary to meet the goal?
- Scout: can I get my hands dirty immediately and witness successful (or unsuccessful) fulfillment of major requirements? The information I can quickly gather in this exercise will make further test planning much more accurate.
Product Elements in Scope
What product elements are in scope, that is, what product elements seem to be connected in some way to the goal?
- Features
- Actions
- Business logic
- Data
- Settings/Contexts
- Devices
- Real world conditions
Some elements will be connected much more directly than others. However, in limiting testing to obviously connected features and behavior, we may miss a large range of bugs which occur in less obvious connections within the product.
Variations of Elements
What variations of those product elements should I definitely make sure to check?
List out and organize your ideas. I like to first list the product elements and then list the variations under each element. We’re creating a Product Coverage Outline.
- Positive and negative values
- Boundaries
- Different sequences/workflows/locations
- Devious values, circumventing interfaces
- Much more
Are there variations/product elements that seem a little more removed from the goal, but which appeal to my gut instinct or logic as being a good idea to quickly check?
Interaction Design
How can I interact with the product to efficiently cover combinations of the product element variations that I think are important?
As I execute these interactions, how does the information I’m gathering help redefine my current knowledge?
As I test, I’m gathering 3 main categories of information:
- Potentially curious/negative behaviors
- Information about my chosen scope (I could observe things about the product which logically or intuitively make me realize they also deserve to be checked)
- Information about the effectiveness of my testing: is my uncertainty increasing or decreasing? Why?
Definition of Done
This cyclical process continues until I “feel” confident that the risk is low.
How do I know if my feeling is accurate? To test myself, I try to make a sound argument that I witnessed a sample of system behaviors truly representative of the risk. I can be fooled: but the ability to accurately assess when you’re done is a big part of what makes a good tester.
Oftentimes making this argument to myself reveals new test ideas which should also be checked.
Production Bugs
Production bugs are invaluable “tests” of our testing process.
By isolating the ticket that introduced a production bug and reviewing the testing on the older ticket in light of the new bug, we gain immense insight into holes in our thinking and aspects of product behavior which will make future testing better.