“Confidence through knowledge through checks through a tool”
We want automated checks to inspire some confidence in a shorter period of time than could be attained by multiple testers
- Is the time really shorter?
- How do we create a report which communicates what was checked and what failed?
- How long does it take to understand a failure and act on it?
- Do the resources spent on the checks exceed those of testers spending time performing the same regression?
- Is an increase in cost acceptable if the return on the investment is a large reduction in the execution time necessary to inspire confidence?
What knowledge do we need to inspire confidence
The whole question of value of an automated checking framework directly relates to the knowledge that it imparts.
- Do we want general, high level, highest risk knowledge?
- Do we want specific knowledge on features?
- Do we want specific knowledge on end to end workflows for different user types?
What checks are necessary to create that knowledge
The checks we use depend on what knowledge we hope to gain.
The effectiveness of the check directly relates to its design and the design of its framework.
Design is therefore the ground level of the value of an automated solution.
- The design needs to be directly mapped to the knowledge it’s supposed to impart, and have super effective methods for reporting the knowledge.
- The design is based on context of the app, the feature, the available data, environment, to name a few.
- A good design requires both system and business knowledge.
- Design should be as simple as possible to support the checks.
- Simplicity invokes less complexity, increased complexity invokes greater risk of failure.
- Simplicity is quicker, maintainable, fixable, extensible, updateable, cheaper
What tools would be appropriate in our context to build those checks
If check design is ground level, the tool is the basement.
- Its goal is to support the design of the check.
- It needs to be the easiest and most maintainable possible solution that will support the necessary checks’ reporting and execution.
- It should be extensible for future needs. Sufficiently simple, but not limited.