With this model, I aim to describe the fundamental elements that comprise any test.

It represents what I’ve learned over the past 3 years as a tester, learning to think about what I’m doing, avidly reading about testing online, taking the Rapid Software Testing Explored course with James Bach, and joining the Automation Guild with Joe Colantonio.

Testing is the evaluation of system behavior in response to interactions. Figuring out what to test, how to test, what to look for, how to interpret results, and how to apply the knowledge you gain during this process makes testing a difficult, highly skilled, and indispensable activity. 

Evaluation

To evaluate, you have to perceive behavior, decipher its meaning within your context using what oracles are available, and make decisions applying the knowledge gained from the evaluation. Those decisions could be an ad-hoc test idea to be immediately executed, a modification to your test strategy (such as a whole new category of tests), the creation of a bug report, etc. Finally, we need to evaluate if we have gathered enough evidence to demonstrate that the level of risk present in the system is sufficiently low. 

Observation

Observation is ongoing throughout the entire process of testing and can instantly trigger evaluation at any point. We need to notice many details and patterns, as any of them might open a new door to an area of testing and risk. Our experience as humans greatly aids us in observation. The consumers of a system are frequently human, and often share similar experiences with us testers. What sticks out to us will probably stick out to them as well, even if it doesn’t contradict a requirement. We can gain additional insight by studying our users and their contexts. Our experience as testers and the heuristics under our belts also allow for wiser, broader, and deeper observation.

System Behavior

I classify system behavior in two parts, visible outputs and state changes within the system. Visible outputs are usually easier to spot and are obvious candidates for testing. State changes within the system are not always externally apparent and therefore take a higher level of skill to identify, but can be equally risky as any incorrect output. For instance, if you write your name on a piece of paper with a pen, if the pen is the system under test, then the decrease in ink inside the pen is a state change. Wear and tear on the tip is another. Flow of ink out of the well and through the tip is a temporary dynamic state which is not (usually) present after the interaction is complete. Some outputs might be the application of ink and indentation from the pressure of the pen on the surface of the paper. There might be many more manifestations of behavior in this example. Using observation and evaluation, you have to figure out which ones are important.

Interactions with the System

Interactions with systems go on all around us. The pen lying on the desk is an interaction between it and the surface of the desk, gravity, air, etc. Grasping the pen between our hands and bending it until it breaks is another interaction. As testers with limited time in a fast paced work environment, we must wisely choose interactions to study. In order to do so, we can draw upon many resources: looking at a code change, talking to team members, reading comments, tickets, or other documentation, and crucially, starting to interact with the system itself, allowing it to inform and inspire us directly.

Result

We test to try to spot problems which might matter to end users or the business as early in the product development cycle as possible. As we test, the knowledge gained through our constant evaluation of observations allows us to continually fine tune our strategy. Much of the time, evaluation is going on subconsciously. We see it come to the surface when an alarm bell goes off in our heads as we spot something curious which was not expressly looked for. Conscious evaluation then takes over to interpret our observation. The curiosity might be a change in system state or an output which is unexplained within our Current Understanding of the System, or mental model. Conscious evaluation is especially prevalent near milestones of testing activities. During the initial stages of testing, evaluation might be triggered by looking at a set of requirements, contemplating the application as defined within our mental model, and planning scenarios which we imagine would quickly put those requirements through the wringer. It takes place again when determining if the system behavior lines up with relevant oracles, such as the expectations of the team. Sometimes we can perform this entire activity within our minds and spot inconsistencies in the requirements even during planning stages.

Automation

Automation in testing is the scripting of human interaction, observation, and evaluation, so as to be performed by a computer. An automation system has the potential to execute vast numbers of interactions and evaluations within short periods of time. In order to do so, the interactions, observation, and evaluation must all be explicitly scripted and maintained. Scripting and maintenance are development activities which can take away large portions of time from test activities. The maintenance of a test server, services, integration with the cloud, etc., are devops activities which also take away time from testing. In addition, since every aspect of interaction, observation, and evaluation must be so minutely and explicitly scripted, system behavior which occurs outside of those narrow scripts will either prevent their execution or go unnoticed. During the automation system’s interaction with the system under test, the enormous opportunity for unscripted rational observation and evaluation is removed from the equation. Finally, a human must still decide which interactions to automate. To be most effective, those decisions require the testing skills discussed above. Given the constraints and large investment, opportunities for automation should be carefully reviewed by experienced testers and implemented with the same care as are the business’s other products.

AI

My perception of what people usually mean when they say “Artificial Intelligence” is a neural network system which defines behavior through analysis of multitudinous interactions and or data. It is purported to overcome the handicaps of automation by providing a tool which can self-script to interact, observe, and evaluate. The elements of testing are by nature permeated with decision making and evaluation in accordance with implicit oracles, often doused in vagueness, directly related to our human interpretations of meaning. For that reason, I believe the makers of tools claiming to use “AI” should do so in a way which expressly seeks to aid, rather than replace, the tester. In other words, to create tools to increase the powers of a tester to interact with and observe the software. We see this empowering usage already in the gaming industry, where AI-enhanced image recognition can help automate regression testing suites in complex game-worlds.