Problem Checking

Michael Bolton’s recent article on test focus has important implications in automated check design:

So rather than focusing on correctness, I’d recommend that you focus on problems and risk instead.

…if you focus on correctness, that’s what you’ll tend to see. You will feel an urge… [to] demonstrate success.

…if you focus on risk and search diligently for problems and you don’t find problems, you’ll be able to make reasonable inferences about the correctness part for free!

This article is pure gold for all kinds of testing, but I’d like to fill in something which Bolton didn’t go into detail about.

You can design tens of thousands of automated UI checks (Selenium, Cypress, etc.) and not be able to prove correctness. You can, however, design just a handful of checks which would prove incorrectness. Here’s how.

Identify critical functions, things that would really matter if they failed. For the time being, ignore the kinds of functions which would be tolerable if they failed, as long as they were fixed pretty fast. For each critical function, design the minimum number of interactions necessary with the system necessary to prove if there were a serious error in that function. Often, for me, this means checking for some value in a response after a web form is submitted. You don’t need to bother writing a locator for the exact element that value is supposed to appear in: if there were an important failure, that value wouldn’t be returned at all.

When “setting the stage” for this check, bear in mind “the stage” itself is not the test subject. You’re writing a check for a problem in a specific critical feature. This check should not fail if there’s a problem in some other feature. Whatever UI elements and features necessary to setup this case should be the subjects of different checks. Keep UI interactions with them to a minimum.

This approach reduces the number of locators you need to use, decreases the length of each check, and keeps the checks isolated and independent from eachother. Nikolay Advolodkin calls this pattern “Atomic test design” and explains it more here.

Leave a Reply

Your email address will not be published. Required fields are marked *