Testing As An Art

“The ultimate goal of art is expression and communication, it also is experimentation and innovation, exploration of the existing state and capturing it to preserve and show to others. The same principle applies to testing. We conduct experiments to explore the product AND to share the results with the stakeholders.”

Creative Minds In Visual Art And In Test

As far back as I can remember, I have always been interested in visual art. I especially loved drawing with all kinds of media, from charcoal to oil. I begged my mom to enroll me in art school, but that didn’t happen… at least not while I was a kid. Instead, I became an IT professional, and now I work as a quality engineer. No regrets! However, my love for fine arts kept calling – and last year I finally signed up for an art class. 

To my surprise, I noticed a lot of similarities between drawing and painting and testing software. Neither painters nor testers have fixed rules for how they do their work. Instead, they experiment a lot, exploring how to achieve an optimal result. From contemplating a blank piece of paper in front of us to unveiling our work to the world, we aspire to arrive at a great product.

Confronting The Blank Page: Creating A Strategy

Artists never start by drawing objects right on an empty canvas. First, they do some underpainting: providing a base color that helps to define contrast and tonal values. This gives the finished work more depth and dimension. It can also be used as an outline for future color placement and composition. 

For testers, the “underpainting” is the test strategy, or the set of ideas that guide our test design. A test strategy can contain: 

• Roadmap: the proposed timeline for implementation, including internal and public release dates and the scope of implementation
• Project specifics: product teams included in development (including dependent teams), functional specification, prototypes and design guidelines, person responsible to provide use cases, links to track development and defects, product elements (sections, features)
• Communication plan: channels for communication (for example, Slack), regular sync (discussion on current state of product, issues to be resolved, state of testing)
• Product outline: the scope of the product under test
• Quality criteria: data consistency, usability, performance goals, and so forth. Must be aligned with stakeholders and goals
• Risk assessment: critical paths, product and process risks, and plans for mitigation
• Test data and tools: environment used for testing, access to the product – specific URL/feature flag to turn on, list of testing accounts with list of their specifics, various tools used for testing purposes – test scenarios, automated tests, API collections, etc.
• Testing activities: Types of testing that will be performed, such as automated API tests or exploratory testing, with charters specified.  For more info on charters, see: ExploreIt! by Elisabeth Hendrickson.

The content of the strategy document will vary depending on the context. The example above follows a format that has worked well for my team over time. Thank you to Irja Straus for compiling it. 

Both underpainting and test strategies help us to visualize the desired result, to see overall what needs to be done and how.

Implementation Time: Adding Layers And Staying Focused

Choosing Your Tools

When the overall plan is clear, with well-formed ideas, it's time to start implementing it. Just as an artist will mix colors and use different brushes, testers will prepare and use their tools.

It is not possible to name every tool that testers use in their work. The tooling is driven by the context and is tailored to the activity we are performing: 

• Test design and managing the testing work
  • Mind maps and other tools for developing test strategies
  • Test data artifacts – spreadsheets, various generators, anonymization tools 
  • Visualization tools to facilitate identifying risks and dependencies 
  • Tools to gather and share the information about the product, such as a Confluence with how-to's and other documentation
• interaction with the product 
  • Developer tools within browsers
  • Automated tests
  • Continuous integration / continuous development (CI / CD) tools
  • Mocks and stubs for simulating states and behavior
  • Tools for API testing
  • Tools for non-functional testing (such as load or security testing)
  • Monitors
• Reporting 
  • Screen recorders
  • Reporting tools
  • Coverage tools

This list is not exhaustive, but it does give a fair overview of what I as a tester use on a daily basis. 

Once the tools are ready, the artist, or the tester, creates magic. Style and technique differ from person to person, of course. But in any case, layer by layer the picture appears, from broad brushstrokes to small finishing touches.

A GIF of a still life painting being created, layer by layer

Working With The Whole Canvas

From the testing perspective, you can add layers by varying the complexity of your testing charters. It would be logical to start from testing the system as a whole – does the product or feature solve the user's need we want to solve? Is the "happy path" working?  It might sound obvious, but it is quite easy to fall into the trap of going into non-critical details (focusing on the component testing, or verifying the UI is pixel-perfect) without testing the critical path. 

Let me give you an example. We once planned to release a new feature, with support for variables inside the message text to personalize it. In other words, you could now write something like “Hello {firstName}!”, and the user’s first name would appear. As I prepared the test strategy, I identified the possible risks. One part of the system that involved some complex processing seemed to me to be a probable source of issues. So when the feature was ready to test, I checked that part first, and found a bug. How proud I was! My preparations worked out well, my gut feeling was vindicated, my knowledge of the system and experience in testing had paid off. So I kept probing that part of the system, and I found a few more issues. However, the most important part remained untested. I didn’t verify that the principal functionality of the new feature, sending a message, was working! 

Since then, I always start my testing by checking the main scenario. And only after this is done do I verify that the system is still backward-compatible, that integration between different parts is working, that components are implemented according to the design and process the data as they should, and much more… 

You can also think of the various levels of testing — acceptance, system, integration, and unit — as layers, moving from high-level validation to finer details. 

The Finished Picture: Knowing When To Put Down The Brush

In painting AND in testing, you need to know when to stop. 

An image is of a painting depicting a still-life composition. It features a purple top in the centre. Below are two cylindrical containers painted in blue and purple. The background on which these objects rest is a green draped cloth.

Remember the Mona Lisa?  Leonardo da Vinci spent almost seventeen years working on it till he died, and he never called the painting perfect and finished. 

Artists can work on one painting for decades, and testers can spend infinite time on one version of one product. The closer you look at the implementation, the more concerns you might have. There’s always one more question to ask, it seems. But if you have a hard time stopping, you might be testing one small aspect of the product and neglecting the system as a whole. 

The secret here is easy – just take a step back and pause. This will give you some perspective. And ask yourself: "Did I get what I was striving for?" 

In Service To Others: Sharing Your Artwork

Every work of art has its purpose. Nobody spends time creating something just to put the result into the drawer and forget about it. We want to show our craft. Behind every creation or production there is an idea we want to share with the world. 

The ultimate goal of art is expression and communication, it also is experimentation and innovation, exploration of the existing state and capturing it to preserve and show to others. The same principle applies to testing. We conduct experiments to explore the product AND to share the results with the stakeholders. 

An image of a painting in progress, placed on an easel. The painting depicts a still-life setup, which is physically present behind the canvas.

When you look at a piece of art in a gallery, it can be hard to comprehend the meaning of all the symbols on the canvas. But there are often interpretations and art criticism that help translate thoughts and ideas the creator has put into the piece. 

Similarly, test results need to be conveyed effectively to the stakeholder audience. Behind the number of issues found, test coverage, charts, and tables in the test reports there is a story that testers are the best qualified to share.

For More Information

• Creating Test Strategies Teams Will Read, Thomas Shipley
• Three Digestible Diagrams to Describe Exploratory Testing, Simon Tomes
• How to Get Automation Included in Your Definition of Done, Angie Jones