Three Digestible Diagrams to Describe Exploratory Testing
by Simon Tomes
Bug-magnet For Life
I'd been working with my team for a while, and I felt like I had their respect and trust. We had a high stakes, high complexity project which was at a critical stage and I was finding bug after bug after bug.
"Can someone tell Tomesy to stop testing! He'll just break the software and find more bugs!"
It completely deflated me and made me feel like all I learned was worthless. Deep down I knew my bug-magnet status was uncalled for but I lacked the ability to share why.
I submit it's easy to talk about exploratory testing in a way that others will understand. When we try to impress someone or overly justify why we do what we do, we can cause confusion or even anger and frustration.
For example, when we adjust our words to help others relate to us, we might compromise the value of our explanation. If we argue our point using terminology that doesn't resonate, we could easily cause confusion or perhaps anger.
Why dumb down what we do when we go exploring? Why hold back sharing the value a team, product and customer achieve from an exploratory testing session? It feels like a disservice to our product development community and the customers we're building software for if we don't share authentically and honestly.
The Hard Part
How might you explain exploratory testing so your team and customers have the best chance to realise its true value?
The term "exploratory testing" is often misused and misunderstood. There's a case for deprecating the term and replacing it with "testing." This might help disassociate itself from ad-hoc testing which is often incorrectly used as an interchangeable term.
Ad-hoc testing is typically random and involves "playing around with an application". It lacks focused intention and is usually undocumented. Exploratory testing, on the other hand, uses and captures information to guide, learn and spark ideas with others. Ad-hoc testing might stumble across a bug but it doesn't discover patterns, opportunities and risks like exploratory testing can.
Exploratory testing helps us explore what's in front of us -- far beyond only the application -- to discover what a product is and what it might be. It's a fantastic way for a team to connect over what they're building. It's something your customers will be grateful for as they reach their ever-evolving desired outcome.
In this article, I attempt to dispel some misunderstandings and share three digestible diagrams to explain the value of exploratory testing.
1. Start with the Why
"People don't buy what you do; they buy why you do it, and what you do simply serves as the proof of what you believe." -- Simon Sinek
Simon Sinek urges us to start with the why. Our limbic brain is responsible for feelings such as trust and loyalty.
Understanding why we do what we do helps people recognise you as a kindred spirit or even an inspirational one. Dive straight in with the what and you lose the opportunity to make a deeper connection.
Who are we selling to? What are their motivations? What do they care about? What problems do they face?
Start with the why and provide the best opportunity to connect with others who might not yet understand the value of exploratory testing.
2. An Overwhelming Sense of Terminology
There's an overwhelming sense of terminology used to describe exploratory testing. Here are some great places to learn it:
- James Bach and Michael Bolton have spent countless years convincing people of the value of exploratory testing.
- James and Jon Bach developed Session-Based Test Management (SBTM) as a method for measuring and managing exploratory testing.
- Neill Boyd shares a useful summary of session based testing.
- A charter helps you direct attention. Elisabeth Hendrickson provides a fantastic way to define your charter.
- Heuristics and oracles have a place in the world of exploratory testing -- Katrina Clokie provides a brilliant summary.
- Automation in testing supports exploratory testing (and removes the misunderstanding that automation will replace testing).
- If it's still confusing, Tony Bruce's 10 excellent ways to do exploratory testing badly will help.
There's one underlying word that connects the terminology: Learning.
Exploratory testing is learning. Learning what is and what might be. Learning about our perception of the world, what's in front of us and how that influences the choices we make. It's learning together to build better products. It's creating an environment that enables a continuous "testing is learning" loop.
3. Pay Attention to What You Capture and Share
Exploratory testing does not mean undocumented testing. Nicola Owen reminds us that we should use test documentation to share information and consider who we're sharing with.
So how might we inform others with what we discover? "Pass" and "fail" are popular and fine for checks -- you want a check to assert that something is expected or not -- yet I believe there's no place for pass and fail in the world of exploratory testing.
In the early days of my testing career I would write manual test cases. I'd mark each test case with either a pass or fail. A "passed' test didn't invite anyone to look at it. Instead, focus lay on things that "failed". That always felt a little weird, like a missed opportunity to share and discuss so much more.
These days I group discoveries into four categories: Problems, Questions, Ideas and Praise (PQIP).
Imagine you set aside an hour to explore a shopping basket with multiple devices to discover circumstances where the basket is out of sync.
You notice there's considerable variance in the time it takes to see a recently added item appear on two different devices. Sometimes it's a matter of seconds. Other times it takes up to a minute. This feels like a problem so you note how you discovered it, capture a video and tag the note accordingly.
You're unsure why some category specific items don't sync across devices. Perhaps there's a reason for this and maybe this isn't a problem. You capture this as a question.
An idea pops into your head. Wouldn't it be cool if items subtly animate into (or out of) the basket when picking up and viewing the basket for the first time on another device -- a way to emphasise that the basket is always up-to-date. Time to capture and share this as an idea.
You're impressed that basket items sync across multiple devices regardless of operating system. You feel it's important to share this admiration with your team. It feels great to share some praise!
Time's up and you immediately debrief your testing story. The discussion flows, ideas are exchanged and next steps identified. You bring your team closer together using your product as a conduit.
Notice how you don't need a script to get started. You don't need detailed steps to describe what to do and what to expect. There is no "pass" or "fail". You explore with intent and capture what you discover.
The PQIP approach promotes information capture and collaboration (see diagram). PQIP helps reduce closed conversation. Or put another way, helps open up conversation.
Apply this approach to exploratory test stories, wireframes, working prototypes, paper prototypes, mockups, data flow models, api docs, new features, fixes and more!
Build better products
Let's treat the output of our exploratory testing session as a product. How much value do we want the user of the product to derive? We wouldn't want to use a product which provides little value so why expect that of our customers.
We have a duty to inspire others with an exploratory testing mindset and approach. Let's be bold and share its true value in a way that doesn't overwhelm, frustrate, confuse, devalue, mislead or anger. Let's enhance our exploratory testing techniques by collaborating with others to evolve and improve the practice and mindset.
I'm excited for the future of exploratory testing. We're witnessing the evolution -- or perhaps resurgence or realisation -- of exploratory testing. Teams realise automating all the things is no longer a sustainable strategy, more a distraction from what really matters.
Tools to support exploratory testing are in their infancy and I believe the opportunity to assist and enhance exploratory testing with appropriate tooling is great.
Imagine real-time collaborative chartering and sessions with note taking support. Offer up prompts to spark heuristics and oracles. Why wait to debrief when we can jump on a session together? Something magical happens when people collaborate. Witness a test session unfold and evolve right in front of your eyes.
Incredible products come from thoughtful, collaborative, purposeful, intentional, focused, unscripted, documented and shared exploration.
Customers can't get enough of great products. Exploratory testing is the key and we've only just begun.