By Nicola Lindgren
A while back, I introduced my colleagues to exploratory testing. Slowly but surely, over the next two years or so, we established it as a practice. In this article, I’ll share with you what I learned.
First off, I seriously underestimated how long it would take and how hard it would be, first for people new to exploratory testing to understand it, and then to implement it in our development lifecycle.
To set the scene:
-
The project was going through an Agile transformation
-
Our development group had multiple Scrum teams with several testers apiece, most of whom were based onsite
-
We worked for a large, multinational organisation
Clearly Define And Communicate What You Mean By “Exploratory Testing”
When I first started talking about exploratory testing, many team members said that they were already doing it.
The thing is, we weren’t actually talking about the same thing. Based on our discussions, I found people thought of ad-hoc testing as exploratory testing. But when I said “exploratory testing” I meant this:
Simultaneous learning, test design and test execution.
To be clear, I’ve consistently used Session-Based Test Management (SBTM) as a way to manage my exploratory testing. With SBTM, your testing is split into distinct sessions where you specifically dedicate time to testing different areas of the application based on charters and goals.
Since I used only SBTM for exploratory testing, for me the terms SBTM and exploratory testing had become interchangeable.
Addressing Concerns
I found that many testers were worried about the effects of exploratory testing on their test effectiveness:
“But how will we know what we have tested?”
“Are we allowed to do that?”
“How do you know what to test if you don’t have test cases?”
These concerns fell into one of three main categories:
-
Knowing what happened
-
Getting into trouble
-
Not sure what to do for test preparation
Let’s look at each of these step by step.
Knowing what happened
“But how will we know what we have tested?”
There’s a common misconception that if you do exploratory testing then you are not documenting anything. But when I explained that we would actually be taking detailed notes of what we covered during our testing, these concerns were allayed.
Getting into trouble
“Are we allowed to do that?”
A few testers were genuinely curious about exploratory testing, but they didn’t want to get into trouble. Fortunately, I had the support of the test manager on the project, so I explained to them that she was on board for this.
Not sure what to do for test preparation
“How do you know what to test if you don’t have test cases?”
While it was fairly clear for everyone what “traditional” test preparation involved (writing test cases), it took a bit of time to explain how you do test preparation for exploratory testing.
I explained the two main elements: test data and writing test charters, which contain missions and goals that help guide your testing.
Preparing test data was fairly straightforward, as it was already something that everyone was doing. But writing test charters was an entirely new concept, and to teach people what to do, we held workshops.
Organising Workshops
I found that workshops were a great way to give people a chance to learn about exploratory testing, try it out, and ask questions throughout the sessions.
Conducting workshops is not my forte, so I asked the test manager to hold these workshops, and she agreed. I supported workshop scheduling by looking up meeting rooms and sending invitations.
An Alternative To Test Cases
The test manager and I didn’t want to force people to do exploratory testing. When I first came up with the idea to introduce exploratory testing to people on the project (and later throughout the organisation), I wanted to offer the idea as an opportunity to learn something new, not as a mandate.
We did mandate one practice, however: we told everyone that they could test either with scripted test cases or exploratory testing, but if they practiced exploratory testing they had to use SBTM. The rationale here was that we wanted to avoid a return to the old habit of testing ad hoc and calling it “exploratory testing.”
Outcome And Feedback
A few testers fully embraced exploratory testing and never looked back. Here are a few quotes from some of the testers who embraced exploratory testing:
“By doing exploratory testing here it doesn't restrict my testing. It invites me to investigate further and also be able to pinpoint problems better. This leads to better and more precise defects being created also….
Another benefit that we've seen with ET [exploratory testing] is that we can feel more confident about the testing that has been done. Usually the checks (test cases) that have been created before, only covers the minimum required. Usually the checks are created directly from acceptance criteria and also only covers those.”
“Exploratory testing gave me more freedom to think more, analyse more and test more. So it helped me to find issues earlier and deliver products with better quality.”
More testers, however, preferred scripted test cases. While they seemed happy to try something new, they also seemed to place more trust in test cases.
An unanticipated benefit was that I was invited to speak to other people throughout the organisation about exploratory testing. I shared my ideas on the benefits of the practice and how we introduced it in our project. It was an exciting opportunity to educate my team and then to exchange thoughts and ideas with other colleagues.
Further Reading
To Boldly Go: Taking The Enterprise On A Journey To Structured Exploratory Testing with Aaron Hodder
Author Bio
Nicola Lindgren is a Senior QA Engineer / QA manager at ustwo. She’s the co-founder of WeTest Auckland (now Ministry of Testing Auckland) and the founder of Stockholm Software Testing Talks. She’s also been a frequent co-instructor of the BBST Introductions courses.
In the past, she has worked on projects in a wide range of industries including trade, education, online and card payments, and retail and e-commerce. She has worked in Agile (Scrum, Kanban), continuous delivery, and waterfall environments.
If you want to read more of her thoughts on software testing, check out https://nickytests.blogspot.com.
