Benefits Of A Bug Bash - And How To Run One

By James Espie

Exploration

If you were to try and think of some great explorers - who comes to mind? Maybe Tintin, the famous cartoon journalist, who explores the globe uncovering mysteries. Or The Doctor (from Doctor Who), The Doctor is an explorer of space and time, saving the universe from one disaster after another. What about Dora the Explorer? She teaches children to spell by embarking on quests and overcoming obstacles.

There’s something to note about each of these great explorers - they never explore alone! The Doctor almost always has a companion or three along for the ride. Tintin always has his faithful dog Snowy, and a cast of other characters that help him on his journey. Even Dora the Explorer always has her favourite monkey, Boots to keep her company! Exploration is seldom a solo activity. So when it comes time to do exploratory testing, you shouldn't go alone! A Bug Bash is one way you can enlist the help of others in your exploration.

What's a Bug Bash?

A Bug Bash is when people from across an organisation come together to test a feature or an application. A good Bug Bash includes a diverse set of people that come from a range of teams and disciplines. The exact shape of a Bug Bash will depend on the context, size and structure of an organisation. But a typical example would be 10-15 people, in a room together for an hour. Each person brings their own device, and spends the time exploring the feature.

A Bug Bash will consists of a  mix of people, including:

  • People that helped build the feature being tested

  • People that haven't worked on the feature at all (especially good if they've never seen it before)

  • A range of people from different disciplines. That means testers, developers, product owners, designers, sales and support. And anyone else you can think of!

  • A diverse set of people. People of different genders, social and ethnic backgrounds, physical abilities, age ranges, and more, all bringing unique perspectives to the table.

What makes Bug Bashes so useful?

So - what makes a Bug Bash such a useful exploratory testing technique?

Snow-blindness

When a development team spends a lot of time working on a feature, they can become 'blind' to some of its quirks or behaviours. There can be very obvious bugs that don't seem so obvious any more, because the team has become so used to seeing them. As an example: when starting on building a feature, a team notices that their UI is slightly broken. But, as there is a workaround for the broken UI, they keep working around it. As time goes on though, they become so used to the workaround, that they simply stop being aware of it. The workaround becomes the norm, and they no longer 'notice' it as a problem. This is how development teams can become 'blind' to bugs. At a Bug Bash, people with fresh eyes can come in and call out things like this.

Other peoples experiences

Everybody in a team, and an organisation, will have a different set of experiences that make up their view of the feature. Bringing these different experiences together can uncover new bugs! Let's say there's a Bug Bash to test a new shopping cart feature, built by the 'shopping cart' team. One of the attendees is from a team that didn't work on this feature - the 'payments' team. The payments team have been working on an entirely different feature - a new payment gateway. The person from the payments team has experience with payment gateways. Their exploration could naturally lead them to explore this. How does the new shopping cart behave with the new payment gateway? There's a reasonable chance there might be bugs here. It's also likely that nobody in the shopping cart team thought to test this yet!

Other people’s perspectives

People from different disciplines also bring different perspectives to the table. Quality can mean different things to different people. The great thing about a Bug Bash is that it brings these perceptions of quality together. This can lead to good discussion, or even a debate, about what may or may not be a problem. Imagine an issue where there is a screen that has two 'save' buttons that perform the same function. But, one of the buttons causes the app to crash, and the other saves as expected.

A tester could see this and determine that there is a bug with the button that crashes the app - the app shouldn't crash. A developer's perspective might be different. We have two buttons that should function identically but don't. To a developer, this might indicate a problem with the underlying code. Both buttons should be calling the same function - but this makes it clear that they're not. A designer's perspective might be different again. To them, the bug might be that there shouldn't be two buttons on the same screen that perform the same function!

Having all these perspectives brought to the table helps give direction, and clarity. By bringing them together, you can find better solutions for problems.

How to run a Bug Bash

It can be challenging to bring all these people together, and have a Bug Bash run smoothly. So here's some advice that might help prepare!

Be prepared!

It's important to think about what's needed for a Bug Bash ahead of time. There might be a time when a feature being tested is specific to mobile devices. If the organisation has a device library, it's good to make sure all the test devices are charged and available in time for the Bug Bash. If there's test data that needs setting up, it's important to do it before the Bug Bash. You'll lose valuable bashing time if you try and do it during the Bug Bash. This might mean setting up test user accounts in the system. Or, giving all the attendees the right level of access ahead of time.

Most of the time, it’s good to let Bug Bash attendees explore by themselves. But there can be cases where it’s useful to give some direction. For example, it might be desirable to have a range of different browsers tested during the Bug Bash. It can be helpful to think about this ahead of time. Then assign different browsers to attendees as they arrive. Allocate one person Firefox, one person Internet Explorer, and so on.

Triage after the Bug Bash, not during

When ten or more people are all finding bugs at once, there can be a lot of information to capture! It's not easy to manage all that information as it's arriving. A good solution is to simply capture that information, and parse it later. One thing that works is providing a place for attendees to 'dump' their findings. This could be a Slack channel, a confluence page, or a word document. Bug Bash attendees will discover what they think might be bugs. They can note the findings in this place, and move on to discovering more. It can be worth considering a template for logging findings too. Some Bug Bash attendees may not have tested before, and so not have an idea of what information is useful. Including some prompts like ‘expected behaviour’ and ‘screenshots or logs’ can help.

Once the Bug Bash is over, then the organiser can triage this information. That means sort it, remove duplicates, and start logging and prioritising the findings.

Make it fun

If Bug Bash attendees have a great time, they're more likely to show up at the next one! So, it's important to make it a positive experience. Snacks are a big part of this! Good snacks are a great way to make a Bug Bash a memorable experience, and people will come back for the snacks. In some organisations, making it competitive can be a good idea too. Prizes for ‘first bug found’ or ‘most significant bug’ can encourage participation. But, this is fraught with danger too. For example, if a prize was offered for ‘most bugs found’, attendees might be tempted to overload the Bug Bash with minor issues in the hopes of winning a prize. Competitiveness can be good, but it’s a tool that needs to be used wisely!

Also, be sure to thank people for their contribution. A thank you message after the fact, and some details about how the Bug Bash made an impact on the business, can go a long way!

Other benefits

There's some other great reasons to run a Bug Bash, too.

Load testing

For many features, a Bug Bash might be the first time you have several people using a feature at the same time. In some cases, a Bug Bash can also serve as a small load test of your product. Interesting things can happen when you start having more users on a feature at once!

Knowledge sharing

Bringing all these people together is a great way to meet other people in the organisation. A Bug Bash attendee could test something that others haven't considered or thought of. This becomes a learning and knowledge sharing opportunity.

Team building

It's also good for teams to mix in this way. One of the points of a Bug Bash is to include people from the organisation that don't normally cross paths. By getting them all together in a Bug Bash, new connections can be formed!

Socialising the value of testing

A lot of people that attend Bug Bashes don’t have the title of ‘tester’. A Bug Bash is a great way of exposing some of the challenges of testing to the rest of the team! By gaining that exposure, it can prompt other team members to think more about testing as they go about their regular work. If it teaches people to think more about testing, or testability, that’s a win!

Summary

In conclusion - a Bug Bash is a great way to leverage the help of others in an exploratory testing journey. Bug Bashes bring together people from different teams and different disciplines. This can reduce snow-blindness, and leverage the experiences and perspectives of many people. In turn, this will ultimately help catch more bugs. Which will help build a better product, and make customers and users happier! Providing the bugs are fixed of course ;)

About the Author

James is a software tester, currently solving problems for tradies as part of the team at Fergus.

He’s a big believer in making life better for everyone - by building helpful, easy-to-use software; by helping other testers get better at their craft; or by taking his friends to the pub.

When he’s not doing the above, you can find him drawing pictures, running around in the forest, or playing video games with his wife and son.

You can find him as @jamesespie on Twitter, or check out his ‘sporadically updated’ testing blog at jpie.nz.