Stories from Testing Voice First Devices, Such as Alexa - Kim Knup

Talk Description

“Alexa, how do I test a voice skill?”

“Sorry, I don’t know that one”

If Alexa doesn’t know, how are you supposed to know?!

The Alexa companion app, which is required to set up Amazon's Echo devices was the top app for Android and iPhone on Christmas day. This is a strong indicator that these voice first devices were the top gift in 2017 and sales aren’t slowing down.

Amazon doesn’t disclose official device sales, but said in a press release that “Amazon Devices also had its best holiday yet, with tens of millions of Alexa-enabled devices sold worldwide.”

This means the voice skill customer base is growing, and more and more companies are looking into breaking into the voice market to engage with their customer.

Having had the opportunity to work at an agency that prides itself on being pioneers of new tech, I have definitely seen this trend with us developing several Alexa skills at the same time in early 2018.

This talk will tell the tale of how a mobile development team, used to working on mobile apps, had to change their design and product thinking, especially when it came to testing the new voice products.

We’re no longer developing throw away proof of concepts. Customers want working and engaging voice skills with real life applications, for their users.

So how do you go from testing a mobile application to testing a voice skill? It can feel really daunting to not be using a keyboard, but instead be shouting through a room testing something.

What are the things to consider? Do you need to be able to do accents? (Note - having a diverse team helps massively already.)

But your mobile testing experience is not lost. Understanding voice design and the platform that will be used (Alexa vs google assistant) is the first step.

Testing can then be split into several, hopefully familiar, stages:

• Test the design, question workflows
• Manual testing using a variety of tools - voice device simulators
• Unit testing the code
• End to end testing using the device - do the user flows work, what happens when they don’t?
• Continuous testing - mostly using monitoring tools - What are your users actually saying?

This session provides an overview of my experience testing voice apps, specifically focused on Alexa, and what tools are available to you. How testing a voice skill may and may not differ from more conventional testing and by knowing how the pieces fit together you can really help inform your testing approach. Also sometimes all you need is a whiteboard and another person to get started.

Takeaways:

• The Alexa platform basics: These are the concepts, definitions, and underlying architecture you need to know before testing Alexa voice apps.
• Ideas for different test phases for an Alexa skill: Some of the most useful testing we did was before a line of code was written.
• Overview of test tools we used - from pens and notepads to the Alexa developer portal to simulators.

By the end of this session, you'll be able to:

Kim Knup

She/her

Senior Quality Engineer

Kim is Senior Quality Engineer and co-organisers of the Brighton tester meet-up.