Stories from Testing Voice First Devices, Such as Alexa - Kim Knup
“Alexa, how do I test a voice skill?”
“Sorry, I don’t know that one”
If Alexa doesn’t know, how are you supposed to know?!
The Alexa companion app, which is required to set up Amazon's Echo devices was the top app for Android and iPhone on Christmas day. This is a strong indicator that these voice first devices were the top gift in 2017 and sales aren’t slowing down.
Amazon doesn’t disclose official device sales, but said in a press release that “Amazon Devices also had its best holiday yet, with tens of millions of Alexa-enabled devices sold worldwide.”
This means the voice skill customer base is growing, and more and more companies are looking into breaking into the voice market to engage with their customer.
Having had the opportunity to work at an agency that prides itself on being pioneers of new tech, I have definitely seen this trend with us developing several Alexa skills at the same time in early 2018.
This talk will tell the tale of how a mobile development team, used to working on mobile apps, had to change their design and product thinking, especially when it came to testing the new voice products.
We’re no longer developing throw away proof of concepts. Customers want working and engaging voice skills with real life applications, for their users.
So how do you go from testing a mobile application to testing a voice skill? It can feel really daunting to not be using a keyboard, but instead be shouting through a room testing something.
What are the things to consider? Do you need to be able to do accents? (Note - having a diverse team helps massively already.)
But your mobile testing experience is not lost. Understanding voice design and the platform that will be used (Alexa vs google assistant) is the first step.
Testing can then be split into several, hopefully familiar, stages:
- Test the design, question workflows
- Manual testing using a variety of tools - voice device simulators
- Unit testing the code
- End to end testing using the device - do the user flows work, what happens when they don’t?
- Continuous testing - mostly using monitoring tools - What are your users actually saying?
This session provides an overview of my experience testing voice apps, specifically focused on Alexa, and what tools are available to you. How testing a voice skill may and may not differ from more conventional testing and by knowing how the pieces fit together you can really help inform your testing approach. Also sometimes all you need is a whiteboard and another person to get started.
- The Alexa platform basics: These are the concepts, definitions, and underlying architecture you need to know before testing Alexa voice apps.
- Ideas for different test phases for an Alexa skill: Some of the most useful testing we did was before a line of code was written.
- Overview of test tools we used - from pens and notepads to the Alexa developer portal to simulators.
Creating a Culture of Quality Assurance - Angela Riggs00:27:43
Manual Regression Testing Manifesto - Brendan Connolly00:28:58
Testing the Front-end, Back-end, and Everything in Between - Bria Grangard00:29:00
The Joy of Monitoring Or: How I Learned to Stop Worrying and Test in Production - Amber Race00:28:05
Climbing to the Top of the Mobile Testing Pyramid - Rick Clymer00:31:47
How to Defuse a Bomb... Wait, I Mean a Bug - Michele Campbell00:30:57
Getting under the Skin of a React Application: an Intro to Subcutaneous Testing - Melissa Eaden & Avalon McRae00:39:48
99 Second Talks - TestBash San Francisco 201800:40:22