Stories from Testing Voice First Devices, Such as Alexa - Kim Knup image

Stories from Testing Voice First Devices, Such as Alexa - Kim Knup

  • Locked
Kim Knup's profile
Kim Knup

Talk Description

“Alexa, how do I test a voice skill?”

“Sorry, I don’t know that one”

If Alexa doesn’t know, how are you supposed to know?!

The Alexa companion app, which is required to set up Amazon's Echo devices was the top app for Android and iPhone on Christmas day. This is a strong indicator that these voice first devices were the top gift in 2017 and sales aren’t slowing down.

Amazon doesn’t disclose official device sales, but said in a press release that “Amazon Devices also had its best holiday yet, with tens of millions of Alexa-enabled devices sold worldwide.”

This means the voice skill customer base is growing, and more and more companies are looking into breaking into the voice market to engage with their customer.

Having had the opportunity to work at an agency that prides itself on being pioneers of new tech, I have definitely seen this trend with us developing several Alexa skills at the same time in early 2018.

This talk will tell the tale of how a mobile development team, used to working on mobile apps, had to change their design and product thinking, especially when it came to testing the new voice products.

We’re no longer developing throw away proof of concepts. Customers want working and engaging voice skills with real life applications, for their users.

So how do you go from testing a mobile application to testing a voice skill? It can feel really daunting to not be using a keyboard, but instead be shouting through a room testing something.

What are the things to consider? Do you need to be able to do accents? (Note - having a diverse team helps massively already.)

But your mobile testing experience is not lost. Understanding voice design and the platform that will be used (Alexa vs google assistant) is the first step.

Testing can then be split into several, hopefully familiar, stages:

  • Test the design, question workflows
  • Manual testing using a variety of tools - voice device simulators
  • Unit testing the code
  • End to end testing using the device - do the user flows work, what happens when they don’t?
  • Continuous testing - mostly using monitoring tools - What are your users actually saying?


This session provides an overview of my experience testing voice apps, specifically focused on Alexa, and what tools are available to you. How testing a voice skill may and may not differ from more conventional testing and by knowing how the pieces fit together you can really help inform your testing approach. Also sometimes all you need is a whiteboard and another person to get started.


  • The Alexa platform basics: These are the concepts, definitions, and underlying architecture you need to know before testing Alexa voice apps.
  • Ideas for different test phases for an Alexa skill: Some of the most useful testing we did was before a line of code was written.
  • Overview of test tools we used - from pens and notepads to the Alexa developer portal to simulators.
What you’ll learn
  • TBA
Kim Knup's profile'

Kim Knup

About Speaker

<p>Kim is Senior Digital Tester at Legal and General, co-organisers of the Brighton tester meet-up; #TestActually and event host for the Brighton Software Testing Clinic. She is passionate about usability and likes to do what the user (apparently) would never do.</p> <p>Over the years she’s worked in linguistic games testing, and worked with big data archiving and asset management tools as well as recruiting and leading a small team of testers. Her main interests are usability testing and using automation tools to aid exploratory testing.</p>

More Talks
Thunderstruck by Serverless: a Tale of Learning and Testing Microservices - Conall Bennett

0h 39m 4s

Lessons Learnt Moving to Microservices - Simon Dobson

0h 27m 16s

Technical Risk Analysis for AI Systems - Bill Matthews

0h 32m 49s

Techniques for Generating and Managing Test Data - Omose Ogala

0h 31m 2s

Testing For Purpose - Ravneet Kaur

0h 40m 13s

The Final Frontier? Testing in Production - Marcel Gehlen and Benjamin Hofmann

0h 33m 45s

On Positivity – Turning That Frown Upside Down – Kim Knup

0h 24m 9s

Lessons Learned From 60 Days of Performance Testing - Kim Knup

0h 29m 17s

First 100 Days as the Only and First Tester in a New Company – Kim Knup

0h 22m 6s