Voice app testing is hard

User experience testing can be a minefield

A common stumbling block in building an engaging voice app is user experience testing. The freeform nature of a conversational voice application allows the user to input virtually anything. Proper testing must take this into account. There is also no standardization in user experience testing. Each voice assistant platform has a different set of criteria for deployment on their system.

Our experience testing voice applications can substantially improve your voice application development time and final product quality.

Best practices for voice testing

Use automated testing

Automated and regression testing allows the developer to execute hundreds or thousands of tests without manual intervention. Common issues can be easily isolated. Corner cases can be tested and expanded, tracking software improvements. In general automated testing can quickly and effectively cover 90% of potential issues.

Test with actual users

Real users, with varying accents, dialects and speaking mannerisms can uncover unexpected issues using manual tests with actual users. As new issues are uncovered, they can be transferred to automated testing, continually improving test coverage.

Our testing procedure advantages

Get consistent quality with automated regression testing

Automating regression testing ensures software iterations do not invoke unintended consequences. Any newly created bugs can be spotted immediately.

We use many open source and API resources

Open source software modules and API interfaces keeps test procedures adaptable to vendor platform changes and removes dependencies to any single-source vendor applications.

Our test automation is developed using device emulators

We rely on emulators and simulators as well as actual device testing to ensure efficient testing and complete test coverage.

Best practices for voice testing

Automated testing

Multiple layers of automated testing (model testing, functional testing, end-to-end testing, …) help to find issues as early as possible.

Manual testing

A lot of issues can be found only during manual testing because people speak differently in voice and text channels, and various speech-to-text issues may show more issues.

Try unexpected inputs

Users may not always give expected inputs to your app. At each point in you script, try a random speech response. Your application should recover gracefully.

an orange circle, two arrows, and a clock of cubes

Conversation-specific tests

Test various conversation-specific cases, such as the presence of follow-up questions, confirmations, context-switching and more.

an orange circle and a mobile phone icon

Test on all supported platforms

Each platform will react to input differently. Make sure you run your full test suite on all supported platforms.

Understand the limitations of each platform

Some issues you may identify in your testing may be an issue with the platform, rather than your application. It is important to understand where the issue may originate.

Test environment matter

Devices such as Echo or Home should be tested across the room. Test with high ambient background noise. Real life environments can identify issues not normally found in the lab.

an orange circle and a group of people icon

Crowdtesting

So many people, so many possible utterances. Only testing on many users will show the real quality of your application, and it also allows you to collect more data for NLU models.

Pre-certification testing

Amazon Alexa and Google Assistant have their own quality requirements which we should follow: security, policy, UX.

Create conversational maps

Create numerous typical and atypical voice interaction scripts. Map out expected responses to compare against actual responses.

Technologies we use

Bespoken is the leader in testing and monitoring for voice apps

Amazon Polly is a cloud service that converts text into lifelike speech

Allure is a flexible lightweight multi-language test report tool

TestNG is a testing framework inspired from JUnit and NUnit.

Protractor runs tests against your application running in a real browser

Appium is an open source test automation framework for iOS, Android and Windows apps using the WebDriver protocol.