Voice app testing is hard
User experience testing can be a minefield
A common stumbling block in building an engaging voice app is user experience testing. The freeform nature of a conversational voice application allows the user to input virtually anything. Proper testing must take this into account. There is also no standardization in user experience testing. Each voice assistant platform has a different set of criteria for deployment on their system.
Our experience testing voice applications can substantially improve your voice application development time and final product quality.
Best practices for voice testing
Use automated testing
Automated and regression testing allows the developer to execute hundreds or thousands of tests without manual intervention. Common issues can be easily isolated. Corner cases can be tested and expanded, tracking software improvements. In general automated testing can quickly and effectively cover 90% of potential issues.
Test with actual users
Real users, with varying accents, dialects and speaking mannerisms can uncover unexpected issues using manual tests with actual users. As new issues are uncovered, they can be transferred to automated testing, continually improving test coverage.
Our testing procedure advantages
Get consistent quality with automated regression testing
Automating regression testing ensures software iterations do not invoke unintended consequences. Any newly created bugs can be spotted immediately.
We use many open source and API resources
Open source software modules and API interfaces keeps test procedures adaptable to vendor platform changes and removes dependencies to any single-source vendor applications.
Our test automation is developed using device emulators
We rely on emulators and simulators as well as actual device testing to ensure efficient testing and complete test coverage.
Best practices for voice testing
Automated testing
Multiple layers of automated testing (model testing, functional testing, end-to-end testing, …) help to find issues as early as possible.
Manual testing
A lot of issues can be found only during manual testing because people speak differently in voice and text channels, and various speech-to-text issues may show more issues.
Try unexpected inputs
Users may not always give expected inputs to your app. At each point in you script, try a random speech response. Your application should recover gracefully.
Conversation-specific tests
Test various conversation-specific cases, such as the presence of follow-up questions, confirmations, context-switching and more.
Test on all supported platforms
Each platform will react to input differently. Make sure you run your full test suite on all supported platforms.
Understand the limitations of each platform
Some issues you may identify in your testing may be an issue with the platform, rather than your application. It is important to understand where the issue may originate.
Test environment matter
Devices such as Echo or Home should be tested across the room. Test with high ambient background noise. Real life environments can identify issues not normally found in the lab.
Crowdtesting
So many people, so many possible utterances. Only testing on many users will show the real quality of your application, and it also allows you to collect more data for NLU models.
Pre-certification testing
Amazon Alexa and Google Assistant have their own quality requirements which we should follow: security, policy, UX.
Create conversational maps
Create numerous typical and atypical voice interaction scripts. Map out expected responses to compare against actual responses.
Technologies we use
Bespoken is the leader in testing and monitoring for voice apps
Amazon Polly is a cloud service that converts text into lifelike speech
Allure is a flexible lightweight multi-language test report tool
TestNG is a testing framework inspired from JUnit and NUnit.
Protractor runs tests against your application running in a real browser
Appium is an open source test automation framework for iOS, Android and Windows apps using the WebDriver protocol.
Bespoken is the leader in testing and monitoring for voice apps
Amazon Polly is a cloud service that converts text into lifelike speech
Allure is a flexible lightweight multi-language test report tool
TestNG is a testing framework inspired from JUnit and NUnit.
Protractor runs tests against your application running in a real browser
Appium is an open source test automation framework for iOS, Android and Windows apps using the WebDriver protocol.
Bespoken is the leader in testing and monitoring for voice apps
Amazon Polly is a cloud service that converts text into lifelike speech
Allure is a flexible lightweight multi-language test report tool
TestNG is a testing framework inspired from JUnit and NUnit.
Protractor runs tests against your application running in a real browser
Appium is an open source test automation framework for iOS, Android and Windows apps using the WebDriver protocol.
Grid Dynamics uses state-of-the-art open-source software modules available from industry leaders. By harnessing the knowledge base of the most industry heavyweights, you get world-class results at a fraction of the costs.
Testing voice apps engagement model
Grid Dynamics makes testing an integral part of every voice application project we do. Our voice application product specification will include plans for comprehensive test coverage.
This includes script testing before coding, unit and system testing during code development, and monitoring after development.
Get in touch
We'd love to hear from you. Please provide us with your preferred contact method so we can be sure to reach you.
Thank you for getting in touch with Grid Dynamics!
Your inquiry will be directed to the appropriate team and we will get back to you as soon as possible.
Something went wrong...
There are possible difficulties with connection or other issues.
Please try again after some time.