Continuous delivery raises the bar for performance testing
Competition between digital businesses is fierce. Application performance issues can be catastrophic, ruining customer experience and brand loyalty. Traditional performance testing requires days or even weeks to complete, and involves a lot of manual effort. With the mass adoption of continuous delivery, DevOps, chaos engineering and SRE, such delays are no longer acceptable. Performance testing should be fast and performed continuously to let the development team fix issues early in the release process, so that the context of the issues are fresh and the costs are minimal.
Pioneers in continuous performance testing
Grid Dynamics has been working in continuous performance testing for years, well before it became a popular addition to the testing lexicon. This practice developed from our expertise in building mission-critical applications, our longstanding tradition of automating everything, and our use of agile best practices. Our architects and engineers have a lot of experience in continuous performance testing, having implemented it with many clients in different industries and platforms over the years. In 2011, we even released our own continuous performance testing tool called Jagger.
Approach to continuous performance testing
Identify KPIs and design tests for them
The first important step of continuous performance testing is to find the most critical KPIs and determine what levels of performance are required for them. Once KPIs and test targets are known, the design of performance tests is similar to the design of functional tests. Some of the important questions that need to be answered involve test data management and dependency management. In some cases, testing with synthetic data and service virtualization may bring the biggest benefit, while in others, using production data and real dependencies makes more sense.
Let developers and testers find defects immediately so that they can be fixed quickly and cheaply. Do this correctly by:
- Creating a distributed test environment with the required dependencies and test data.
- Implementing load scenarios and collecting performance metrics from infrastructure and applications.
- Building statistical and machine learning models for data correlation and results interpretation.
- Creating pass/fail thresholds and delivering results via reports or data visualization.
Augment results analysis
A couple techniques can be used to automate the analysis of results:
- Set KPI thresholds based on observed performance metrics.
- Learn KPIs with machine learning and find anomalies if the new results differ too much from the old ones.
Continuous performance testing is often performed on smaller environment to save costs. In order to predict absolute numbers, the correlation of metrics between production and the performance lab should be learned based on the past results.
Meet performance targets
Catch performance issues early
Integrate with CI/CD pipeline
Automatically analyze results
Save on environment costs
Avoid costly performance failures
High level reference architecture
We prefer a hands-on approach in our engagements from day one. With the help of clients, our architects and engineers can identify challenging applications and services which would benefit the most from continuous performance testing. We then begin our assessments by looking at the application requirements, performance KPIs and existing methods of testing.
After we understand the current situation in depth, we make point improvements to bring the most value to our clients with the least amount of work, and in the shortest period of time. Once these show value on a small scale, we move to long-term improvements in the process and tooling on a larger scale, adopt continuous performance techniques across the board, and educate client teams to spread knowledge and know-how within their organization.