Pre vs. Post Comparison Regression Testing: Cost-effective

This method of testing by comparing multiple API responses between test and production code versions is highly effective and yields the desired results release after release. To meet changing requirements, improvements and changes are necessary. This holds true for almost all tech solutions, but not all. The economic principle “Law of Diminishing Marginal Utility” also applies to software. Technology solutions that were initially popular with stakeholders may become outdated very quickly. To meet evolving expectations, a redesign or a new solution may be necessary.

Overview

This method of testing by comparing multiple API responses between test and production code versions is highly effective and yields the desired results release after release. To meet changing requirements, improvements and modifications are necessary. This is true for most if not all tech solutions; the economics principle ‘ Technology solutions that were initially popular with stakeholders could soon become outdated. To meet changing expectations, a redesign or a new solution will be needed. This results in a world that is ever improving.

Parallel setups with legacy and tightly coupled code are created to fix leakage issues in critical business workflows. The DB is then restored from the same production backup. Each workflow step is completed simultaneously on both setups, and all reports are compared at milestones. The differences between the setups are due to identical data. These differences are then checked for unexpected and expected changes.
Two challenges arise from this approach:

• Parallel setups
• Production data not available for testing

This solution is extremely effective in identifying all issues in a critical flow which are not found in feature testing, where the specific result is verified. This solution is also expensive. Due to the requirement of parallel setups, licensing for testing and h/w costs are increased. A production backup is another constraint. Using production data gives you a level of confidence that is unmatched. However, this can also cause problems when there are more clients. Testing with only one client dataset is too complicated. Clients are often uncomfortable with sharing data or because of company policy cannot.

Single Setup instead of Parallel Setups

Multiple setups can be used to compare multiple versions. However, in practice, the purpose of testing is to compare production and the version being tested. Initial thought was to use one setup for multiple versions, one for each version. Then save responses and compare the results. One can prepare the setup using the production version, run the workflow steps and then save the API responses for comparison. Next, refresh the setup with the test version. Then execute the workflow steps and compare the API responses at milestones. This option was a disaster as it doubled testing time. It would be very difficult to meet the release deadline.

This was done to optimize the process further. The API responses for previous releases were stored in the code repository. Let’s go into more detail. The GitLab pipeline creates the release branch for the features to be released. It then deploys the build on the test server, and triggers the test suite. The test suite makes API calls and compares the response to the last release build. For manual verification, the response difference is reported to determine if the change is expected or unexpected. If the change is expected, the saved response will be updated in the repo. If the difference is not expected, a bug log is created to fix the problem.

This solution is both cost-effective and time-saving. It only requires one setup, and each release will only require one execution. However, it does have a drawback: the comparison cannot be performed with more than one version.

Using Non-Production Data

Testers who have access to production data are able to test with confidence. This is possible for a single client such as a captive company. Production data can also be made available for testing. Production data may not be available in all cases. Sometimes, data policies of clients prohibit data sharing with service providers. Or data can only be accessible to a limited number of employees within the company. This means that there may be many reasons why production data is not available for testing. As clients grow, multiple scenarios are required to be tested. Therefore, a single backup of production data cannot cover all possible scenarios. Test data is required. The data must be consistent across parallel executions in the solution that we are discussing. The data must be consistent in order to allow for comparisons.

An obfuscation program can be used to solve the problem and preserve production data’s integrity. An orchestration tool, such as Jenkins, can be used to restore production backup. The script will then be executed to obfuscate data and create a new backup. This backup is used to test. This second backup is our base. It is used to prepare the test setup. The obfuscation script cannot be executed. A script to add new data must be executed. This new data will improve coverage and cover new features. The next step is to make the schema changes in the new version. This ensures that the testing backup is always up-to-date.

Conclusion

These steps are combined to solve parallel regression testing issues. Comparing responses from multiple releases reveals potential issues and production problems. This is a better approach than testing the expected reaction to an action. It also verifies all other aspects to identify any potential production issues. Stable releases are built on the principle of testing all aspects of critical business workflows. It increases client confidence and leads to business expansion. This approach is more practical, easier to use and easy to implement because it only requires one setup.