Testing in production is an inevitable need in today’s quality strategy given how connected the development and production environments are. Teams are very cognizant of this today and it is welcoming to see a lot of testing in production both manually and in an automated manner. However software test automation in production is easier said than done. The test suite has to be smarter than what it is and does in a test environment given the load it deals with, live user data it handles, on demand decisions that need to be taken, the varied test attributes especially non-functional test attributes such as performance, usability, security that are covered. So, as part of the strategy to ensure test automation, whether open source test automation or not, is effective when it handles testing in production, it is important to ensure:
- Attended test runs – while unattended test runs are good for the large part, it is important to monitor the results of the runs more closely and take immediate actions in live
- Maintenance effort is accounted for – to determine what to include, what to update, what to eliminate on a periodic basis so the tests included for automated production runs yield the most returns
- Automation suite is augmented for – based on user feedback, any important regressions that set in etc.
- Automated testing data feeds into live monitoring data – to enable business and stakeholder teams to take up tasks such as predictive analysis, user satisfaction surveys, future product changes etc.
Software testing automation in a non-live environment still largely focuses on catching issues/defects, while in production it has a larger role to play in monitoring usage patterns, complementing what monitoring tools provide, enabling decisions etc. To that extent it needs to be a smarter tool to be leveraged not just by the test team but by the product team at large. To this effect, the above set of points, though may not be exhaustive, forms a good starting point to ensure software test automation is smarter in the production environment.