Global penetration of software and information technology has increased leaps and bounds in the last decade reaching out to even the developing and under-developed countries through various domains such as banking, health care, education etc. When the diversity of software usage increases, so does the associated variables/parameters through which software can be put to use. What this translates to, is the need for testing software on such potential variables and parameters before release. While on one hand such increased variables necessitate more testing, the time to test has not proportionally gone up. Interestingly given the competition in the marketplace and the available options, companies are forced to release sooner than before. How can test management handle this ironical situation yet ensuring it has taken ownership for overall product quality guaranteeing product acceptance?
The answer to this challenge is effective Test Optimization. Test Optimization has long existed in the history of the testing realm, but as discussed above, it has gotten more complex in the recent years. A decade ago options used to be restricted to supporting your product on just the Operating Systems and Browsers (OSBs), from the commercial players. There were hardly 8 to 10 combinations to support. Now even in the space of such OSBs you have a lot more to test for, including products from the Open Source world. Backward compatibility scenarios increase your combinations to test for, in addition to the latest OSBs.
Besides OSBs on personal computers and laptops, a lot more computing devices and service offerings will now have to be taken into account, such as smart phones, electronic readers, hosted solutions, on premise solutions, to name some. To add to this list, if your product were to have several SKUs, those need to be included as well to your optimization list. With all these in play it is not surprising to see that test organizations are building artifacts just to optimize test efforts as part of their core test plan. A few things to keep in mind in optimizing your test efforts:
- It is very important for to build a list of supported combinations to test on and keep monitoring this list periodically to ensure it is up-to-date
- After having created the list, test organization should constantly work on prioritizing this list based on the users they support; such a prioritized list is very important to ensure the right test coverage in the limited amount of time
- It is also good to build a core team that specializes in testing for supported combinations as it can easily turn out to be a cumbersome effort if not planned for. Such a team not only builds the required expertise to test, but is also ready with the required infrastructure, images, virtual connections, devices etc. needed for testing. This is also an area where a company can leverage on demand testing solutions including Infrastructure as a Service (IaaS) not having to invest heavily in-house
- Besides major commercial cloud players such as Amazon, Microsoft, Google you also have several small players who also help you set up customized on-demand offerings to simulate your test bed such as SkyTap, Device Anywhere (for mobile environment). One can leverage such specific on-demand solutions to save time in test environment setup
- Re-usability is the key to saving time when you are working on a plethora of combinations. Re-use test artifacts, cases, scenarios, frameworks etc. from previous iterations however paying attention to whether they apply to the current iteration as often times past scenarios (especially platforms tested on) might become obsolete quite soon
- Every tester should be encouraged to build his/her own testing strategies to make the test optimization effort successful. For e.g. if a test case were to be run on 5 different combinations, some may prefer running the same test case simultaneously on the 5 platforms before moving to the next one. Others may prefer to finish testing on one platform fully before moving to the next one. Each has its own pros and cons, so empower your test team to choose their individual test strategy which will help them get the best testing results
- Defect management is also an area to pay special attention to, when working on an optimized test effort with several platforms to test. Test management team should work out the larger best practices and protocols around defect management and within that pre-defined boundary allow the tester to use his/her creativity in reporting and finding defects especially when it has to be reproduced on several platforms
With the increasing set of computing options available to the end user, the complexities around test optimization will only increase. This is where a seasoned test management effort becomes imperative to chalk out the right matrix to test within the bounds of available time, resources, cost yet not compromising overall coverage and product quality.