Having talked about promoting the use of exploratory testing in your teams in my last post, I next want to touch upon how to interpret the results of exploratory testing as unless this is done in an informed manner, to the untrained eye, it may reflect badly on the overall test coverage:
1. Timing of the exploratory testing is very important in interpreting its results. I’ve seen some teams take on this testing technique even before a formal test pass can start, as this gives them an opportunity to play around with the product and flush out bugs as early as possible before they are given formal tests to run. I’ve also seen teams take on this testing after formal testing has been completed, thus using this as a supplemental method to find bugs that may not have been caught formally. Most teams that adopt an agile methodology follow the former approach. Regardless of the approach, be cognizant that in the former, most bugs would be found through exploratory testing. This is in no way reflective of poor test coverage.
2. Often times, tests are designed using the product specifications or requirements, so to some extent scripted testing will follow the path of the stated specifications. However exploratory testing knows no bounds and often finds bugs in the unbeaten path. This should be looked at in positive light and that the combination of scripted and exploratory testing has helped drive the product towards a quality bar that will be well received by the end user. To make these tests repeatable and constantly work on improving test coverage, the tester should add test cases for valid exploratory bugs filed. This will help ensure they are not missed during the regression cycle.
3. Bugs found from bug bashes especially towards product release, greatly determine the product’s ship date. Whether towards the end date or not, when analyzing bugs from bug bashes create and involve a triage team, representing the program & project mgmt teams, development and test teams. This is very important to truly analyze the bug’s end user impact.
4. Having talked of all the value that exploratory testing brings to the table, we also need to discuss one important limitation for which we need to implement a work around, failing which, exploratory testing may be seen as an overhead by the rest of the product team. In case of scripted testing, very formal design and review methods are often implemented, where test cases are vetted for validity before they are executed. Due to such stringent reviews the bugs found through such test cases have a very high validity rate. However, since exploratory testing does not follow such set bounds, there is a higher chance for false positives in the bugs filed. This may bring down the value of the test team and the exploratory test technique amongst the rest of the team, especially when everyone is working on tight deadlines. To ensure such false positives are kept low, peer reviews or reviews with the test lead/manager is often recommended for exploratory bugs. One may also attempt to introduce some formality yet not losing the spirit of exploratory testing through test techniques such as “session based testing viz. sessions and charters”. This will also help the test lead/manager tangibly analyze the results of exploratory testing and focus on specific areas that need more testing attention.
The right combination of scripted and exploratory testing is the secret recipe to balance the time available on hand to test and strive to improve product quality and end user acceptance. Exploratory testing cannot be taught/explained formally. It is an art and a science that one needs to pick up through experience. Herein I’ve shared with you all some approaches I’ve used on my past projects. Try these techniques on yours too and let us know if you have any suggestions, comments, questions.