“Are you done testing?”, is a very subjective and difficult question to answer in the field of software quality. Over the years, managers and leads have toiled to identify a customized set of test exit criteria with which they can confidently say, an application is ready for release. Even then, it does not mean they stop testing. They would move from testing the product in pre-production environments to evaluating user feedback and testing as required in live environments. While this was the scenario even in the pre-mobile days, mobile computing has made this question even more difficult than ever before, to answer.
Testers have been putting together an out-of-box strategy to maximize test coverage within the limited time, money and other resources they have on hand, with clear comprehension that this is important to give their product an edge in the ever competitive marketplace. The rest of the product team is also collaborating very well with the quality function in making this possible. When I was recently reading online about how the mobile market is further maturing and shaping itself for the coming years, I happened to stumble on a bunch of very useful mobile KPIs (key performance indicators). It almost immediately struck me, as to why not leverage these KPIs with pre-defined values for our test exit criteria. Wouldn’t these help bring in a lot more objectivity into the mobile apps testing market? For example, metrics such as cost per installation (CPI), cost per acquisition (CPA), brand lift, view ability, time spent in app, screen visits etc. While these are great indicators to bring into the testing fold they are often not easily translatable into testing scenarios. For example, if you define a certain number for brand lift, you will have to explore on how that brand lift is achieved – is it from the app’s usability, performance, richness in mobile content, brand loyalty etc. Also, most often these KPIs are live app indicators rather than ones that can be incorporated pre-production. It is here that the tester can step up to show how the quality function needs to mature and take in these indicators to benchmark certain test results in determining the applications readiness to release. Even if you don’t directly consume these indicators in your testing scenarios, at-least being cognizant of these and what they mean to the business will help the tester develop a holistic appreciation for the development effort, end user and the market place, helping him think beyond core defined requirements. Try this for yourself the next time around, and let us know what you think.