QA Info Tech - A Brand Built on Results

Testing Chronicles - QA InfoTech Corporate Blog

The Bugs Stop at Our Bay

Mobile apps testing has changed the testing landscape in many different ways

Mobile apps testing has changed the testing landscape in many different ways

The software testing landscape has changed in many ways in the last decade. We often call the agile style of development as one the main reasons for such a major change. While that has certainly been a major reason for such change, there are several others too, of which the mobile computing landscape is also a significant reason. With Agile, one of the changes was how quality moved upstream rather than being seen as a final activity before product release. While such upfront testing gave a major facelift for quality, mobile computing and mobile apps testing brought in another change where big upfront testing (BUFT) is not always necessary and valuable. Mobile apps testing operates in a very large landscape – the combinations to test are huge, not all devices may be available in house and even if they do, it often does not make business sense to invest in such a huge mobile lab. With such increasing complexities that mobile apps bring in, a user centric approach to testing has become the need of the day. Users are being roped into the test efforts, whether as subject experts, real end users or as pure spike testing to take on the load that big upfront testing brings in. This makes compatibility testing more real, achievable and wide in its coverage. Mobile apps have changed the testing landscape in varied ways – this includes, greater focus on non- functional testing, more awareness of end user centric scenarios, pulling the shift left test approach into a shift right approach thereby bringing an unbiased balance between understanding the system implementation to the required level and building end user empathy.

This trend will only further strengthen given that mobile computing is expected to touch newer disciplines and geographies in the coming years. Newer devices, technologies, engineering implementations are all first looking at their seamless functionality in the mobile world. While mobile apps testing has changed the testing landscape in varied ways, the one significant one to call out would be the push back on big upfront testing bringing in a greater focus on end users. At the end of the day, while the focus away from upfront testing may contradict with the core agile manifesto, the end goal both are attempting to reach are the same – the one where solutions that are faster, cheaper and of better quality reach the customer’s hands, with a competitive advantage.

Optimizing Your Mobile App Testing Effort

Mobile applications have significantly expanded the scope of a test effort primarily due to increased testing combinations that have become necessary. Compatibility testing on the various device, OS, screen resolution combinations cover app functionality, layout and rendering, performance, usability amongst other testing attributes. While it is a no brainer that we want to maximize test coverage, given the constraints around time and cost in app development and more importantly, there are smart ways to optimize test coverage, to reap maximum benefits, testers are always on the look-out for such best practices to arrive at the right mobile app testing strategy. Specific to Android app testing, there are human interface guidelines (HIG) published by the Android team helping testers identify a good suite of tests to cover for their application. Such a set if also available for iOS by Apple.

We at QA InfoTech, have been testing mobile apps for the last several years now. We primarily focus on iOS and Android app testing services that we have now built a separate practice, ATE that focuses on mobile application testing. Based on our experience, we recommend some tried and tested practices in formulating a robust mobile test strategy. These include:

  1. Functional issues are more OS dependent rather than on the device size or screen resolution
  2. User interface issues are more specific to a device size or screen resolution rather than on the OS and typically show the following characteristics:
    1. Different devices with same screen size and OS version would give identical results
    2. Devices with different screen size but identical OS version would give different results

With the above inferences, we target a test matrix which covers varied devices, screen resolutions and operating systems at least once. We also keep in mind usage patterns such that more tests are run on the heavily used devices in the market. With such an approach, we have been able to help clients reduce their mobile app testing time and cost by about 30%, bringing in significant market returns for them.

With newer computing technologies coming in around wearable devices, augmented reality, SMAC, and the never ending set of new devices that enter the market every few days, the mobile app test matrix will only get increasingly complex. It is in the tester’s best interest to optimize the testing effort, to keep the overall quality effort relevant, user centric and realistic in achieving business goals.

Crowd Testing becoming a formal Mobile Testing Strategy

In any discipline, we often see practices gain prominence, not in isolation but together with another practice which is also taking shape. This is exactly what is happening in the mobile testing industry, and the crowd sourcing world. With the growth in the mobile industry, a lot of businesses have sprung off of it and mobile software testing is a significant discipline therein. While testers from the core team have been practicing newer testing techniques and acquiring newer devices to enhance their mobile testing coverage, given its vast scope it is not possible to test it all in-house. Slowly, testers are herein opening their testing options to newer practices, including leveraging the rest of the organization as part of their crowd team. In more mature test groups, external crowd is also brought in to enhance test coverage and more importantly device coverage. The growth in mobile computing has definitely given crowd testing a face lift and an important avenue to showcase the value crowd testers can add. Mobile app testing efforts continue to leverage in house devices as well as mobile simulators, but has started increasingly using crowd sourced testers for three reasons:

  1. Better access to mobile devices besides the ones they have in their internal mobile test labs
  2. Connect with end users, who possibly can be signed up for testing the product before release
  3. Test under realistic end user environments such as bandwidth and locales

Crowd sourced testing is not limited to just mobile testing. It can be used in places where end user feedback is very valuable – such as performance test scenarios, acceptance tests, usability and accessibility tests etc. The mobile device has become such an important platform to test on, that crowd sourced testing has further gained prominence and relevance in the software testing world in the recent years, that it can now almost be considered as a formal practice in the mobile testing strategy of an organization.