What's Special About Mobile Testing?

Posted Sep 6th, 2017

This article will lay out some background and approaches to creating an effective mobile testing strategy to help teams move forward.

This section’s great title is taken directly from Daniel Knott’s book Hands-On Mobile App Testing: A Guide for Mobile Testers and Anyone Involved in the Mobile App Business.

Mobile app testing and mobile website testing are indeed special! They’re both considerably different from other forms of testing. The business goals are different, the app technology is different, the interactions are different, user expectations are different, and of course the hardware is extraordinarily different.

Creating an effective mobile testing strategy requires teams to get their arms around all those factors, and more. This article will lay out some background and approaches to help teams move forward.

What Type of Testing?

Any high-quality delivery strategy requires a solid mix of different kinds of testing. The more complicated a system being delivered, the more complicated the testing mix. Teams delivering traditional website, back-end, or desktop systems look to a broad mix of testing. Some of the various testing types might include:

  • Acceptance
  • Unit
  • Integration/API
  • Functional / Acceptance
  • Performance¥ Availability
  • Security
  • Accessibility

These various are generally covered by a mix of automated and manual testing across a number of different environments: think development, QA, staging, etc. Of course testing needs to cover all the various environmental factors such as varying client operating systems and browser versions.

Adding mobile testing in to the mix explodes the complexity. Not only do teams need to look at a potentially soul-crushing matrix of supported devices and operating system versions, they also need to identify what will be tested on emulators, simulators, and real devices—and that’s across the already large matrix. As identified by Gartner, “A recommended approach is to find a healthy mix of emulators, simulators and real devices to get the best out of your test automation. Gartner, Market Guide for Mobile App Test Automation Tools, Maritess Sobejana (April 2016)

Emulators vs. Simulators

It’s important to clearly understand the difference between emulators and simulators so that you can properly lay out your testing strategy. Emulators and simulators both enable mobile app testing to be done manually and automatically, albeit within the constraints of their limited environments.

Emulators are complex software environments that do their best to exactly mimic a mobile device’s hardware. Emulators provide varying support for components like GPS, accelerometers, batteries, storage, etc. Simulators, on the other hand, are less-complex software that provide only minimal environments for applications to run in. Simulators’ hardware support is much less exacting than an emulators’. Again, from Knott’s Hands-On Mobile App Testing: "a simulator attempts to duplicate the behavior of the mobile device, while an emulator tries to duplicate the entire inner architecture of the mobile device and is therefore closer to the target platform.”

A considerable benefit of emulators and simulators is the ease of deploying apps to them. Deploying an app to a real device often involves actions such as “side loading” or “root kiting” which bypass processes like App Store verification.

Apple provides a simulator for iOS development. Android and Windows Phone developers have access to emulators for those respective platforms.

Real Devices

Naturally real devices offer a complete, exacting environment for testing your mobile applications and websites. Testers have full access to a device’s components including light sensors for screen brightness adjustments, cameras, accelerometers, GPS, audio (microphones and speakers), and all other systems in the actual device itself. Obviously access to the complete environment is a huge necessity for accurate, deep testing.Testing on real devices requires teams to determine the exact matrix of devices and operating systems to work through. This matrix can quickly grows to a large set as different generations of devices and models come in to the mix, e.g. iPad Wifi Gen 4/5/6, iPad Cellular Gen 4/5/6, iPad mini Gen 2/3/4, etc. Android’s matrix is far more expansive due to the huge number of manufacturers supporting Android.

The Testing Pyramid for Mobile

Good test automation mixes a cross of unit, integration (or service), and functional/User Interface tests. While every project, system, and delivery team are unique, generally that mix should appear as a pyramid. (The “Test Pyramid” concept was first laid out by Mike Cohn and has been written about by a huge number of thought leaders across the industry.)

Testing Pyramid

The majority of automated tests should be unit tests: small, compact, fast checks which run in complete isolation from dependencies such as databases, web services, etc. This forms the base of the pyramid.

Next up are service tests: checks which require some form of interaction with a service, external component, etc. These tests are far slower and more complex than unit tests due to those dependencies. There should be far fewer service/integration tests than unit tests, and these should generally focus on larger behaviors.

Finally are the most complex tests: Functional or UI tests. These tests spin up the app and drive its user interface around to simulate workflows from the users’ views. These tests are brittle, slow, and hard to write and maintain. Ergo, there should be just a few in any good suite.

Mobile app testing generally tends to flip this pyramid on its head: There are a tiny number of unit tests due to the complexities of the platforms, operating systems, and toolsets. Instead far more test automation occurs at UI level.

Again, these are sweeping generalities. Each organization, project, and team will approach their test automation mix in different fashions.

Manual Testing Still Required

Automated testing in a mobile app environment needs to avoid the challenges of dealing with complex capabilities like haptic feedback, multi-touch screens, or accelerometers. It’s also very difficult to automate checks around subtle interactions with things such as notification banners, power-saving features, and a device’s sleep modes. Good mobile test automation focuses on an app or mobile site’s overall functionality, and leaves skilled testers to explore those difficult areas.

Good manual testing of a mobile app or website lets the tester look at overall usability factors such as consistency, content accuracy, appearance and function when in different resolutions or rotation aspects. Good testers also use mobile app testing to explore how an app behaves when other notifications appear on the device, how the app behaves when the device drops in to power savings mode, whether data is properly saved locally on the device, and how well the app conforms to the platform’s guidelines (Apple, Microsoft, and Google all have specific guidelines for apps on their platforms).

There are a host of other factors for testers to consider addressing with skilled manual exploration when testing mobile apps or websites. This was just a brief coverage of them.

Mobile Automation Tools

Teams considering automation tools for mobile app or website testing need to evaluate a number of factors. What platforms must be supported? If an app is being automated, is the app native, hybrid, or a mobile web app? Does the organization want a commercial product, open source software (OSS), or OSS with commercial support? How will the automation suites be executed?

A significant part of this consideration must include questions about infrastructure and execution. For example, if the organization, if the organization wants their automation suite to run on real devices, will the selected toolset support internal and external device clouds? Where will emulator and simulator networks be hosted?

Perhaps the most significant question is will the tool support all the platforms your organization is using, or will you need to write separate automation suites for iOS, Windows Phone, and Android?

WebDriver Adaptations and Implementations

Selenium WebDriver is a low-level driver that drives user interface sessions enabling teams to write automated tests on top of a real UI. Initially WebDriver interacted only with web browsers such as Firefox, Chrome, or Edge/Internet Explorer. Recently tools such as Appium, Selendroid, iOS Driver for iOS, and others have leveraged WebDriver to drive native applications as well.

These tools benefit from a huge body of knowledge and expertise from the WebDriver community. It also means these tools have a somewhat similar approach to how they interact with their supported platforms.

Commercial Tools

Commercial tools have had varying levels of support for automating mobile app testing. Some have had years of early, if somewhat rough, support for mobile devices, while others have been somewhat slower to provide support—but have dropped in with a broad away of tooling. Commercial offerings often use proprietary systems for their UI automation, though a few are now supporting various WebDriver implementations.

Commercial offerings benefit from having training and support available directly through the company. The downside of commercial tooling can be a narrow, non-standard implementation for their proprietary tooling. Just because it’s a big company behind a tool doesn’t mean you’ll be able to find a huge community of users through the industry…

Platform Specific

Platform-specific tools also exist for all the various vendors (Android, Windows Phone, iOS). Note that some platform-specific automation tools are maintained by non-vendor organizations. This means those tools aren’t beholden to the particular platform vendor; however, they may be limited in their access to internals for that particular platform. Espresso for Android and XCUITEST for Xcode are two examples.

Sauce Labs: A One-Stop Shop

Sauce Labs brings all the above concepts into a single solution for mobile app testing and mobile website testing. Sauce enables teams to write their tests using WebDriver based tools, execute those tests on cloud-based agents via any common CI/CD pipeline—or use Sauce’s own execution infrastructure—,and use a mix of emulators, simulators, and real devices.

Moreover, Sauce Labs gives teams the ability to work with real devices on premise at the organization’s location, or utilize Sauce’s own device cloud. This makes it easy for teams to offload device management and focus on delivering high-quality, well-tested mobile apps and websites.

Written by

Jim Holmes


Mobile TestingEmulatorsSimulatorsReal DevicesManual/Live testingAutomated testing