Getting Started with Public Device Clouds for Mobile App Testing

One of the challenges faced by mobile developers is to combat OS and device fragmentation to make sure their app works across different devices used by their consumers. I delivered a talk recently at Android Summit to compare popular public device clouds - Google Firebase Test Lab, Amazon Device Farm and Microsoft App Center to demonstrate how easy it is to get started with cross-device cloud testing. Read on to learn more or just see the video from my talk.

My talk at Android Summit in Washington, DC.

There are over 1.5 million apps each in iOS and Google Play Store. The mobile app market is growing faster than ever and building a mobile app is not a rocket science anymore. The need for testing Android apps across multiple devices has never been more relevant than now. However, testing a mobile app still remains one of the most challenging tasks of any app development cycle. Fragmentation is at an all-time high, with 6 OS versions having over 5% user share, 127 mobile device makers and the multitude of variations within them make it extremely hard to test on different devices. Bugs in an app lead to a bad user experience and may end up as bad reviews in the app store. Testing, debugging, and profiling form the core part of the app development cycle. When it comes to testing the best rule of thumb is: TETO - ‘Test Early and Test Often’.

Fortunately, there are several public device cloud offerings by major cloud computing companies, including Amazon Device Farm, Google Firebase Test Lab and Microsoft App Center. This provides a variety of devices available for test for developers to run automated test scripts. In this blog, we will share our experience with public device clouds and will empirically compare and contrast the different services, and how testing on them can fit into an agile CI/CD process. We are going to explore the device cloud options for your mobile app testing needs. Cloud based mobile app testing providers are an amazing option for testing applications quickly and easily with numerous benefits including:

  • Large scope of testing environments - It provides access to a large range of customizable testing environments rather than having to invest in physical hardware and infrastructure.
  • Cost saving - Cloud services can be paid for when in use. Also, the ease and efficiency of cloud testing allows for more bugs to be caught early on; at later stages of development they would be more costly, time consuming, and difficult to fix.
  • Scalable - Testing resources can be increased or decreased wherever necessary based on business requirements.

But before we dive into the clouds, let us discuss about selecting which devices you should be selecting for testing your app.

Selection of the Device

Although it sounds imperative that a mobile app should be tested on a mobile i.e. a real device, before selecting a device to test out your app, it is wise to ask yourself - Real or Emulated: What’s the Best Choice for App Testing?

Mobile emulators/simulators are software programs that are specifically designed to simulate situations related to some salient features of the application.

Real device testing can provide more accurate results because the false negatives or false positives are avoided. You are testing the real performance of your app and compatibility with hardware, different networks, and data. The disadvantage of testing on a real device is the additional cost and task of managing all these physical devices.

Testing on a Real Device

If you have decided to test the app on a real device, you need to pay attention to decide which device to select. There are several parameters that can help you in picking up a device -

  • Device’s coverage in the market or within analytics - To help conquer the issue of fragmentation you can use Google’s platform distribution chart to see the distribution of a given platform version of the Android and how to target your application to devices based on it. Additionally there is information about popular screen sizes and densities so you can pick devices matching such criteria. Other sources to get popular devices are public data analytics sources such as AppBrain and Device Atlas, or from existing app analytics tools such as Firebase Analytics.
  • Devices with known crashes or issues - On Firebase you can navigate to Crashlytics under the Quality Section of the dashboard to see the current crashes along with associated devices and the number of users that were affected by the specific crash.
  • Devices by year class as per Facebook - Device Year Class is an Android library that maps a device’s RAM, CPU cores, and clock speed to the year when that combination of spec was considered high end. This helps the developer alter the behavior of an application based on the capability of the phone’s hardware. While choosing devices for testing, a developer can choose to select representative devices within each supported year class.

Mobile App Testing Device Clouds

Mobile app device testing clouds are useful when you want to test your application on a variety of real devices, along with emulators and simulators. When using cloud devices, developers don’t have to worry about securing their own devices for testing.

The cloudscape - a fancy term for the gamut of clouds - is an excellent platform for mobile app developers but can be overwhelming at times with so much to offer. To bring it to a concise point, here are the top public device clouds.


Firebase Test Lab

Firebase Test Lab provides a cloud-based service which enables testing on a variety of physical and virtual devices and device configurations including Android apps and API levels. All of the testing is performed at a Google data center which allows for a range of testing on different device platforms without the need to buy the device or create and configure multiple emulator sessions.

Firebase recently launched its support for testing iOS apps as well. It runs Espresso and UIAutomator tests for Android and XCTests for iOS apps. It has over 50 devices with multiple versions and locales.

Once the test matrix is defined and the APK file uploaded, testing can start with just the click of a button. You can test your application using your own test script or its built-in Robo test which is a smart app crawler. Test results are displayed in the Firebase Console.

Robo Testing

Robo testing has the advantage of allowing apps to be tested using a pre-written test script. It identifies the hierarchy of the user interface, then runs through a sequence of user actions that are intended to test as many areas of the app as possible. It will continue until either a crash occurs or the test times out.

Robo testing crawls the app quickly to find bugs that will cause the app to crash and helps to identify UI discrepancies between multiple device models.

The Robo test does not know the logic of the way the app is meant to function, so it cannot verify that a response apart from a crash, has responded in the way it was meant to. This is where instrumentation testing is needed.

Instrumentation Testing involves creating tests that are designed to test an app and validate that it behaves as it’s expected to in response to the users actions. Instrumentation tests are currently support on Espresso, Robotium, and UI Automator Android testing frameworks.

Game Loop Testing

This is a new Firebase Lab Test feature that allows user actions to be simulated within game apps that use third party frameworks for the user interface. Code is added to the app to simulate user interaction.

Espresso for Android

Espresso is a testing framework that allows for the creation of automated instrumentation tests that simulate user interaction in an Android app.

Getting Started with Firebase Test Lab:

  1. Get your Testing Environment Ready - iOS: Write XCTests, then build and package your app for upload. Android: Write an instrumentation test, modify the app to support a Game Loop test, or select a Robo test.
  2. Choose a Test Environment and Matrix - Using the Firebase console, define a test matrix by selecting a set of devices, OS versions, locales, and screen orientations.
  3. Create New Project - Go to the Firebase console and click create new project.
  4. Build the Tests - Write XCTests, then build and package your app for upload.
  5. Run the Tests - Package the tests into zip files then upload them. Click either Get Started or Run a Test.
    iOS: Select Run an XCTest. Click browse and find the zip file. Then select the box for each device, version, orientation, and locale to test on and click Start Tests.
    Android: You can Run a Robo Test, Instrumentation Test, or modify the app to support a Game Loop Test.
  6. View the Results - Once the tests are complete, view the results in the Firebase console.

Pricing: There are 3 Firebase plans available currently: Spark, Flame, and Blaze. Spark testing is free but limited to 10 virtual devices and 5 physical device test executions a day. Blaze is the only one that includes Firebase Test Lab. It allows for unlimited testing and testing is charged at $5/device/hour.

Strengths: Firebase Test Lab allows apps to be tested on a wide range of devices and operating systems just by uploading an APK file to the Google cloud and selecting the tests to by performed. It provides an emulator as well as a real device testing platform. It is good for testing gaming loops and has a built-in Robo test. It can be integrated into an existing environment such as Android Studio, the command-line, or Jenkins Continuous Integration tool.


AWS (Amazon Web Service) Device Farm

This is probably the most robust device cloud offering that supports Android as well as iOS applications, with nearly 400 devices available for testing in the AWS Cloud. AWS offers all real-device testing without the option of using emulators while supporting native as well as hybrid apps. Its salient feature is the option of manual remote access and private access for debugging.

You can upload apps and run Fuzz tests to simulate random activity or use the built-in app explorer which crawls the app and captures screenshots as well as performance metrics and logs.

Getting Started with AWS Device Farm:

  1. Create an AWS account - AWS recommends creating a new IAM (Identity and Access Management) user account rather than using your root account to access the service.
  2. Open Device Farm - Once inside the AWS console, select Device Farm from the Services option in the header of the page. They will provide you with a free 1000 minutes to get started.
  3. Create New Project - Click on Get Started once you are at the homepage of AWS device farm. The will prompt the user to enter a project name. Input a name for your project and click Create project.
  4. Start your Test - Once a project has been created, select the project, on the automated tests page, choose Create a new run, on the Choose your application page, select Upload.
  5. Upload your Android or iOS file - For android the file will be .apk, for iOS, the file will be .ipa. Once the file is uploaded select Next step.
  6. Configure the settings - On the configure page, select a test suite. If you don’t have any tests available, choose Built-in: Fuzz to run a built-in test suite. Select next step. On the Select devices page, for Device pool, select Top Devices to select the device pool, then click Next step. Add any additional setting needed on the Specify device state page.
  7. Start Test Run - On the review and start run page, select confirm and start run.
  8. View the Results - The run is complete once the progress icon changes to a result icon. Choose the completed run in the Device Farm console to view the results.

Pricing: AWS Device Farm comes with a free trial of 1000 device minutes. The base price is $0.17 per device minute. There is also a flat plan for $250/month that allows you to run as many tests as you want on one device at a time.

Strengths: Automated testing and remote access; 364 real devices; robust platform. Streamlines the testing process and saves money by providing a wide range of access to the latest hardware, platforms, and operating systems. You also have the option to buy a private device which is deployed on your behalf and removed at the end of your subscription.


Visual Studio App Center

Like Firebase and AWS, App Centre supports Android as well as iOS apps while providing not just testing but a complete CI/CD system. You can not only test but also build. However, the option to test is also available as a stand alone option. App Center has over 500 devices. It is both Android and iOS devices where Android goes from 2.2.x to 9.x and on iOS there is support from version 9.x to 12.x.

Several popular test frameworks are supported including Appium, Espresso, Calabash, Xamarin, UITest and XCUITest. However, App centre doesn’t support UIAutomator tests and doesn’t have its own built-in test system like Robo of Firebase.

Getting Started with App Center Test:

  1. Create an App Center account.
  2. Install the App Center CLI to execute the test runs.
  3. Create an App Project in App Center
  4. Prepare your test for upload. Steps vary on the test framework you use.
  5. Start your test run by first selecting the devices and then selecting the test series, system language and test framework to use for the test. It generates a prototype command that can be run using the CLI.
  6. View your test results. The test reports dashboard shows test pass and fails for each device and provides screenshots of every page in the app for each test run.

Pricing: It is free to start with for first 30 days on one device. After that its $99 a month for 30 device hours which turns out to be $0.055 for every device minute. There is also an enterprise pricing option at $499 but with unlimited device hours.

Strengths:

  1. The App Center Build service helps you build the mobile apps. It supports building apps in Git repositories hosted on Azure DevOps, Bitbucket and GitHub.
  2. You can distribute apps to beta users easily. The speed and integration capabilities make this tool perform admirably for release of pre-production or beta software to testing devices. App Center Distribution supports the distribution of application packages for iOS, Android, UWP, and macOS.
  3. Distribute apps directly to stores from App Center. You can directly publish your iOS Apps to the Apple App Store and Android Apps to Google Play Store.

So, which one is the best?

The answer to this one is also ‘It depends’. All the three options offer different features and thus will suit different developers. It is important that you pick a cloud platform suitable to your needs and test your application thoroughly before publishing it.