How to Automate Mobile App Testing

This article arms you with foundational information and knowledge of tools to get you started with test-automation for your mobile apps. There are many resources and tools out there which can be overwhelming to some. We are here to break it down for you. Try it yourself.

Considerations

We are going to provide some answers to these questions for you.

  1. Why invest in test automation?
  2. Which parts of the mobile app should you automate first?
  3. What tools should you use to create test-automation scripts?
  4. How to set up your testing infrastructure for automation?
  5. How to execute your tests successfully.
  6. When should you run the automated tests and when should you perform manual tests?
  7. How to measure the success of your test automation investments?

Why invest in test automation?

  1. Testing is slowing down the app release cycle - Perhaps your team needs to release features faster so you want to reduce the manual effort required to run the regression tests, and allow the testers to focus on testing new features.

  2. There are many errors reported after the release deployment to the App Stores - Automation supports continuous delivery efforts so that each commit can be deployed safely and frequently. The faster the feedback, the less expensive it is to fix an issue that is found.

  3. You want to challenge your Quality Assurance teams and get them excited. When the goal is set to increase testing speed and coverage, the QA team finds ways to be more innovative.

Which parts of the mobile app should you automate first?

Strategically consider the areas that would benefit from automation and specific components to focus on for testing. When done right, it will speed up your testing process and give your team more time to focus on other aspects of testing, which are more exploratory and hacky in nature.

Define the scope of the tests that will be automated and decide on the most urgent testing priorities. This would include any core component already thoroughly tested by the manual team. Consider the frequency of testing, business priority, and what parts are capable of being automated. Anything that can be automated has the potential to save time and money in the long run.

In reviewing your current manual testing process, look at the functional test-cases which:

  1. Are the most time consuming and mentally exhausting for your manual testers
  2. Can be simplified and structured into logical steps and expected outcomes
  3. Are critical core-flows in terms of the app’s functionality
  4. Have been known to cause issues in the past
  5. Require a large amount of data for testing

Tools to create test-automation scripts

There are plenty of automated testing tools on the market. Choose a reliable tool that works on your app, meaning it supports the platform and technology, integrates best with your current process, and matches the skills of your QA team.

Appium - Open-source, can be used to create a single test script for multiple platforms. You can find more information on getting started here or subscribe to the Appium pro newsletter which has a wealth of helpful information for the Appium community.

Espresso - Espresso is Google’s Android UI Test Framework. It is used by developers to write UI tests for black-box testing. Espresso is easy to learn and customizable for those who understand the codebase.

Record and playback tools - For those who don’t want to learn to code for testing and quickly get started on their test automation journey. These tools are combination of testing frameworks with a graphical UI, which makes it easy to use. You can simply build tests by recording a series of specific user actions on selected elements in the app, just like a test script. The tools generate the test-automation code on the fly for you. Try out Appium Desktop. Appium Desktop is a graphical frontend to Appium with additional tools.

Although open source test frameworks provide you with more granular control over your test script, they are difficult to learn and code. You might even need to hire a Test Automation Engineer to do this. When you are just getting started with test automation you may want to try out the record and replay option. While it may have less features and functionality, it can help you get started with automating your basic test cases and fetch you more time to do exploratory testing.

MoQuality is working on making it easier to get started on the test-automation journey. MoQuality’s Barista was developed with the idea of helping manual testers to automate tests without writing any coding. Using Barista, you can easily build test cases for your mobile apps. There is no test infrastructure setup required. It provides the ease of creating tests visually and combines the powerful customizations of underlying open source test frameworks. The tests can be executed on real mobile devices on the cloud.

How to set up your testing infrastructure for automation?

Once you’ve selected a testing tool and know the scope of your automated testing, you can create the testing infrastructure. Test cases should be simplified into a logical sequence of user actions and expected outcomes that are easy, fast, and repeatable.

Identify commonly run test scenarios which look at the core functionality of the app. To start, pick the less complicated scenarios which can be executed in 10-15 steps. Effective strategies for creating test automation suites are to:

  • Create end-to-end tests covering the core user flow such as login, signup, checkout, etc
  • Add unit tests on data sensitive code functions such as tools used for monetary transactions
  • Execute integration tests for points of 3rd party integration to ensure proper data flow
  • Create automated tests early in the process to generate data that a human tester can evaluate and assign as pass or fail
  • Automate use cases that are most likely to fail - To set the right expectations for test automation consider the likelihood that a given test will find a regression in a particular code change

Executing your Tests

After planning and designing your test-automation scripts, it’s time to execute your test. Refer this blog to choose which is best for your use case Real Devices or Emulators. Ways to get started:

  1. Running tests on Emulators - Prepare your app for test and run it using Appium on Android Emulators or iOS Simulators. Here are the steps.
  2. Real Devices locally - If you are using appium server on your machine you can simply connect your mobile devices and run the test scripts. Here is some more information on how to setup parallel tests using Appium.
  3. Cloud-based testing using platforms such as AWS Device Farm or Firebase Test Lab provides accessibility anywhere and anytime with the opportunity to scale. A vast variety of software and hardware environments can be accessed and there is no need for maintenance.

When should you run the automated tests and when should you perform manual tests?

In deciding whether to test manually or automate, consider the types of tests that will be run. Discuss with the testing and development teams, what key areas would benefit from automation.

Areas where manual testing may be better include exploratory, user experience, and accessibility testing. Reliability of test results should be more important criteria than automating wherever it’s possible. It’s not necessary to automate all testing, tests that only need to be performed a few times are best left to manual testers (with the exception being the test requires a large set of data to execute). Generally speaking, the more repetitive tests that require a large amount of data are best to automate.

It makes sense to move from manual to automation testing for testing types such as:

  • Smoke Testing: A core set of smoke tests can be developed once the unit tests are passing on a consistent basis. They will help to quickly determine if the build will require further testing or not.

  • Regression Testing: Used to test old and new functionalities after every release from one test cycle to the next.

  • Data-driven testing: Functionalities that will require multiple sets of data for testing or have complex calculative areas that can result in human error.

  • Cross-device testing: Manual testing continuously on multiple devices takes a lot of time and effort. Automating the application and running of the tests will save time and ensure that applications are functional and accessible across a large range of devices and OS versions.

  • Performance Testing: When there is not a manual testing option and tools are needed.

Once the automated tests are complete, manual tests should be run on the verified build passed by automated checks. It is your final test before the apk is released.

How to measure the success of your test automation investments?

Automation projects can fail due to unrealistic expectations. It’s important to know the end goal and keep track of the success of your automation projects. This way you know the return on investment (ROI) and it ensures that you’re meeting the test expectations set forth by your team. Any areas where progress is not being made towards the end goal can be reevaluated.

Evaluating Test Results

As tests are completed, the results are reported with the overall outcomes. You will want a tool that can log videos and screenshots of every test and take you directly to the bugs so they can be easily shared with developers. In the test report, you can measure and monitor your quality metrics. Quality metrics are helpful and can be considered like the north star for QA teams. Some examples of such metrics are:

  1. Bugs reported in production - The goal should be to track your progress before and after releasing your code. A metric to measure this is how many defects are found during pre-production testing versus in production. Give your users a way to report bugs in your app to you. It can save a bad review going public.

  2. App crashes - If an app crashes it’s a significant concern because it may be uninstalled or removed by the user. It’s a severe issue that can cause a loss of interest in the brand. For this reason, it’s important to carefully track and address any crashes that show up.

  3. Bad user reviews - Bad reviews in the app store can be damaging and are not something your team wants to end up with. Critiques can be helpful but not about bugs that could have been fixed before production. Continuous deployment is closely tied with high reviews in the app store and apps that are updated more frequently tend to be rated higher.

Before starting you automation project, make a note of these metrics along with the time it takes to execute manual testing. Although a lot of time is spent in the beginning building an automation suite and making sure that it’s robust, as more areas are covered, it will start to pay off and regression tests will take minutes rather than weeks.

Test automation is a continuous journey. The aim of test automation is not to eradicate manual testing, but improve your current test suite and reduce the number of test cases that are run manually. This gives you more time to do a thorough check. Although automation can save time and money in the long run, there will need to be initial time budgeted to write and configure the tests and then an ongoing time consideration for maintaining the tests with each new deployment. Keeping this investment in mind, automated testing can pay for itself over the long term and increase the stability of your software so that you can be confident that you are releasing high quality products on time.