Learning Test Automation from Facebook's Mobile App Testing

Ever wondered how mobile apps with a large user base such as Facebook are developed and tested. Outlining Facebook’s continuous deployment process, we look at best practices and the benefits that shorter release cycles provide that can help your team improve their current workflow. A look under the hood shows how Facebook executed on their vision with careful planning and experimenting with new techniques to achieve such a fast pace of development.

First we briefly explain what it means to have continuous deployment and then dive into specifics of Facebook’s testing strategy.

Continuous Deployment is the progression from Agile development to Continuous Integration (CI) to Continuous Delivery (CD). Here is how it works:

  1. The software is developed iteratively in Agile development
  2. In CI, the software updates are integrated at the end of each cycle after they are verified by an automated build and tests
  3. CD ensures that the software is built, tested and made ready for production
  4. Continuous Deployment automates the deployment of software to production as soon as it’s ready.

Shortening the Deployment Time


Image Source: Rapid release at massive scale, by Chuck Rossi

Over a period of 4 years, Facebook has mastered the way to deliver code faster, while at the same time increasing the quality significantly. The Android release went from a deployment every 8 weeks to a deployment every week. This change can be attributed to improved development activities, deployment activities and use of automated testing tools.

Development - The developers push code to master branch 3-5 times a week. They use a mechanism called Gatekeeper which dynamically allows them to control the availability of features on mobile devices via server-side (called Feature Flags). It protects the app from misbehaving due to new buggy features and can also be used for A/B Testing.

Deployment - Facebook’s Release Engineering Team cuts a Release branch from Master every week. Certain important updates during this time are ‘cherry-picked’ to be merged onto the Release branch. This app version is made available to Facebook’s internal users as ‘dog food’ to stabilize the branch. A Beta version of the Android app is also shipped to ~3 million external beta users every day via Release branch. The soak period only accepts launch-blocking updates.

Release - In the next three days, the app is incrementally rolled out to first only 20%, then 50% and finally 100% of users. This way the push can be stopped if any problems arise.

So how did Facebook end up with higher quality code going live if they increased production pushes so significantly? It seems near counter-intuitive; how can you increase the speed without sacrificing some quality? In order for continuous delivery to work and achieve high quality, near error-free code, a rigorous automated testing process to be run alongside development had to be implemented.

Continuous Testing on the Facebook App

Facebook applies many types of tests and most of them are run in an automated manner. Let’s take a deeper look.

a. Types of Tests

Facebook uses numerous types of tests including unit tests, static analysis tests, integration tests, screen layout tests, performance tests, build tests, capacity tests, conformance tests, and manual tests. Both white-box and black-box testing strategies are applied by running tests on simulated and emulated environments.

b. Testing Infrastructure

Facebook runs a real device lab at its Prineville Data-Center. It’s outfitted with a custom-built mobile device rack that allows for automated testing of mobile and web applications to be run on thousands of real mobile devices. The lab can be accessed remotely, with cameras set up to observe how each device reacts to a change in the code. The mobile lab has a focus on testing for regressions, looking at speed, memory usage, and battery efficiency.

c. Test Automation

A dedicated developer team is focussed on developing tools. Facebook improved its test automation with increased regression testing. Automating tests running as often as possible allows regression testing to be performed frequently and with precision. When a bug is discovered during a regression test, an attempt is made to auto-detect the developer who caused the regression so they can be notified via email immediately. Smoke tests are run within 10 minutes of a developer build, targeting specific changes to the code and high-level functionality.

D. Testing Tools

Various testing tools are designed to improve the workflow. Chef, is a tool used to manage the configuration for test environments. Facebook also makes use of multiple tools such as XCode, Android Studio, XC Test, JUnit, and Robolectric.

E. Manual Testing

Manual testing is utilized as well since not everything can be automated. This is handled by a contracted manual testing team of about 100 people who primarily test new features on mobile apps before automated tests are written. Smoke and edge-case tests are performed to ensure that the app’s behavior is as expected. Finally, they test the final look and feel of the UI to ensure it’s usable.

F. Quality Metrics

Quality Metrics are the north star for testing. These include:

  1. The number of issues and crashes found in production code as a measure of quality
  2. Number of false positives and false negatives to determine the precision of tests
  3. Issues reported by users

When are tests run

Pre-Push Testing: A developer frequently runs many unit tests locally or on the servers in simulated environments as when required during development. There is no separate testing team. After a code review, the code is ready to be pushed to Master.

On Push: Before the actual push occurs, a number of unit and smoke tests run to test most commonly used features and key flows. The build is merged onto master and any conflicts at this stage are addressed by the developer.

Continuous Testing: Comprehensive tests are run on both Master and Release branch every few hours. These include build tests, integration regression and performance tests in the mobile device lab. The Alpha and Beta versions are tested daily with a dedicated user base and these app versions are blocked if errors exceed a certain threshold.

What we learned from the continuous deployment of mobile software at Facebook

Shortening the release cycle proves to be beneficial in the following ways:

  1. Faster time to market - This allows you to experiment with new features, provide a better user experience to your customers, and respond to user feedback.
  2. Improved quality - Your developers are less likely to rush last minute code commits to making the deadlines as next release cycle is coming up soon enough. An analysis shows that code committed on the cut-off date is of lower quality. Also, it reduces the number of files worked on concurrently thus lowering chances of errors even further.
  3. Increased productivity with improved tools and workflows - Your teams are forced to innovate and push harder to make the continuous deployment a success.

As we mentioned automated testing plays a key role in achieving short release cycles, continue reading our article on “Getting Started with Public Device Clouds for Mobile App Testing”.
This article is derived from Facebook’s whitepaper on Continuous Deployment of Mobile Software at Facebook. For more information, read the full article here.