Real or Emulated: What's the Best Choice for App Testing?

Everyone wants a mobile app of some sort these days, and they want it for every major platform. That’s great news for developers who need the work, but mobile app development comes with plenty of complications you don’t get when developing software for Windows, Linux or MacOS.The most obvious problem is that the machine you do the actual development on is nothing like the device that must actually run the software! So how should you approach software testing in this tricky context? That’s a great question! However, the answer needs some explanation.

App Testing’s Big Problem

The most effective way to test if an application works as intended, is to use the same hardware the end-user will. However, there are thousands of handsets! All with different specifications and hardware configurations. That’s before you factor in that some manufacturers like to customize the OS. Which means it is impossible to test your app thoroughly for every real phone or tablet hardware configuration.

At best, most developers can afford a few handsets, which hopefully represent as much of the install base as possible. Luckily, you can use a technology known as emulation to expand your testing methodology without having to purchase a truckload of consumer hardware.

So What Exactly is an “Emulator”?

An emulator is a software application that recreates the hardware environment an app is designed for on a system with a different architecture. Think of it as putting your app into the Matrix. It thinks it’s running on an actual phone, but actually it’s a virtual device that only exists as lines of code running on another machine.

The Case for Emulation

There are many reasons why emulation has become such a popular way to test applications. For many developers, it’s literally the only choice they have. At least until their first app makes it big! If you’re starting to sweat as the testing phase of development nears, here are the main things you should consider about emulation:

  • Emulation is cheap - compared to the cost of buying real test devices, emulation is much easier on the pocket.
  • It’s easy to spoof locations and modify network parameters.
  • Emulation makes it easy to test your UI under different screen shapes and resolutions.

That doesn’t mean emulation is perfect! There are some serious limitations too when compared to real test devices.

Real Devices Versus Emulators

So why do we even bother to test on a subset of real devices? If it works in the emulator, then surely it will work on any real device, right? The truth is that emulation usually comes with more than a few compromises.

For one thing, emulation is much slower than using a real device. Emulation strains hardware significantly. So the only real way to know if your app will perform well, is to test it on devices that represent common performance classes in the mobile market.

There are some ways to improve the situation though. On Windows you can use Intel’s HAXM to accelerate aspects of Android emulation. If you’re developing on Linux then you may get better performance running an Android guest on KVM.

You also can’t emulate motion sensors properly, so if your app depends on them you’ll have to do some testing on a real device.

Finally, emulators only emulate a vanilla Android OS. Which means that manufacturer-specific bugs can crop up when your app makes it out into the wild.

Android Emulators

There are plenty of emulators for Android by now, but not all of them are made equal. First of all, some have worse emulation than others. It could be that a flaw in the emulator itself could create false positives for bugs.

Apart from quality differences, Android emulators also support different CPU architectures. Most modern devices use the ARM architecture, but some also use the X86 architecture that’s found in the majority of PCs. x86 Android devices are rare, but phones like the Asus Zenfone 2 use Intel Atom CPUs, which are x86.

The Samsung Galaxy S9 on the other hand, is an ARM device. Obviously you should use an emulator that matches the architecture you are developing for. Otherwise it won’t even run!

Android Testing in Docker

Docker is a service that deserves its own article, but in this context it can help your testing by letting you run code on your desktop machine, without any interaction with your own native OS. You can use it to manage dependencies and isolate your applications, which provides a pristine and consistent development and testing environment. Docker is still pretty new though, as is the idea of “containerized” applications. So the learning curve is still a little steep.

iOS and Simulators

Simulators and emulators are often confused with each other, but they could not be more different. A simulator presents the user with something that looks and acts the way the real thing is expected to act. When you sit in a flight simulator, it looks and acts like a plane. Well enough that you can literally learn to fly a real plane by using it. However, you are not really in a plane. You can’t crash or travel anywhere. It’s an illusion. The same goes for software simulators.

Apple’s MacOS Xcode software tools are a primary iOS development resource. Xcode comes with an iOS simulator. When you compile your app for this simulator it’s translated into x86 code. It looks like the app when running, but it’s not exactly the same code that will run on the device.

The Opinion on Emulators is Divided

Most developers will probably admit that emulators are a necessary evil when it comes to app testing. No matter how good the emulation, it will never be exactly the same as real hardware running your app. Which means that certain bugs or constraints can still slip through. Relying on emulation only inevitably means your users end up being your real-world testerst. Which means more work post-release to patch those faults as they are found. Still, emulators make development possible where otherwise no apps would be written at all! So for now it will remain a core part of app development and testing at all scales.