So many apps depend on Bluetooth functionality these days so that mobile phones can interact with other devices and sensors. Think of apps connecting to your Bluetooth speakers, home automation devices, wearables and much more. From an app developer's point of view, it can be a challenge to test such functionality. Imagine an army of testers manually pairing and unpairing devices over Bluetooth for each test. In this article, we share our insights on how you can automate Bluetooth testing.
Why do we want to automate Bluetooth testing?
Today, there are hundreds of smart devices available for purchase and thousands of apps supporting connectivity via Bluetooth. For both the app developers and device manufacturers it becomes critical to test the Bluetooth functionality on most popular devices and apps at a minimum. Most often these tests are run manually which becomes time-consuming and frustrating sometimes. Quality assurance teams can gain efficiency by creating automated tests that are easily reproducible for testing Bluetooth features.
Testing Bluetooth accurately can be challenging at first. It’s a technology that seems simple enough for the user who typically just sets it up one time, however, there’s a lot of complexity going on behind the scenes. You are dealing with two connected devices that may have different development cycles, and any synchronization problems can lead to an issue.
When Bluetooth fails, it can be for several reasons, including not properly handling all the state combinations, or due to a manufacturer’s faulty OS customizations or device compatibility issues that need to be handled in the app's code. Being able to quickly identify a bug in an app's Bluetooth feature is important. Connecting to Bluetooth is often the first step in running the app and you can’t proceed otherwise.
Therefore, it’s important to have a good testing process in place.
Bluetooth feature tests inside apps are dependent on whether we want to test the end to end functionality or just emulate different states:
- On Real Devices, you can write test automation scripts to automate Bluetooth functionality. Real devices have the benefit of testing real-world conditions and can provide valuable insight into how the app will function in a variety of situations. Due to the range of platforms and devices, it is not feasible to test manually across them and thus automation is needed. However, as we will show later, setting the devices in different states can be tricky with automation.
- On Emulators, the Bluetooth functionality can be faked. It is useful in situations where having access to real devices may be challenging or budget constraints affect the ability to invest in a device lab. In addition, emulators can allow setting the device in different states that make certain tests possible. Although emulators are helpful, they do not accurately mimic the intricate nuances in real devices and thus cannot completely substitute real device testing.
Automating these basic tests creatively and plugging them into the CI/CD pipeline can decrease the time to deliver new features significantly. It can also help to avoid false positives and ensure the results measured are not affected by the performance of the network.
To gain a little more insight into how companies are performing their testing, we talked to FIXD Automotive. Their product is a sensor that connects to the car and displays relevant information about the car's health and potential maintenance alerts on the smartphone. The FIXD sensor plugs into the car's diagnostic port (called the OBD2 port), collects the information in realtime and communicates it to the companion mobile app periodically via Bluetooth. To understand how FIXD tests its product, we talked to their CTO, Charles Julian Knight to understand their testing process, and how they are thinking about automation.
How does FIXD do testing today?
Currently, FIXD performs end-to-end tests manually both with a real car and with an On-board Diagnostics (OBD2) emulator.
CASE 1 - Testing Manually inside a Car (End-to-end Tests)
Testing the sensor manually involves setting up the test scenario to ensure the application is free of errors and working according to all the specified functional requirements.
Here’s an example of a manual test scenario:
- Launch FIXD App and find your FIXD sensor on Bluetooth search screen.
- Pair the devices.
- Verify connection with the car.
These steps are repeated manually by team members for each build on each device every time.
CASE 2 - Developer Testing using Emulators (Unit Tests)
After a developer build is released, the new app version is installed on a phone emulator. It’s then connected with virtual FIXD Sensors without the need for an actual car. The developers program the car emulator to emulate various scenarios of car maintenance and check engine light notifications. Secondly, to perform a quick check on the new build, where the developer just wrote a new app function or modified the code slightly, they need not focus too much on Bluetooth testing, but it is a necessary step to proceed and test other app functions. In this case, they run the app on an Android Emulator and fake the Bluetooth functionality using dependency injection. Dependency injection can be accomplished manually or by using a library, such as Dagger or Kodein on Android. The example is posted at the end of this blog.
How does this work? The FIXD App’s test version basically bypasses the need to access Bluetooth hardware, and it calls this test API which basically emulates functions of the car and provides input to the app.
This is a good approach for developers to quickly test their new code without going through the entire end-to-end case. But in an ideal world, FIXD wants to automate this testing process on real mobile devices and FIXD sensors.
CASE 3 - How do you get to the ideal integrated test-automation solution with real phones and a Bluetooth device?
The basic question was how can we test the device systematically? The goal, in this case, would be to make sure all the Bluetooth features work as expected on real devices.
To do this, the scope of the testing must be determined along with which test cases will be automated. Then the automated tests can be written and included in the Bluetooth Test Protocol (BTP).
The tests will be used to communicate with the Implementation Under Test (IUT) which is the host running the Bluetooth. This could be an emulator or actual hardware. Wondering how they simulate the OBD2 port states? There is a device similar to the Raspberry Pi, that can simulate car functions. The FIXD sensor just snaps to this car emulator.
Integrated Test-Automation Solution for FIXD
For FIXD there are three parts to the automation solution we imagined. The areas of testing can be divided into Setup, Script, and Teardown.
Setup (Phone, Sensor and Car)
- Install FIXD app on the phone
- Set Car Emulator in State A (API Call)
- Turn Bluetooth devices on and pair them
For the setup, install the app on the phone, set the emulator for the car, and then turn the Bluetooth on. Using automation tests, different Bluetooth profiles can be tested such as pairing, hands-free, file transfer, etc.
Script - Write test-automation script to simulate the steps below:
- Open FIXD app
- Navigate to add the device
- Add FIXD sensor
- Verify connection
- Run the other test scripts.
- (Optional) Simulate resetting Bluetooth connections.
The goal for the automation test covering the script is to confirm the device and FIXD sensor were added, verify the connection, simulate resetting Bluetooth connections, and run any other test scripts created that cover features besides Bluetooth.
- Uninstall App
- Clear data on sensors
- Set car emulator to State B (API Call)
Finally, once the testing is complete, we want to uninstall/delete the app, clear the sensor data, and set the emulator to State B. This can be included as commands in the automated script.
Creating Your Own Automation Solution
To create an ideal automation solution focus on automating test scenarios that help identify issues related to connectivity, functionality, and implementation. You can look at the Setup, Script, and Teardown categories and decide which tests will be most important in your CI/CD pipeline.
Examples of What to Test:
- Power supply On/Off functionality.
- Ability to search all active Bluetooth devices within range and the radius.
- Setup pair connection with other devices.
- Disconnect after use.
- Battery performance and charging time.
- Test different types of devices to make sure all are supported.
- Ability to transfer files.
- Ability to connect with one device for a specific purpose and multiple devices for different purposes. For example Wireless keyboard, headset, etc.
Mock Bluetooth Interface Example
This is an example of how FIXD mock out the FIXD Sensor on test platforms that don't support Bluetooth, such as the Android Emulator. They declare an interface for the service. The real app uses the real Bluetooth implementation, but under the test Gradle build variant, they use dependency injection to provide the Mock implementation instead. This isn't exactly the code in the FIXD app, but it is representative. On iOS, the dependency injection is based on compiler flags instead of Gradle build variants, but otherwise, the method is the same.