Good testing is always performed in a sandbox environment. When you test in production environment, you will end up polluting your production data with test data. In some instances, it is not possible to test in production at all. For example, you can create fake financial transactions in your staging environment, but not in your production environment.
If you currently don't have a sandbox environment to run your tests, you can do the following create test accounts and run tests against them. After the tests are done, you can clean up these test accounts.
You also need to consider sandboxing your device. When you test on the same device, again and again, you tend to bias user experience to that device. We have observed that sometimes tests work locally, they don't work on cloud devices. This happens due to several reasons:
- Your app relies on cache data: When a new user installs your app, they don't have your app's cache data.
- Your app relies on cookies stored in the browser: All devices do not have the same browser or even if they have a particular browser, they might have different versions. It is usually a good practice for native apps to not rely on browsers for authentication.
- Your app uses the system's Google or FB login: Your personal phone can have a pre-authenticated user account, but cloud devices don't. It is an important consideration to remember while designing your tests.
Some apps rely on authentication using OTP SMS or magic links via email. Consider the case of authentication using SMS. You could probably write a test that works locally, but it won't scale on cloud devices since some of the cloud devices are not connected to a mobile network and even if they are, most device cloud platform don't provide a phone number.
Authentication using one email has the problem that you can't run tests in parallel. If you want to run tests in parallel, you will need to create multiple email inboxes.
We have helped our enterprise clients solve the OTP authentication problem in testing at scale on cloud devices. If you will like to know more, please write to us at [email protected].
Creating a good test suite can be a work of art. However, there are some established principles we can follow to ensure we have created a good test suite.
A good test should have the following properties:
A test should be repeatable. The best way to do this is to ensure that you reset the app before starting the test again. For example, resetting the app ensures that the user is logged out every time and the test is performed on a fresh app.
A test should be concise. If a test is testing multiple functionalities, it should be divided into multiple tests each of which tests a single functionality. Then you can rearrange these tests to create a longer test. For example, user registration, login, and sign out should be 3 different tests.
A test should be specific. A test should test functionalities that are measurable. For example, if your app orders a list it is hard to check that the order is "correct". It is easier to write a test that checks the first few items in the list and checks for specific values.
A test should be debuggable. The key to setting up a culture of testing in your team means making testing useful. Tests are going to break because you have an agile dev team that moves fast. If the tests are debuggable, then you can make your testing process agile too.