Why is it important to keep mobile tests inside the native development framework?
by Jas Manigundan
Mobile app testing has a number of similarities to testing on the web, but there are some aspects that are totally different. In this article we will be exploring my personal experience with mobile app testing. I am Jas, an Android developer turned mobile app tester. With the help of concrete examples, we will discuss why it is worthwhile writing mobile app tests within the native framework. Every test tool or framework has its place. We will not be talking about the pros and cons of a specific framework. Instead we will talk specifically about my experience with native mobile frameworks of two of the most popular mobile OS - Espresso for Android, and XCUI Tests for iOS and why you should give it a try. We will also explore some good examples to get you started with your own mobile app testing journey.
What do I mean when I say native mobile app frameworks?
Android and iOS apps can be written in a variety of ways. The most popular way is to write apps using the tools and frameworks provided by Google and Apple respectively. For Android, we have Android Studio, the official IDE to write android apps using Java or Kotlin. For iOS, we have xcode, the official IDE provided by Apple to write iOS apps using objective C or Swift.
But there are other ways to write Android and iOS apps. Recently, React Native and Flutter have gained a lot of momentum to write cross platform apps. Previously, Cordova, Ionic, Phonegap, Xamarin have also been used to develop cross platform apps including for Android and iOS.
When it comes to testing mobile apps, there are a variety of frameworks or tools to do so. The testing frameworks that come along with the native development frameworks are Espresso and XC(UI) tests for Android and iOS respectively. But again, these are not the only ways to write tests. Appium is also a very popular framework to write cross platform tests. Lately, there has also been a lot of GUI based services that make use of the previously mentioned frameworks to record tests.
In the following sections, we will be discussing how these native frameworks have helped me to achieve some of the things that I needed as part of my testing, with real time examples.
Setting up the state of the app using Launch Arguments
First things first. Setting up a test is by far the most important step, especially with mobile apps, to achieve efficiency in the way you run tests. Let's start with an example.
Real time scenario: Onboarding screen.
Most of the apps have an onboarding experience, which will pop up during the first run. But subsequent runs will not have the onboarding screen. So when you have run two tests after each other, you will need to consider how you handle the onboarding screen.
Some of the ways I’ve seen testers handle this is..
This is one of the main reasons for flaky tests, because the state of the app at this point is still unknown. A good test in my opinion, should know what the state of the app is exactly and should assert based on that. But the best way to tackle this is to set the state of the app by passing state based values/flags to the app through launch arguments, in this case, showOnboarding. Your test will now have a setup section where you pass in this flag. When you are running tests on the onboarding screen, you set this to true.
Another way to handle this scenario is to have a launch argument to define the screen you want to test, so the app would handle the landing page of your test.
Mocking network requests
Real time scenario: Skipping network calls during tests for efficiency and consistency.
Let’s say you want to write a test to check the carousel content that hosts the latest deals. Every time you hit the API, there is no guarantee that you will receive the same items. This is when in app mocking becomes really helpful.
For example, in Android, we could potentially use MockWebServer from Okhttp to mock the network layer. The best part is that you don’t need to touch the app code here. You should be able to manipulate network requests from your tests itself.
Your code will look like
This way you have the ability to inject a test response and set up the app the way you want. It ensures consistency so that you can assert on controls with 100% certainty that data isn’t going to change. This is by far my favourite use case.
Real time scenario: Testing localisation for 15 languages
One of my recent projects went global. We had to localize the app for 15 regions and testing it was one of the most challenging things that we had to do. The biggest challenge being able to test the releases across 15 regions (or languages), across different device configurations. Instead of spending hours of manual effort on some of the trivial tasks, we developed a visual testing framework that required one time baseline creation. The following test runs had screenshots of the screens tested against the baseline of that particular language.
This was very much possible because of the support that we had from the native testing frameworks. Taking a screenshot is just as simple as calling the activity.
Un-isolated test runs
I have seen testing being considered as an isolated operation that is performed after development. I am not a fan of the sentiment personally. Having native tests helps break that.
Real time scenario: Developer wants to run the tests and contribute to it
When tests become isolated, it becomes hard for people other than testers to be in touch with them. But when your tests are sitting along with the app code, it automatically intrigues the developers. This is something I have noticed across different companies and different teams.
Running tests is now just a click away. This also helps maintainability in the longer run.
It also allows testers to add new tests since the tests are sitting in a framework they are very comfortable with. They also tend to see the value the tests bring. It is no more something that is too alien for them.
Real time scenario : An accessibility ID for a button has changed.
This is a very common scenario that breaks tests. When tests are siloed, changes in the app don't navigate to the tests. But with native test frameworks, when a button is refactored by the developer, it is automatically refactored in the UI test instance as well. If some of the components are modified, the test code won’t compile, which the developer can notice right away instead of waiting for feedback from the tester after the development of that component is complete.
Running tests as part of Continuous Integration
Unlike running tests against a WebUI, running tests on the mobile via Continuous Integration (CI) requires added configuration. Setting up the simulator, the right scheme or build variant etc. have to be set up properly.
Real time scenario : Running UI tests via CI
When you have a CI pipeline that builds the app, adding native tests is just another step in the pipeline, given the code is already in the same repository and the same native development framework. The configuration like variant, schema are already set up with the app build process. So running tests just becomes a single command.
With Android, building the app is done through running the following command:
And running the UI tests is done through running the following command:
Some additional benefits…
Tester - Developer collaboration
This is something I have noticed in every app I worked in. All of a sudden, the developers are more interested in how the tests work and want to actively contribute to the testing process. A larger portion of this is due to the fact that the developers now understand what tests are being used and how to run them in the first place. In some projects, I have also noticed that the unit tests and UI tests go hand in hand with most of the code and test data being reused across the aisle. This also helps reduce duplications of testing and test data across both levels.
Better understanding of the code itself
Another advantage of working with native frameworks is that it forces you to understand the app code itself. Right from adding an accessibility ID to writing code to enable launch arguments, performing minor tasks like this in the app code, allows you to understand the app code. And over time, the curiosity also adds in and testers usually tend to dig deep into code to find better ways to enable testing. The testing approach also changes from black box approach to gray box testing. This also very much increases the testability of the app, given you have a tester looking at the code at a very different perspective.
One of the best parts of writing tests in the native framework is the toolset that comes along with it. We get access to the same level of tooling that developers have. Starting from Layout Inspector to having breakpoints in app code to see what the tests are actually doing, it is a totally different world out there.
Overcoming some of the challenges
Cross platform testing?
A common question that I keep getting is why do you write every test twice, one for android and one for iOS? As much as I don’t agree with the sentiment, I agree with the context switching for testers. The way I navigated this issue is by having a layer of abstraction.
My tests in both android and iOS look like this..
The first layer of the test also remains the same across android and iOS. For example, let’s take a look at the “when_login()” function.
The only section which has the native level coding is the Page Object itself.
For example the tapSubmit() function will like this..
Where do I start?
Now that you are really intrigued, I would like to recommend some resources to get you started.
For Android, I’ve found the best way for me to get started is to look at the official samples here https://github.com/android/testing-samples
The most common place where testers usually struggle is to test list views (known as recycler views in android). This is a great place to move past it once and for all. https://github.com/android/testing-samples/tree/master/ui/espresso/RecyclerViewSample
For iOS, there are two tutorials that I really recommend https://www.raywenderlich.com/960290-ios-unit-testing-and-ui-testing-tutorial and https://www.hackingwithswift.com/articles/83/how-to-test-your-user-interface-using-xcode.
Might also be interested in
Beginner's Guide To Mobile Testing - Daniel Knott
Testing Ask Me Anything - Mobile Testing - Daniel Knott
Mobile App Testing Mnemonic - Daniel Knott
I’m Jas, a developer turned tester. I have spent most of my life doing some sort of programming. I started by drawing rectangles and circles using LOGO and now I work on building mobile apps for a living. I can speak 5 human languages and 5 programming languages 😉.
Even though writing code to build applications is so much fun, my now best friend showed me a few years ago that helping write good quality apps is much more fun. So I jumped into testing from being a developer and have never looked back since. It has been a very rewarding transition for me.
I am very passionate about native mobile testing frameworks, especially Espresso (Android) and XCUI testing (iOS). Most of my day-to-day work involves testing mobile apps using native testing frameworks, but at the same time, making it as easy as it can be for developers and fellow testers to contribute to it.
When I have some spare time, I spend it trying to automate some of the boring tasks. Which brings me to my favourite quote.
Automate within reason.
- A very wise friend of mine.