Mobile Testing – That’s Just A Smaller Screen, Right?



There is no denying that mobile phones, smartphones in particular, are hot these days. Mobile phones have become an essential part of daily life, and as an essential part of life, they also need to be reliable, secure and easy to use. Imagine if the alarm on your phone didn’t go off in the morning, your old texts suddenly went missing, or you couldn’t make that important call.

As mobile phone usage has increased and mobile applications have become more and more important, so has the need to test them become more important. It’s an exciting part of the industry to be in. So if you have ever thought about a role as a mobile software tester,then read on and find out what it’s all about.

Mobile Software Testing Has Many Flavours

Mobile phones are pretty complicated things. Under the hood there is a lot going on, and so as a software tester it’s important that you decide early on where you want to focus your efforts. Broadly speaking you are looking at:

  • Hardware level testing – this is testing at the device driver layer, where the software controls the interface to the processors, screen, memory and the radios themselves. This is ahighly specialised area and somewhat complicated to get started in. Testing is primarily performed as part of development work, low-level, and automated.
  • Protocol testing – your mobile phone contains a number of different radios in it that enable it to talk to the mobile phone networks in different areas and countries, and also for Bluetooth, Wifi, etc. Testing at this level is sometimes called protocol stack testing, and means testing against various universally agreed specifications (an example would be 3GPP), using some pretty cool and pretty expensive pieces of equipment which simulate what the networks are doing. It can also be varied work, and can mean some fun getting out of the office doing drive testing. This is where you drive a specific route, where the network signal is known to change, and record and analyse the results, to see if the device you are testing is able to work in all sorts of different signal environments. And, based upon my experience, you also get to spend a lot of time eating out at service stations.
  • Applications testing – this is probably the easiest area to get into if you have a background in software testing in another field. Mobile phone application usage has exploded in recent years, you can get an application for almost everything now, and we all expect those applications to work well. Applications can be embedded in the phones firmware, meaning they are available when you buy the phone, or downloaded from app stores. This is a great area of software testing to get involved in so let’s look further at this area.

Testing Applications

Firstly, what’s different about testing mobile applications when compared with working in the desktop world? On the surface not a lot.Your software testing skills will treat you well here. Remember those equivalence partitioning and boundary value analysis techniques you learned onthat ISTQB course you did? They’re just as useful here as if you were testing a desktop based application. But some techniques are different, or just require a bit more focus.

Probably the most important thing to take into account when testing a mobile is that the software that you test could be in the hands of millions of people once you have finished. Releasing a phone with bugs in it, which mean customers need to return the phone, loses a company a huge amount of money. The mobile world is a low margin, high volume game. There are few players and brand is important. The mobile applications world is also very competitive, and customers can give feedback straight away, into the applications stores, for all to see. Good quality is extremely important.

For mobile applications a lot of testing is performed using emulators or remote device access services. Remote device access services enable you to connect over the internet to real devices which are provided by other companies for your testing use. However, final testing should always be performed on the devices that the customer will either buy in the shops, or download the application onto. There’s too much going on below the application level to just use the emulator. So even if you use emulators a lot, you should still get to play with the cool kit sometimes.

Interactions Are Important

When mobile phones started they were pretty simple. Calls, texts, and maybe a few games and not much else were about all your phone could do. These sorts of phones have not gone away, but even these are more complicated than they look, and smartphones are even more so.

Look under that UI, and under that client, there’s a whole software stack running.While your application runs, the phone is still talking to the network, working out where you are, maybe receiving calls, texts, Facebook and Twitter updates. There are a lot of interactions going on that you don’t see.

Understanding how the OS of the device you are testing (iOS, Android, Symbian, Windows Phone 7, etc.) works is important. You don’t need to be an expert but you do need to be able to design tests that focus on these interactions. What happens when you have your application running, and then get a call, and a text, whilst uploading video to YouTube? Test for it and find out.

Performance Is Limited

Mobile devices have limited memory and limited processing capacity, and so it’s important to test how the devices work when stressed. Fill up the phonebook, the photo gallery, etc. Then see how well the device performs under pressure. If you are testing an embedded application, maybe a ‘feature phone’ rather than a smartphone, then test for memory leaks and general performance of the operating system and applications. There’s not much memory on a cheap phone and the processors are very slow.
It’s also important to think about battery life. How power hungry is your application, does the battery run down quicker with it running, what happens to it when the battery runs out? Good battery life is very important to users, we’ve all been in the situation where you need to make that important call but the phone is dead.

Third Party Compliance

You may need to test for certification compliance; all phones are certified to certain standards. Without meeting these standards then it’s not possible to sell a particular device, or have a particular application accepted into an apps store.

These include:

  • 3rd party software runtimes – most, if not all devices are capable of running Java Mobile Edition applications and some others support runtimes such as QUALCOMM’s BREW platform. This requires that certification test sets are run and all tests pass.
  • Network operators have certain technical acceptance test sets to pass before they will start selling devices. Since most devices are sold by operators then passing these test sets is very important.
  • Most mobile applications stores have certain submission criteria; does the application meet these?

Usability Matters

You should test for usability. Usability means testing it on a panel of real users. First of course test it yourself, but then find a group of typical users and observe them interacting with the device or application. Typical areas to focus upon are:

  • How easy is it to use the application? Do users ‘just get it’ when they first start the application or pick the device up or do they need training? Difficult-to-use applications are often downloaded from app stores and deleted almost as quickly once the user finds they actually need to learn how to use them.
  • How well does the application fit the small screen? Is the text easy to read and is that button in the right place? Look and feel, especially on a small screen is very important.
  • Does the application or device feel fast? Do the users feel they are waiting too long for a screen to load or response to be received?

Usability is more important in the mobile world; the screen is smaller, it’s more difficult to interact with it and mobile devices are typically used one handed. The difference between a device or application which has good usability and one that does not is very obvious. It’s what’s led to the huge success of the iPhone which has excellent usability.

Automation Is More Complicated Than In the Desktop World

Automation is also a very important testing discipline to consider. Using automation for regression testing for mobile applications and devices is a very cost-effective strategy. A mobile application or device regression testing suite can be huge, with many interactions between other applications and areas to consider.

Automation in the mobile world is not as easy as it is in the desktop world. Your device is, after all, not able to run a large automation tool on its own. There’s not the memory to install it. The UI of typical mobile devices is highly dynamic; the user can use swipes and other gestures on the screen to interact with the device. This means that automation solutions typically run using emulators, or are bespoke systems running on the devices themselves. But the good news is that there are more mobile specific automation tools beginning to become available. A lot of automation testing for mobile phones takes place below the UI level as well, making it easier to drive the tool from the desktop, and then communicate with the device via USB or Bluetooth.

So Should I Get Involved?

So, is mobile phone testing just like testing on the desktop but a smaller screen? Do you just need your existing skills and maybe some glasses? Yes and no. Techniques you already know are equally useful in the mobile world. But how you apply them, and what you do in addition to them, is more important.

Working in the mobile applications and mobile phone testing world can be really varied work. If you work for a manufacturer or network operator then you get to test using a lot of new, prototype kit, especially when testing the devices themselves. Maybe you always get to have the latest phone. The work is fast paced, and you can quickly see the results of what you do, either on the high street or in the apps store.

As someone once said to me “Be proud, your work will soon be in the hands of millions”. It’s very satisfying to see someone in the street using a phone that you worked on, and knowing you played a part in making sure the software was high quality. It’s the quality of the software being released that decides if those millions love their new phone, love their new application, or if they return the phone or delete the application. No pressure on us testers, right?

Author Profile – Stephen Janaway

Stephen Janaway

Stephen Janaway

Stephen Janaway has been working in software testing for over 12 years, focusing on the mobile device and applications area. He’s worked for companies such as Ericsson, Motorola and Nokia, in software testing and test management roles, and is currently heading a software test and quality assurance group in Nokia UK, focusing on mass market devices.
Stephen blogs on his favourite subjects (software testing and Agile techniques, particularly Kanban) at and you can follow him on twitter @onegreenhill.

You might be interested in... The Dojo

Dojo Adverts-05

Tags: , ,

2 Responses to “Mobile Testing – That’s Just A Smaller Screen, Right?”

  1. JimNovember 18, 2011 at 1:40 am #

    The FUTURE of web design is going to be mobile. Having a responsive website design is critical. I take all my mobile designs to task with remote Usability Testing (such as to see what real website users really think while they use my mobile websites. Even when i think I got it right I am always amazed at what I find watching others use my designs.

  2. Viresh DoshiDecember 1, 2011 at 3:58 pm #

    We have lots of fun testing smartphone applications over desktop applications. The complexity of test environments, emulators and integration of systems is exciting and challenging and keeps the test team motivated. Usability testing, agile, continuous Integration play major roles in the smartphone testing arena. It’s easy to provide negative feedback on mobile apps so you need to nail the testing from the outset.