Get Out and Test

The image of the software professional, be it developer, tester or otherwise, is traditionally that of someone in an office, probably sitting in a cubicle, staring at a screen for eight hours or more a day. I guess for many of us this is also the reality of our day, we have our space and our connection to the world in front of us, and most of our testing work is conducted via that connection. Log on to a server, download and install an app, start some automation in the cloud. Our surroundings do not change and before we know it, we’ve spent 8 hours sitting in the same chair, wondering where the day has gone.

I have some good news. If you are testing mobile, then it does not have to be like this. In fact, it should not be like this.

A mobile tester, yesterday.

A mobile tester, yesterday.

Knowing how to think like the user and how to act like the user is a very important skill for a software tester. We reduce the risk of unsuccessful product launches by giving our stakeholders early feedback on the product and to make informed decisions on what needs to be done in order to launch into the market with confidence. In mobile, where are the majority of our customers using their devices? Outside. In the fresh air. In the sunshine or in the rain.

Getting out of the office and testing the device or application in the same environment as a typical user is a critical mobile testing skill. An extremely common mistake that is made when testing for mobile is to test only in the office where the network conditions are perfect, the cellular coverage is high, the number of people using the connections is low and the light is good. It shouldn’t be like this – you need to get outside and see what the real world is like – but often it is merely not considered necessary, and sometimes management doesn’t trust testers enough to even allow it. If this is the case then hopefully what you learn by reading this article will help you convince them otherwise.

So, what should you be looking to consider testing when you are out and about? Here’s a handy list of areas to consider:

Poor Signal Quality

As you walk around, maybe near the office, watch the signal strength on the phone. The number of bars displayed shows how strong the connection with the nearest cell towers is. Try and see where it starts to dip and then try use cases which rely on the applications data connectivity. How resilient is the application to a less than perfect connection? What happens when the connection drops completely? Does the application just get stuck trying to load something or does it gracefully time out?

Handovers Between Speed Networks

Keep a look-out to see where your mobile device switches between the different speed networks. You’ll be able to see this by looking at the icon which represents the speed of network. In most of the world this will change between 2G and 3G and in some areas also 4G. Again, try using data connectivity parts of the application in these circumstances. How does the application work over these different speed networks, and what happens to it when the phone is switching between them? The connection will be momentarily dropping as it switches. Don’t forget to try the typical use case where you start on a cellular network, then switch to Wi-Fi, and back.

Slow and Overloaded Wi-Fi

I’m sure the Wi-Fi connection in your office is great, and pretty fast too. But what about the Wi-Fi connection in the cafe around the corner? It’s probably much worse, frequently overloaded and slow, and that’s exactly the sort of connection most people are using when relaxing over a coffee. It’s worth putting yourself in their shoes and doing the same.

Don’t forget to bear in mind that it’s most likely to not be particularly secure, so being at the cafe could also be ideal for some security testing on your application.

Get fast

Walking is OK to test how the application works when the data connectivity isn’t great but driving or being on a train is better. Mobile networks are split into different cells with each cell covering a particular area. Mobile devices enter and leave more of these cells if they are moving more quickly. Since there needs to be a handover between the different cells, then this affects connectivity to the network. The application that you are testing will be momentarily losing and re-gaining connectivity regularly, and the signal strength will be rising and falling. Use cases which rely on an active data connection will be affected. If you are driving, then just remember you shouldn’t be the one actually carrying out the testing.

Get Crowded

Mobile networks have a certain capacity and often exhibit different behaviours when busy. This can affect how they handle data connections, calls and messages. If you’ve ever tried to use a phone at a large concert for example, then you’ll know they are not always as reliable as usual. You could try and test how your application deals with the losses in connectivity and connection rejections in busy cells. Sure, maybe your boss won’t buy you tickets to Glastonbury just for testing purposes, but if you can find somewhere crowded then that helps.

Map It

Above all, once you start finding places near your office or home where you can reproduce the conditions above, then make sure that you record them on a map or favourite them in the mapping application in your phone. That way you can build up a route that you can take in the future, which will enable you to quickly go back and test any application under the same conditions. This is exactly what the device manufacturers and network operators do; they have set routes to follow when executing, for example, acceptance tests.

Try It

So, when testing on mobile, why not try and get out of the office? You will be testing under conditions that are closer to those of the user and you will be able to find a whole bunch of bugs that you wouldn’t find otherwise. Besides, in the northern hemisphere it’s about to be summer. You never know, the sun might even shine. It’s a great excuse to get out and put your new found software testing skill to good use.

About the Author

I’m Stephen Janaway.

I’ve been involved with software testing for a little over 12 years, starting as a Test Engineer and then working in various Test Management roles for a variety of companies. Currently I’m a Test Manager focusing on mobile, and test automation, with a sideline in DevOps. Previously I’ve worked for three of the major mobile device manufacturers, as well as advising a number of mobile application developers on test and release strategies.

I’m equally at home with a device in one hand and a spec in the other, getting my hands dirty, or managing projects and teams.

I blog at and you can follow me on Twitter @stephenjanaway.

You might be interested in... The Dojo

Dojo Adverts-05

Tags: , , ,

3 Responses to “Get Out and Test”

  1. PrithiverajJuly 3, 2013 at 12:08 pm #

    Kindly clarify my understanding, the factors you described should falls under performance test right. With having pre-condition as all set , we can focus on application funtctionality right. After this we can test these factors impacting performance of functionality.

  2. Jenny MulhollandJuly 10, 2013 at 8:16 am #

    Even if you can’t convince your manager to let you out of the office, you can still do some of this stuff – testing the 3G-wifi handover is a good one. Also I’ve found that metal boxes such as filing cabinets and lifts are good for creating poor signal conditions inside the office!

  3. MelissaJuly 10, 2013 at 4:47 pm #

    i couldn’t agree more as we do mobile testing and require our testers to test the applications while out and about plus you get some fresh air and stretch you legs :). i will definitely be adding the map it to our test strategy.