Creating a Shared Mobile Test Lab

Mobile Testing is complicated.

One of the big challenges is testing things on different versions and phone models.  Can you realistically see how your mobile website or app works on multiple phones and software versions?  When it is hard enough to get budget to do any kind of testing, how on earth are we supposed to justify splashing out on multiple devices in addition to spending time on them?

There are companies out there that allow you to access a wide range of phones remotely, but the reality is they are slow and often don’t truly reflect how a person would truly use them.

Creating a local mobile Test Lab is what the peeps at ClearLeft have done.  They have always maintained an open door policy, welcoming people into their office.   The limitation here is that the devices are meant for testing mobile websites only, not apps.  It would be great to see more local companies and coworking spaces pulling together to create some kind of shared resource.

There is also Mob4Hire.  I’ve not tried them out.  They are likely to have lots of people on their database, but how up to date is it? And the problem is that their mob are not necessarily professional testers.

One idea I’ve had is to add a section to member’s profiles on Software Testing Club where they can state which mobile device they currently own.  This would then mean that anyone could contact them and ask them if they would like to do some testing.  I had thought of maintain a separate database, but think the reality of it is it would be too difficult to maintain.

Do you have any ideas of how we could create an open and valuable shared Mobile Test Lab resource?

 

 

You might be interested in... 1 Day Python for Testers Course

MoT Courses-06

6 Responses to “Creating a Shared Mobile Test Lab”

  1. Rudolf OlahJune 11, 2012 at 4:50 pm #

    Isn’t it difficult, legally, to have a shared testing platform due to the sensitive nature of the data? When I’ve seen testing done for websites, it’s been done with a backup of the data from the live site. We only generated test data for specific cases to replicate a bug or to quickly prototype the website.

    The devices that are being tested on would need an automated suite of tests to run, maybe using Selenium?, and there needs to be a documented process for this so people can sign up as mobile testers and get up and running quickly. The easier it is to become a tester, the more likely people are to do it. There should be a way to dump all the specs of the device and to somehow certify or ensure that the device isn’t doing anything funky/weird :S like testing with a nightly release of Chrome or Firefox would be misleading.

    • Rosie SherryJune 11, 2012 at 9:15 pm #

      I guess the legal thing depends on the type of project you work on. There are lots of smaller scale projects out there that wouldn’t necessarily have a problem with data.

      Why would they need to have automation tests? I don’t think this is a necessity. If automation on a specific device is needed, then I suspect that the company would invest in the kit, or perhaps use a cloud solution.

  2. KristineJune 11, 2012 at 8:43 pm #

    Our App supports only Android devices and it is quite complicated. We choose to build our own Mobile Test Lab and additionally we created Beta User Network.

    If you are looking for external portal, than check uTest.com. They offer geo-located testers with specific device/network provider e.g. Germany Samsung 2 T-Mobile.

  3. Rosie SherryJune 11, 2012 at 9:18 pm #

    Hi Kristine…uTest do offer this service to a certain extent, but it’s not ‘open’ – ie you need to be a customer of theirs, which isn’t an option for everyone.

    I’d love to create something more open and accessible.

  4. Len AyeJuly 5, 2012 at 9:49 am #

    Regarding shared mobile testing, I’m not sure how it would work in reality. I don’t know about Andriod, but for iOS, Windows Phone and Sumbian at least, don’t you have to have a Jailbroken or Developer Unlocked phones in order to install and test apps which aren’t published through the app stores?

    • Rosie SherryJuly 5, 2012 at 12:07 pm #

      Hi Len,

      I did some testing a short while back on an iOS iPhone and iPad app, it is possible to install the apps without going through the app store. He had to do something his end and I had to provide my UDID (http://whatsmyudid.com/). I then installed it via itunes with the files he provided.

      There are also things like TestFlight, though I’ve not personally used it – https://testflightapp.com/