Synergetic Testing

“Oh, so you play casino games all day…”

I have a fantastic life.

It’s true… I have a great private life, and my work life is truly awesome!

I bet you don’t hear many testers say that.

I read so many horror stories  about the lives of other testers who suffer at the hands of seemingly ignorant employers, who pay little to no attention to their testing staff, that I figured it was high time I stood up and shouted that it doesn’t have to be that way…

In case you’re wondering, I’m a Brit, now very much established in  Sweden, and my official job title is Game Test Engineer, for the award-winning online casino games company NetEnt.

As I get more involved in the testing community I realise that there is a pecking order with testers. Years of experience carries most weight and kudos, followed by automation test engineers. If you happen to work in telecoms or banking then you can double your status. Even testers working on mobile apps enjoy a decent level of respect. At the bottom of the heap are game testers. And even below that are game testers in the online casino games industry.

When I attend seminars and test meet-ups, and that point in time always arrives when someone asks the all-important question “So who do you work for?” My reply seems to be greeted with confusion (not many people outside of the casino industry know who NetEnt is), and then when the realisation hits them I get that look that says “oh, you are one of those testers.”  I smile at their disdain and move on, knowing that it would take me too much time and effort to convince them that I am just as good a tester as they are.

However, the real problem comes when trying to recruit testers. I had one graduate tell me that they weren’t interested because they were looking for a more ‘serious’ testing job.


Well, this is my chance to even up the playing field and make it very clear what we casino game testers do, both in terms of risk of failure, and testing practices…

Generally speaking, most people think that stock exchanges and banks have the largest financial transactions of all (maybe that is why testers for such institutions feel part of the elite). But did you know that currently NetEnt has no less than 1.7 billion game rounds played each month? And that each month we have over 2.4 billion euros going through our systems? And those numbers are increasing all the time. Can you imagine the business impact of released games that were faulty and were paying out incorrectly? Can you therefore imagine the level of importance testing has in a company such as NetEnt?

We casino game testers take quality no less seriously than any other tester in any other industry. But, perhaps what is different for me and my colleagues is that within our company, we are very much valued and respected. We don’t sit at the end of the development chain, waiting for the ‘almost’ end product from the developers. We are in fact considered developers in our own right.

As a game tester in an agile environment, I’m in the development cycle right from the get-go. I don’t just see code, or graphics, or sound like my fellow developers in their skill areas. I see the whole, the big picture. I see the sum of their work, as the functionality blends with the artistry of the graphics and animations gurus, and the code enables game play. To my mind, I’m in a very privileged position – I’m a game test engineer who has the mandate to recommend design changes during development, because I am seeing the big picture. And more importantly my voice, and that of my testing colleagues, is heard. From personal experience I’ve been able to change the math of a game, have features removed to make gameplay simpler, name characters, create game names and so on. And it’s all because I have had my eye on one thing – quality. ‘NetEnt quality’ to be precise.

What’s NetEnt quality?

Some months ago, we game testers gathered together to thrash out a definition for ‘NetEnt quality’. What we came up with was this…

‘The definition of software quality is to achieve a measurable level of user satisfaction in terms of functionality and usability.’

I was working on a presentation recently, and I decided to take another look at this definition. I quickly realised that we had missed out one vital part of the jigsaw puzzle, perhaps because we were focused on the ‘software’ aspect of the definition rather than the ‘NetEnt’ aspect. The missing factor was of course Aesthetics.

Games are about entertainment, and in my opinion, in order to achieve the best possible game quality, you must have an equal balance in terms of functionality, usability and aesthetics, during development.

What do I mean by aesthetics? Well, if you spent all your resources on producing a game that functions perfectly, and to a degree is intuitive to use, but has terrible graphics and appalling sound, then players won’t play. Likewise, if you spend all your resources on producing stunning graphics and animations, but functionally it was so buggy that it was unplayable, again players won’t play.

Usability is an essential but often overlooked part of software development. I have a golden rule when it comes to thinking of usability for our games:

“As a player, don’t make me think!” If the game isn’t intuitive to play, or easy to learn, this will yet again put players off. I consider myself to be of average intelligence, but even I will refuse to use an app or game if I don’t find it intuitive to use within the first 30 seconds.

equilateral triangle of qualityThis equilateral triangle of quality is important to know from a test standpoint. Each of the three quality factors requires different testing methods. You wouldn’t use automated tests for usability testing for example. And much of the aesthetics is best tested manually. However functionality, or at least a fair amount of it would be best tested with automated tests.

Many papers have been written on agile testing practices, but I suspect that if you asked several testers, each working in different agile environments, you’ll find that each one is using their own hybrid testing methods to get the job done. Is any one method better than any other? No, of course not, because each development and test situation is unique. There is no right or wrong way. There is only the most efficient, effective and relatively painless way. And that will differ depending on which company you are in, and what development processes you are using. I’m not saying that you cannot point-blank use automated tests for aesthetics for example. I’m just saying that it might not be the most effective method to choose.

Likewise, manual testing is inferior to automation testing when it is being done inefficiently, and for the wrong reasons. A classic example is manually checking all the payout values for 10 coin values in 26 different currencies in a casino game. Manually it would take several man days – clearly a waste of resources. A script can do this in just a few minutes, and takes less than an hour to write and set up.

Being a test lead for successful games such as Starburst™, South Park™ and Aliens™, requires designing test strategies that ensure minimal impact on development time, and the continuation of the highest possible NetEnt Quality. Each of these games is unique, and were very complex to design and create. But it doesn’t mean that the testing strategy for any of them has to be complex. I use a mix of testing methods depending on what it is that has been developed – Implemented Code Verification, STB, Exploratory, Automation, Usability and Regression. And the golden rule is the test lead must know the game inside-out.

I kid you not… When you have been working on a game every day from day 1, you get to know the game so intimately that every subtle tweak in animations, every tiny change in performance and every sound enhancement becomes glaringly obvious even without checking the overnight code commits.

The key

It has to be said that testing efficiently is all about communication. Forming good working relationships with every member of the development team is essential, because if you know exactly what each developer is working on, you won’t end up testing something that has not yet been completed. Even in large teams it’s possible to create the bonds needed to be ‘on the pulse’ all of the time.

Frequently I find something strange in a game, and instead of writing a bug, I’ve simply had a chat with whichever developer is responsible (each code commit is signed so finding out who is responsible is easy). Much of the time they say “yeah, I messed with that code yesterday, and I’m working on it just now.” or “It’s not yet finished… it will be ready for test tomorrow.”

By focusing testing on what code has actually been committed and built, rather than what tests need to be carried out before development is complete, you immediately eliminate the creation of invalid bugs. Additionally, it reduces the stress levels in the team when nearing completion of development because there are way less bugs to wade through. Interestingly, one major positive outcome is a much higher game quality. How we measure that is quite simple – game success, and of course the number of production bugs.

I call our way of working synergetic testing because we are testing with the real-time flow of development rather than the expected flow, or at the end of development. In my view this is the most Agile way of testing that there is.

There is no ‘us’ and ‘them’ when it comes to development at NetEnt. We are all in it together – coders, technical writers, testers, graphics, animators, sound and server gurus – and we are all striving for the same goal – ‘Better Games’.

Yep… I do indeed have an awesome job!

About the author

Dawn Möller is currently a Game Test Engineer at NetEnt. With over 26 years experience in the IT industry, mainly in data communications, quality has been a constant but backseat companion. Over the past 5 years however she has focused solely on testing and the QA arena, challenging working practices and test methodology paradigms. Her new test practices are gaining popularity within NetEnt, and the last few game releases have shown major improvements in test work efficiency without compromising on quality.

Dawn currently tests both desktop and mobile games in an Agile development environment. As a Certified Usability Analyst, she sees usability as a key component of success in any product. She is recognised as a company Ambassador; representing the company at external functions, and is an active member of the NetEnt Academy; giving talks about quality. She is also the chairperson for the company’s Game Test Group.

You might be interested in... The Dojo

Dojo Adverts-05


3 Responses to “Synergetic Testing”

  1. Teri CharlesDecember 14, 2014 at 8:52 pm #

    I love this article! And I never understand any tester’s disdain for what other testers do and I have no patience for it. I love your positivity and enthusiasm for what you do. Your company and team is extremely lucky to have you!

  2. Alan KerrDecember 15, 2014 at 1:42 pm #

    Really enjoyed reading this article. So many good points made. Your enthusiasm for your company, and role is commendable.

  3. BayleyDecember 15, 2014 at 1:57 pm #

    I was with you right up until you started looking down on other testers. You don’t like it when people assume you’re not good at what you do, so why go on and on about how only you are considered developers, only you are valued and respected, only you see the bigger picture? Shouldn’t we ALL strive for those things? There are companies in many different industries that value testers, and it pains me to see anyone insisting that their industry is the only good industry.