Cultural Patterns in Agile Testing

Stick around in Agile testing long enough, and you start to see lots of different patterns and cultures when it comes to testing, test automation, and integration of testers in the Whole Team. Over the course of the past years, I collected some cultural patterns on different teams regarding how they solve the problem of testing, and become more and more agile in their approach.

This is a half-serious collection of these patterns, highlighting some of the puzzles at each stage, and providing some hints about the downsides to tackling them. Although the article might indicate a natural transition, I have seen teams start directly at pattern 4, or completely skip pattern 3 on their journey from pattern 2 to pattern 4. So, don’t take this as a prescribed sequence.

Pattern 0 – Oblivious

Believe it or not, there are lots of companies outside in the world doing software development that are totally oblivious about the need to test for the results. In an oblivious to testing culture, programmers are producing code, and there are no tests run at all. Developers might even be totally oblivious to the fact that they are doing any programming at all.

For example when you create a spreadsheet for your expense accounting, you might use a formula to sum up the different columns. When filling in =SUM(A2:H2) you are probably not aware that you are writing code. Of course, you will test it by filling in different data, and check the results. But you won’t be doing any thorough testing on it.

Depending on the size of the program, the team might be perfectly fine with this approach. For example, if you created the program for yourself only, then this kind of testing might suffice. On the other hand delivering a multi-person large project in this manner will most likely end in misery when the product faces a real customer. Especially when the customer finds critical bugs right from the start. The project team will have a hard time changing the code to fix them in the long run. With the code base becoming worse and worse, and without the feedback loop that helps to improve the quality of the work, the team will spend less and less time on delivering new features, and more and more time on fixing problems from the past.

At this point, either the company becomes so busy with bug fixing that they run out of new innovative business, or a leader steps forward and asks for proper testing. This is when the first testers get hired and start their work.

Pattern 1 – Separation

When a team starts working in iterations, the most widespread pattern is to train all the programmers on Agile methodologies, and get them started on producing code iteratively. In my experience more often than not, testers are left out of this first training wave.

That means that programmers might sooner or later notice the pressure to produce code also comes with a price. Over time they become more and more afraid to change anything in the code base. Sometimes programmers will start with some automated tests that can help to prevent slowing down the team too much. The programmers can add value from iteration to iteration, but they start to slow down.

In pattern 1, testers are still working outside the iteration. There is a release every three months or so, and testers work traditionally to find all the bugs in these releases. Since the product is delivered to them in a rather larger batch size, they will also find bugs in a large batch size. Of course, the programmers who already have slowed down due to lack of early feedback become even slower, asking for longer release cycles.

What might seem counter-intuitive at this point is that longer release cycles will only increase the number of un-validated decisions from the programmers to the testers even further. The way out of this crisis lies in introducing iterative testing, and bringing critical feedback from testers closer to the programmers.

Pattern 2 – Pipelining

Pipelining happens when programmers work in iteration n+1, and testers test the product increment of iteration n. At this point feedback to the programmers is delayed by at least one iteration’s length. That means that the overall cycle time from new feature idea to working product is at least the length of two iterations. With two-week iterations, the overall cycle time becomes at least four weeks. With four-week iterations, the cycle time is at eight weeks at least.

The testers working one iteration behind the programmers still become swamped with un-validated decisions in the code base. They find enough bugs to fill half the iteration. That also means that the programmers will produce fewer new features. When a product owner at this point asks for more features to be delivered, of course, the quality of the programming will decrease even further bringing the whole team to a full stop.

The only way to solve this dilemma is to shorten the feedback cycle even further. At this point the team should explore options to work in the same iteration. If functionality cannot be delivered at all due to the burden of outstanding bugs, the team might decide to work together in the same iteration by either doubling the iteration length or getting the testers on board. Or by bringing the testers on board, but committing to just half the work – after fixing the most critical bugs, hopefully.

Pattern 3 – Mini-Waterfalls

In a mini-waterfall culture, the iteration consists of a waterfall miniature. That usually means that programmers and testers work in complete isolation. There is a little business analysis and design happening at the beginning of the iteration, followed-up by some coding, and in the final few days of the iteration all the testing takes place.

Of course, such an approach comes with similar downsides as traditional waterfalls. The feedback might be shorter than on a nine months time frame, but still doing all the testing at the end of a two-week iteration comes with the risk, that there is too little time to fix those critical showstoppers right before the demo or review. This also has the downside that the team might be lucky enough to deliver the right results when no bugs have been introduced, or that the results from this iteration are completely useless.

The pressure is slightly higher to come up with good automated tests. Programmers are most likely doing some amount of unit testing, and some amount of integration testing with more than one class on the lowest level. Testers have probably adopted a risk-based approach at this point in time. But usually they are still working mostly manually, meaning that test effort increases from iteration to iteration. Although a risk-based approach to testing might help to reduce the scope of testing down to a manageable size, the left-out tests become more and more of a burden for the whole team. Fewer side effects will be found in the risk-based tests, leading to more and more bugs released to production.

When the team avoids the blaming culture that testers should have found these bugs, eventually the team gets a chance to deliver business value together as a whole team. This may be achieved by working closely together during the whole iteration.

Pattern 4 – Integrated

When testers and programmers are fully integrated, testers help the programmers coming up with good unit test ideas. With their special knowledge, they make the background of automated regression more thorough, and help the programmer understand their thinking. Programmers on the other hand help to automate some of the tedious tasks of the testers. They can help come up with tools to aid good exploratory testing, and they might even bring some relief to the repetitive tasks like end-to-end business-facing tests.

The team at this point might start to adopt practices like acceptance test-driven development, and session-based test management. The team works hand-in-hand together to solve not only their daily problems in the stand-up meeting, but also from iteration to iteration taking responsibility for the experiments that come out of the iteration retrospective. Testers and programmers understand where they excel, and where they can help each other out to deliver business value.

Eventually the testers will take a step forward, and become more and more of a first-class citizen to the team. They might start to ask the right questions at the right time – for example during a specification workshop where acceptance criteria for upcoming stories are discussed with the business.

At this point, the big problem of the IT organization – how to reliably deliver a working product continuously – might be considered solved. However, when you solve your problem number one, you promote problem number two. Problem number two at this point is the up and downstream processes to building the product – that is getting the right business ideas, quickly bringing a new product version to production, and keeping it in operation.

Pattern 5 – Continuous

In order to solve the up and downstream problems, the team should adopt certain practices. For the operations part, Continuous Deployment keeps the software in production, and the team can get early feedback from customers. When the team starts to collect data about product usage from production, this can create a feedback channel for new business ideas. Since the product can be rolled out instantly, the team can play around with hypotheses about the target customers, and how the product is used. Thereby they gain knowledge about whether the business ideas are on the right track, or whether the team needs to pivot their product to a different customer group or market.

Testers and programmers work closely together with the business stakeholders to generate new ideas about potential business value. They help to formulate and falsify different directions for the product. At this stage testers not only contribute with their critical mind-set to the overall product, but they also come up with tests to check whether the whole product is on the right track.

For the operation part, especially technical testers can contribute scripts to quickly release a new version of the product after it has been built-in the continuous integration environment. Build and deployment scripts are crucial to get the product to the customer quickly. Since the team has grown a safety net of automated tests, and backup procedures, this enables the team to quickly switch back to an older version when they find a bug slipped through in their deployment process.

So far, this is the last pattern that I have observed in teams all across the globe. I think once we have solved the up and downstream problems on a larger scale, we will find ourselves in the next frontier – and I am quite certain we will find a way to solve the number one problem then as well.

Which culture are you?

Maybe while reading the descriptions of the different cultural patterns, you already thought about where you are right now. Whether you are more part of the oblivious pattern, or continuously testing already, don’t consider the different numbers a natural progression. Although I have given the different patterns ascending numbers, this does not mean that you cannot start with a fully continuous testing effort right from the start. In fact, I hope our education systems will at some point become advanced enough for that generation of software developers – that includes programmers and testers – to consider continuous testing a medieval practice, since they already solved problems one to ten and are working on a solution for problems 11 to 20.

The discussion however hopefully provides you with some strategies to try out to solve your number one problem today, so that you can become a little more productive tomorrow. Of course, the biggest journey always has to start with the first step.

About The Author

Markus Gärtner works as a testing programmer, trainer, coach, and consultant with it-agile GmbH, Hamburg, Germany. Markus, author of ATDD by Example – A Practical Guide to Acceptance Test-Driven Development, a student of the work of Jerry Weinberg, founded the German Agile Testing and Exploratory workshop in 2011. He is a black-belt instructor in the Miagi-Do school of Software Testing and contributes to the Softwerkskammer, the Germany Software Craftsmanship movement. Markus regularly presents at Agile and testing conferences all over the globe, as well as dedicating himself to writing about testing, foremost in an Agile context. He maintains a personal blog at He teaches ATDD and context-driven testing to customers in the Agile world. He has taught ATDD to testers with a non-technical background, and he has test-infected programmers in several domains.

You might be interested in... The Dojo

Dojo Adverts-05

Tags: , , , ,