By Doron Bar
While Agile has helped us focus on the correct subjects, made us more efficient and helped us gain more vertical knowledge, it is sometimes misinterpreted or misused. While the Agile Manifesto prefers "responding to change over following a plan," it doesn't mean in any sense that it opposes strategy and planning.
The idea of Agile is to use tools as long as they work for us. As long as they help us deliver working software, and not vice versa, e.g. creating and maintaining measurements that are not actionable, then we should continue to use them.
On the other hand, I happened to see in some places that testing in agile has become much more ad-hoc testing than something planned, on all levels. I want to suggest a method to be used throughout testing, from the strategy, planning and executing, while still maintaining an Agile approach.
Creating A Test Strategy
"Tactics without strategy is the noise before defeat." - Sun Tzu
A test strategy is the first step in the testing process, and it is the higher level of the test process.
Let's say you are responsible for the testing of a new feature on Amazon. A chat module with the possibility of other people looking at the same product as you. You have already read the requirements, and understand the purpose of this feature. Now you are in a meeting with the stakeholders, and they want to know how will you approach testing the feature.
Naturally, they don't expect you to state each test, we are only at the beginning of the project. They also don't want a full plan, this deserves a meeting of its own. What they do expect is to hear ideas, principles, techniques, and motivations that will guide testing.
Why is a tests strategy essential:
- Test strategy enables clarity and organized processes.
- Easy to explain to interfaces.
- Align the team.
- Focus your effort.
- Identify risks.
- An opportunity to receive reviews.
A test strategy should be relatively short and kept to a topics level without going onto planning details.
For example, prioritizing testing types in accordance with system and SLA requirements. Due to needing an SLA uptime of 99.9999%, we'll need to make performance testing a priority after establishing the system is stable enough to test.
The strategy itself doesn't necessarily require long documents. It can be a short paper or a mind map. Complex subject matter can be difficult to describe in a few words. To succeed in that we must understand that it has a lot to do with the audience that we are facing. You can get some useful ideas from the following article, The Best Way to Explain Complex Concepts.
Establish A Test Strategy
The concept of a test strategy can sometimes be a little vague. To help us create a structured strategy we can ask the right questions, the ones that by answering them, we will gain a comprehensive understanding of the different building blocks of which the test plan will be developed.
We can start by asking six ‘w’ questions and two 'h' ones (originally this method is called 5w2h). I assume that if you answer those questions your approach to the test plan will cover most topics at a high level.
The Six “W’s” and Two “H’s”
- Why? - Describe the goal of the project.
- Why not? - Describe the risks.
- What? - Describe the scope of the tests, including what we will not test.
- Where? - Specify the test environment and other relevant data.
- When? - When will you conduct the testing?
- Who? - The testers, significant interfaces.
- How? - Specify the testing methods and process.
- How much? - Is there any particular cost as a prerequisite?
This tool is more of a heuristic that can help establish ideas and techniques. It won’t fit every instance, but having a tool like this as a prompt can help begin the process of creating a test strategy.
Below is a mind map created around the Amazon chat feature example mentioned earlier. The major advantage is that you can see in one glance the full picture, at a high level. (If you would like more information about how to make mind maps, or if these the first time you’ve seen one, this could be a useful resource or this software testing focused article.)
Notice it answers the Six Ws and 2Hs questions in the above heuristic.
You don’t have to use all the questions to create a strategy, however, I wanted to demonstrate what all the questions would like with this example. The link between the “what” and the “how” is just to highlight that there can be a connection between some of the topics, e.g. the scope of testing and the testing types.
This approach to a test strategy can be used with large and small projects. It might not make much sense with a smaller project, but with a visual method like the one above, even a smaller project could reveal itself to have more areas which need to be covered than previously thought. Visualizing work can allow everyone a better understanding of what testing is a priority and what could be nice-to-have.
Now, taking the above Amazon example, you can give a focused and comprehensive (though at high-level) report to your peers about how the team is going to approach new feature testing.
After we have a crystallized idea about your test strategy, we can continue to the next more detailed level of the test plan. While it still needs your effort, you might find that after you have your strategy, the test plan creation will be more focused and efficient.
Creating A Test Plan
“Alice: Would you tell me, please, which way I ought to go from here?
The Cheshire Cat: That depends a good deal on where you want to get to.
Alice: I don't much care where.
The Cheshire Cat: Then it doesn't much matter which way you go."
- Lewis Carroll
Like other terms in our profession, the definition of a test plan (or System Test Plan, STP) has different meanings, depending on the company you work at or other test engineers you meet. One definition of a test plan, sometimes backed up by test management tools, refers only to the actual test description (test cases). In this article, when I write about test plan I refer to wider scope. Test plan includes the topics we will test (without the actual steps or charters), and it also includes the time frame, actual labs, the team and stakeholders, risk management, DoD etc. Unlike the test strategy, there is plenty of material about this subject on the web.
On the other hand you might ask, what is the difference between test plan and test strategy, and doesn’t the former include the latter?
If you plan to build a house, you will probably want to have a strategy that will include the architectural style, size, principles like being eco-friendly. When you have these in mind, you can start planning how the house will actually look. Without a strategy the house will consist of parts that do not merge with each other, it will be uncomfortable to move through it and generally will not look as you wanted at first.
The same with test strategy: you need to you establish the motivation for testing - is it a new feature? New infrastructure? What will the customer and the company gain from these changes? The test strategy could look very different if the new feature is open to 100 users or 1,000K.
The strategy includes the ideas that will guide you later on in the planning up to the testing.
If you start planning without strategy, you might test parts you didn’t need and avoid tests that might be crucial.
Let’s again take the strategy of the Amazon chat feature example mentioned earlier. While planning you’ll start breaking down the functionality into the needed tests. Since we established in the strategy that this is a new feature with new technology that might be a risk, we understand that the tests must be thorough (but the regression can be reduced since the new code has less influence on the old one). We also understand that besides covering the requirements, we will understand the needs of the customer and add tests that will be more realistic with emphasis on being user-friendly. We will also test aspects of general questions - will this engage the customer more? Will the customer gain from this feature?
We plan the performance tests after we have already established the SLA in the strategy. Now we need to plan which tools we will use to achieve that, will you do different tests for the front-end and back-end or will you load the front-end hard enough to test the back-end at the same time? What is the appropriate tool/s to achieve the goals we have?
Every test we add to the plan must be driven from the strategy (of course you can change the strategy during the process, but you have to have good reasons for that). The strategy will be the light that guides us through the process.
Test plan as driven from the strategy example:
Some of the issues can be saved in the test plan and don’t need to be developed any further, for example, the risks. Others need to be more detailed.
Execution - Test Charters
At this point, we have figured out, amongst other things, the approach and scope of the testing, we have one last mission for this planning stage - develop the content of the tests themselves. This is true not only for the “scripted testing”, in which you write the actual steps and expected results per test case, but also to Exploratory Testing (ET). Here too, I find people still confuse free testing (just work on the system as you please) and structured ET, which uses charters. Charters are not as detailed as a scripted test but include the scope and means for the test, and they need to be ready before the test begins (3).
Those charters or test cases are driven from the previous level - the test plan. In the test plan we stated the test topics, here we break down those topics into actionable tests.
The functional tests below which came from the How in the test plan:
Another good idea is to add the results to the charters in the mind map.
Agile Is Not An Excuse For Chaos
Working in agile mode is not working fast, it is working smart and efficiently. It means using the minimum needed planning, documents, and tools. Sometimes “Agile” is used as an excuse for chaos, which is not the intent. As the examples show, procedures can be kept to a minimum, and yet can be comprehensive and systematic in their method. Maintaining a simple process for a test strategy, test plan, and test cases can help you have a clear view of testing activities for any feature or project you might encounter.
- Slim Down Your Test Plan Documentation - László Szegedi
- Test Plan - Software Testing Fundamentals
- Explore It!: Reduce Risk and Increase Confidence with Exploratory Testing - Elisabeth Hendrickson
- The One Page Test Plan - Claire Reckless
- Lessons Learned in Software Testing: A Context-Driven Approach - Cem Kaner, James Marcus Bach and Bret Pettichord
- The 10 Minute Test Plan - James Whittaker
- What Should A Test Plan Contain? - Michael Bolton
Doron is bug fighter since 1998, doing what he loves: hands-on testing on the one hand and managing on the other. Doron is an expert in methodologies and improving the test processes.