Getting Important Quality-Related Information to the Right People
By David Greenlees
In the past when people have asked me what I do, and I have responded with “I’m a software tester, I test software”, their follow-up question of “What does that mean?” hasn’t always been so easy to answer. For a long time, I struggled to accurately define what software testing is in words that others would generally understand and that I would feel proud of.
I initially developed this model as a way for me to visualise the answer to what software testing (referred to as ‘testing’ herein) is. This not only has helped me explain it better, but has also allowed it to grow from a model to help with an answer to a question, into more of an approach model. It feels like only yesterday that I first wrote about it on my personal blog. Perhaps because it’s something that is in the back of my mind all the time as I work. However, it’s evolved since 2016, and therefore needs to be let loose on the world once more. In this article, I’m going to explore the subject in a little more depth and introduce the evolution that has taken place in my mind.
Building A Quality Relationship
It’s important to understand that quality is a relationship between the user and the product, it’s not the product itself, nor an element of the product. The subjective nature of quality makes it hard to understand, and even harder to assess. For this reason, just like usability, we need to create boundaries and use guidelines and heuristics which can generally help us to meet the user’s desires. Note – Desires, not expectations; as a user, I can expect a product to be horrible but I would rarely desire it to be.
As mentioned previously, I first developed this model to help me talk about and explain testing. That need is still as valid today as it was when I first wrote about it, but it can also serve as a model which drives your approach to quality and testing, in a broad sense of course.
What is the goal of testing? For me, the goal of testing is to identify information; and to extend that, to identify quality related information. In that extension, I’d like to focus on the word quality.
I use Gerald Weinberg’s definition of quality:
“Quality is value to some person”.
I like it not only because I believe it, but also because it’s broad and begs to be further defined. To this point, it is then perfectly complemented by James Bach & Michael Bolton’s extension:
“Quality is value to some person who matters”.
I then further extend this:
“Quality is value to some person who matters at some point in time”.
This further highlights the subjective nature of quality and the need to constantly revisit it.
I recall seeing a tweet from Damian Synadinos which instantly struck a chord with me. It was too long ago to quote the exact tweet, however, in essence, it explained that a child’s security blanket is something which holds immense quality to the child until they grow up, at which point they may just think it’s a tattered rag and holds no value at all. After thinking about this example more, you could then say that once the child is done with the blanket, it may now be the ultimate quality product for the parent; thus creating a new quality relationship that will need to be factored in. After all, the sentimental value is one of the most important forms of value.
So, we use testing to help us identify information about a product and yes, testing can provide a whole bunch of information, but is that information valuable to anyone? Does it directly relate to the quality of the product? Maybe, maybe not. You can spend a lot of time and money gathering information that no-one cares about. This not only hurts project budgets and timelines, it also does a disservice to our craft which further propels the naysayers and their “testing is a waste of time and money” sentiments.
In order to provide quality-related information via testing, we need to know what quality means in the context of the product being tested, and what risks there may be to that quality.
Let’s explore the Quality and Testing Information Model together, and hopefully, it will provide you with some valuable information which can assist your future testing endeavours, and the conversations that will inevitably be required in order to compliment your efforts.
The Information Model
What the model looks like in its most simplified form…
For the purpose of this article let’s use the following diagram as a representation of all the information you could gather about the product you’re testing (it may not actually be finite but I don’t want to make the whole page blue!). The explosions represent the particular information that is valuable for you and stakeholders to be aware of and use at this particular point in time:
Seeking the Right Information
So in a sea of blue, the difficulty comes in how you pinpoint the explosions of valuable information. More importantly, how you know what valuable information is in the first place, and what makes it so valuable. To get the answers you need to do what testers do best; ask questions.
For this exact purpose, I have three questions that I like to ask in order to get the process started. The first two:
- Who needs information?
- e.g. Product Owner, C-Level/Exec, Devs, Ops, Customers, Business Users, Auditors/Regulators, Project Manager, etc.
- What information do they need?
- e.g. Risk related, bugs, information to help make decisions (but what decisions?), evidence, data/calculations, answers, performance metrics, security holes, etc.
The list of responses to each of the above questions could go on and on and will vary widely from context to context. It’s also important to understand that what was once valid in a previous project may be completely invalid in your current project, so these questions need to be asked – generally, and in a way that suits – often. Not only that, but within the same project, things can change quickly so these questions need to stay at the forefront of your approach throughout.
With answers to the first two questions you can then ask the third:
- How do we get that information efficiently and accurately?
- e.g. Strategy, plan, missions, charters, explore, question, automate, tools, etc.
Repeating The Cycle Of Seeking Information
As the responses can be many and varied and change considerably over time, please remember to remain flexible in your approach and ask yourself often if there are better ways to identify what you’re seeking.
Please note that this process of asking questions is cyclic, not linear. You don’t stop after one pass.
Using Tools To Seek Information
What are some of the tools you can use to help you work out how the required information will be identified? In my experience, this is where my test strategy comes in, and I constantly find myself referring to James Bach’s Heuristic Test Strategy Model (HTSM) to help me create it. On all of my previous projects, this has been the perfect companion for idea generation and risk identification. It also prompts many questions that can be asked of the project and its stakeholders which can help you identify what’s important and to whom.
The Product Elements combined with the Quality Characteristics are a great way to get you and the rest of the team thinking about what’s important and where the risks to the product may be lurking. Armed with a general consensus on this, you’ll have a fantastic starting point for your testing; and as you begin to identify the information you’ll likely need to revisit and change directions according to what you find and what people learn from it – Remember, cyclic not linear.
Another tool I reference quite often is a post by Michael Bolton, Context-Free Questions for Testing. A wonderful list of questions you can tailor to your context. The answers to these sorts of questions can be ultimately revealing and help drive your approach. Often, the answers and the resulting collaboration will also help many other members of your team perform the various activities they need to undertake.
Once you have a better understanding of what quality currently means, what information you need to begin with, and who needs it; you’re ready to start identifying it.
Your starting point will be guided by the type of information you’re seeking, as planned in your strategy and approach. This will help you focus on the most important elements of the product first. If the goal of testing is to identify quality-related information, then for me a good definition of testing is Cem Kaner’s:
“Testing is an empirical technical investigation done to provide stakeholders with information about the quality of a product or a service.”
When I first saw this I recall having to double check why ‘empirical’ was included in the definition and once you familiarise yourself with the meaning it makes total sense.
Empirical – Based on, concerned with, or verifiable by observation or experience rather than theory or pure logic.
It’s the human element; the tacit rather than explicit knowledge that’s required while performing good testing. So what can you do, and what can you use to help you perform good testing and identify the information that is valuable?
Firstly, have a mission. There is no doubt you can find all sorts of weird and wonderful things wildly interacting with a product, but remember that we’re trying to find those explosions of valuable information in the sea of blue. A mission will help you focus on a particular element of the product and it could be broad or detailed depending on what works best for you and your team. I won’t derail this post by talking about the dangers of procedural step-by-step test cases, but be cautious as these can blind a tester to everything else that is happening in the product as they are too focused on following a ‘script’. See Test Cases are not Testing by James Bach and Aaron Hodder for a great read on this and other dangers.
An example of a simple mission can be seen in a testing session report that I wrote a few years back after testing Google.com.au. I simply wanted to practice mind-mapping in the context of capturing notes while testing, and you can see in the report that the mission changed as I tested. Had I not entered with a mission I wonder what may have happened, and more importantly what I may have missed by not capturing information.
James Bach’s and Michael Bolton’s Rapid Software Testing methodology describes an oracle as “a means by which we recognize a problem that we encounter during testing.” One of the greatest tools a tester can have is a list of oracles to help them identify potential problems and risks (i.e. valuable information) while testing. One list I use on almost every project is a mnemonic by James Bach and Michael Bolton (with inclusions by others overtime) known as FEW HICCUPPS:
- Familiarity – We expect the system to be inconsistent with patterns of familiar problems.
- Explainability – We expect a system to be understandable to the degree that we can articulately explain its behaviour to ourselves and others.
- World – We expect the product to be consistent with things that we know about or can observe in the world.
- History – We expect the present version of the system to be consistent with past versions of it.
- Image – We expect the system to be consistent with an image that the organization wants to project, with its brand, or with its reputation.
- Comparable Products – We expect the system to be consistent with systems that are in some way comparable. This includes other products in the same product line; competitive products, services, or systems; or products that are not in the same category but which process the same data; or alternative processes or algorithms.
- Claims – We expect the system to be consistent with things important people say about it, whether in writing (references specifications, design documents, manuals, whiteboard sketches…) or in conversation (meetings, public announcements, lunchroom conversations…).
- Users’ Desires – We believe that the system should be consistent with ideas about what reasonable users might want.
- Product – We expect each element of the system (or product) to be consistent with comparable elements in the same system.
- Purpose – We expect the system to be consistent with the explicit and implicit uses to which people might put it.
- Statutes – We expect a system to be consistent with laws or regulations that are relevant to the product or its use.
A more general oracle, and one that really can fit within each of the above, is feelings. As a tester, it's important to be aware of your feelings while testing. Are you getting frustrated, angry, confused, etc? Each of these feelings could be an indication of a potential problem and are easily missed if you don’t recognise their importance.
A heuristic is an approach or method that aids you in making a decision or finding a satisfactory solution to a particular problem. For this reason, they are also wonderful tools for idea generation. A particularly good presentation on this subject is Karen Johnson’s Software Testing Heuristics and Mnemonics.
The trick with heuristics in testing is being aware of them so that you can consciously employ them when needed. You will use them all the time without knowing, but when you’re truly stuck on a particular problem, having a heuristic toolkit to call on can be very helpful.
Imagine you came across a dog in the park, the same dog you saw on your walk last week. It was very friendly and was happy to receive a pat. So this week you go straight up to it to give it another pat without giving your safety much of a thought.
Now imagine you were in the same situation however you had no short term memory and therefore didn’t know if the dog was safe to pat or not. Because of your short term memory issue you keep notes on various things, so you check your notes to see if you recorded anything about the dog. Yes, you have a note that the dog is safe to pat. Therefore, by being aware of your particular problem – short term memory loss – and calling on a heuristic method – your notes instead of your memory – you enjoyed patting the dog again this week (extreme example only to aid understanding).
Please note though that heuristics are fallible. Just because it helped you in the past doesn’t mean it will this time. The dog may be in a bad mood and not want to interact this week. So when using heuristics in testing, don’t see them as infallible silver bullets.
Over the years I’ve tested several different web applications and the most notable difference between them has been their respective user bases. In one instance it was an online banking application with a very broad user base. In another, an internal application with a very small group of users. The usability heuristics I used while testing (Jakob Nielsen’s) the online banking application proved very useful and reliable for identifying important information, however, these were not so useful for the internal application. Yes, I still identified information in relation to usability, however, this was met with some frustration from many stakeholders as usability for such a small user base was not deemed very important (rightly or wrongly).
I’m not specifically referring to ‘testing tools’ or tools that are made specifically for software testing. Of course, those are great but don’t limit your thinking to these in the tools space. I’ve worked on many data migration projects in the past and my best friend every time was MS Excel.
Any tool that can take information about the product and help you identify potential problems will not only save you time but also help you find things that you may not have otherwise.
There are so many possibilities with so many context variables that I really only want to suggest searching for something like “tools to help testing <insert problem/product here>”. You’ll be surprised what you may find and how many others have faced similar problems who have shared their tooling solutions.
Not necessarily referring to being an advocate or a ‘voice’ for the user, although I do think you can and should do that if you’re passionate about it and there appears to be no one else doing so. What I’m referring to is being conscious of how your users may interact with the product and what context they may be in at the time. If you can, get a sample of the actual users involved. Use them as tools to help you identify valuable information – I mean ‘use’ and ‘tools’ in the nicest possible way of course. They can also be anchors for what is truly important. Within your development team, you may have determined one particular element of the product as being the most important, and one session with users could throw that straight out the window.
Personas are another great way to help you gain a more realistic user perspective. They too can fail you, but if the least they do is get you and your team talking about the users, it’s a win in my book.
Gather And Report Information
So it’s great that you’re now testing and identifying important information, but what do you do with that information? Thinking about how your audience may need to consume the information you’re identifying you’ll find there are almost endless possibilities. What you need to report drives the way you gather it – green ticks may be enough for some, or you may need to provide qualitative commentary for others.
You need to ask questions like:
- Who is the report (i.e information) for?
- For yourself, so you can reference it at a project meeting.
- For yourself, so you can include it in an experience report for a Peer Conference.
- To be distributed to the project team.
- To be distributed to various stakeholders far and wide.
- Does the intended audience understand testing?
- I mean really understanding testing, as you understand it.
- Does the intended audience have knowledge of the product you’re testing?
- Knowledge of how it’s intended to be used.
- Knowledge of the architecture, platform, etc.
- What does the intended audience want to see in the report?
- Red, yellow, and green indicators or emoticons.
- Ticks and crosses.
- Commentary (your testing story).
- Subjective opinions.
- Does the intended audience know what your testing mission is?
- Do they agree?
- Is there a consensus on what quality means and therefore what’s important?
The above really is the tip of the iceberg, but you’ll need to stop asking questions and begin reporting the information eventually.
Test reporting can be very painful, or it can be a breeze. You need to know the answers to some of the questions above before you start testing to help make it the latter. The type of evidence of testing you need to provide will greatly impact the way in which you gather the information before reporting it. How can you gather the information so that it translates in the easiest way for consumption? Some stakeholders will trust you’re doing what you say you’re doing, while others will want to see it – think screenshots, or data outputs, etc. I’ve found this to be of particular importance when going through an audit for a project. The level of detail required by the audit was high, and so establishing that possibility upfront saved a lot of time and ‘begging for forgiveness’ after the fact.
The goal of test reporting is to tell the testing story and to highlight important issues quickly. This is most often a combination of the testing approach & ideas, progress, and results:
Testing approach & Ideas – What has changed in this area since the testing was initially planned, or since last reported? What new testing ideas have been thought of and how will they change the approach?
Progress – How is testing progressing versus the testing mission? Any major blockers? Any areas looking good? Any issues that need escalation? Will the testing mission be met, or does that now need to change?
Results – Bug reporting. What has resulted from testing? Have important risks been mitigated? Do the decision-makers have enough information in order for them to make those decisions? Have the testing results-driven new testing approaches and ideas?
A Visual Reminder
Now is a good time for a visual reminder of the Quality and Testing Information Model, now updated:
I cannot emphasise enough the cyclic nature of this model. The only end to it is when you stop testing a product – the project ends, you won the lottery or any other valid reason you may not need to test anymore. You may work through each step of this model in a week, or in 1 minute. It depends on your context and how fast things move.
When reading an article like this it’s easy to think in a linear process, and that you would work through it once while testing a product. That’s not the case at all. My intention is to have you thinking about each step in the process constantly and working through it over and over again depending on what you discovered at each step. You need to be checking in often with your team and stakeholders to make sure the information your identifying is still important to someone who matters.
- Gerald Weinberg – Quality is value to some person
- James Bach & Michael Bolton – Quality is value to some person who matters
- Damian Synadinos – Child’s security blanket
- James Bach – HTSM
- Michael Bolton – Context-Free Questions for Testing
- Cem Kaner – Definition of testing
- James Bach & Aaron Hodder – Test Cases are not Testing
- James Bach – FEW HICCUPPS
- Lynn McKee – Mnemonics
- Karen Johnson – Software Testing Heuristics and Mnemonics
- Jakob Nielsen – 10 Usability Heuristics
- Nielsen Norman Group – Why Personas Fail
David has been testing software or managing it for almost 20 years. A community builder Down Under (Australia), he is passionate about product quality and the craft of software testing, and focuses heavily on usability and user experience. He has published several articles, and blogs semi-regularly. You can keep up with his adventures at http://martialtester.wordpress.com/. In 2012 David founded the Australian Workshop on Software Testing (http://ozwst.wordpress.com/), and in 2014 helped bring Let's Test to Australia by co-organising Let's Test Oz. He’s an author (https://leanpub.com/u/davidgreenlees) and a co-organiser of TestBash Australia. David can be followed on Twitter @DMGreenlees.