‘Quality’ No More?

Problems with the Q-word in the testing-domain.

I have a problem with ‘quality’ – not quality, but ‘quality’ – the word, especially in the test-specific domain. Why? It’s a word and concept that has so much priming for different people that (1) It means different things (as a concept) to different groupings; (2) There are so many preconceptions about the word quality that there is a bias – of different gradations – amongst people that think or talk about it.

In fact you almost have to think extra hard or be extra clear to even discuss the concept of quality. So, why do we even need to discuss it?

My personal opinion is that the ‘quality’ of something is relative to the expectations placed, attached or foisted upon it. In the testing arena we’re often dealing with them all the time – realistic, unrealistic and partially-formed expectations – both related to a product and the activity of testing.

Suppose I ponder over a problem (e.g. a requirement specification) and produce a whole range of relevant questions that catch potential issues in the design. Someone might think of that thinking/input as high/good quality or that the thinking/input met (or even exceeded) their expectations of me. Which is it? Does it matter? Is one form slightly less ambiguous than the other? I think so…

Objective vs Subjective Ratings

In some senses it’s a ranking or rating compared to some expectation. Are there any non-testing areas that try to rate quality?

Look at rankings for international football teams – if one team is ranked above another then they might be expected to be “better”. The football rating system tries to be objective – it just goes on the result, the relative strength of the opponent, the regional strength of the team, in competition or a friendly – but it’s still a model with limited usage. It aids seeding for competitions. It doesn’t tell you about the teams that are attractive to watch or distinguish those that grind out a 1-0 win (scoring a goal and then playing with 11 in defence) – it can’t that’s an emotional element, even if an independent spectator might find that to be useful information. Problematical?

Look at hotel (or restaurant) star ratings – they are judged against some criteria. This rating is intended to give a quality rating or stamp on the establishment – but this is mainly an individual judgement (someone inspects according to some guideline) – what if the inspector had a bad day – was coming down with a cold for instance, what if the hotel/restaurant had a bad day, what if they had an unusually good day? Problematical?

So, if someone talks about quality assurance, assistance or control do we have any common ground, “objectivity” even? Do we have “enough” common ground? I’d suggest that common ground varies from industry application to industry application, and varies a good deal even within a given industry application (maybe not the niche or highly controlled ones, e.g. some areas of defence research.)

So, talking about quality is problematical. I sometimes think I’m going to get caught by the QI (Quality Inquisition)!

A: Of course I know what quality is.
Me: Oh, I didn’t expect the Quality Inquisition.
QI #1: Nobody expects the quality inquisition! Our chief definition is value!
QI #2: … to some person.
QI #1: Yes, our 2-point definition of quality is (1) value – (2) to some person!
QI #2: … that matters and has the right authority.
QI #1: Yes, our 3-point definition of quality is (1) value – (2) to some person – (3) that matters and has the right authority!
QI #2: … in some situation or context.
QI #1: OK, our 4-point definition of quality is (1) value – (2) to some person – (3) that matters and has the right authority – (4) in some situation or context!
QI #2: … at some moment in time.
QI #1: Oh, I’ll come in again…

Me: Can’t we talk about the colours of testing instead? 🙂

OK, so I find quality usage problematical. There’s a lack of common ground. Does this mean we need some pseudo-objectivity. A standard perhaps? Noooo!

I like Jerry Weinberg’s form of the quality definition, “quality is value to some person” – this emphasises the inherently subjective nature of the word ‘quality’, and that highlights the problem with using ‘quality’ in any generalized sense.

To me it’s more important to understand the expectations of the customer/stakeholder. This is really saying the same thing without using the word “quality” – and that’s important as it’s the word ‘quality’ which immediately allows people to be primed (subconsciously) by the word and start thinking along a certain line, remembering a certain experience or living a certain bias.

Yes, the concept of quality (within testing) is problematical and thus I try to avoid it.

In a really unfortunate world some might equate the ‘quality’ of the testing to the ‘quality’ of the product. I’ve heard about activities securing quality – the trouble is that this sometimes comes from testers – the testers can possibly secure the quality of their testing and input into the product – believing anything else is just playing with expectations.

So, no common ground with the word quality – in the testing domain – and no desire for a standardized definition. Then how do we compare (or talk to each other about) our testing activities without using the ‘q’ word? Well, do we need to compare? If we want to talk about testing without using ‘quality’ I’d say it’s very easy, and potentially much more precise.

What to use then?

When asked about my testing, and if I really need to describe the detail, I say it produces a subjective assessment of the product, under certain conditions (some or all of which may be specified by the stakeholder) which may be environmental, time and third-party related. My testing story (reporting) will describe the findings (under said conditions) and also talk about the silent evidence of testing – what was excluded, not considered or not covered. This picture will be one piece of the puzzle in a ship/release decision for the stakeholder.

If someone asks about it being subjective I’ll explain to them how my modelling of the problem space means that I’m making an interpretation of the product and my way of testing it – as opposed to those coding the product and their modelling of the problem – if there’s an issue with that piece of terminology let’s just say “it produces an assessment …”.

All of this might fall somewhere into a project or organisation’s “quality plan”. (The quality plan, if one exists, doesn’t usually describe the test strategy or test plan, if those exist ;-).) But this is usually an easier way to talk about quality in a project or organisation – because there is usually a very fixed view of what quality is. Then I can talk about quality (which is not necessarily talking about testing) as I know what and who I’m dealing with, and I don’t mix in my usage of quality (if I need to use the word) with my testing.


So, maybe the distinction I’m looking for is the use of the word ‘quality’ within a testing-specific domain as opposed to use outside a testing-specific domain. It’s the use within the test-domain that can be problematical – outside the test-domain people have usually gone to the trouble of saying how they have defined their quality concepts, plans and processes (whether they do it consistently or not, or even to a good-enough level, is another matter.) This is fine (in a containment respect) and means I can generally avoid using the word “quality” within my testing-specific domain.

Not using the word is, I believe, adding value to my testing activity.

About Author

Simon Morley started testing by asking many questions of his parents at a young age. He has been a professional tester since 1992, thinks of himself as an emergent learner and divergent thinker and enjoys continuous learning from a wide variety of sources. Occasional speaker.

Blogs at http://testers-headache.blogspot.com/
Tweets at @YorkyAbroad

You might be interested in... The Dojo

Dojo Adverts-05

Tags: , ,