How (Not) to Measure Quality

PO: “Hey folks, we need to add this new feature as fast as possible! It will bring that revenue boost we need.”

Devs: “But we need time to refactor the code!”

Testers: “And to test it properly!”

PO: “Hm… how much?”

Devs: “Uhm… 3 weeks?”

Testers: “Rather 4!”

“OK, I'll give you 2. See you then.”

Sounds familiar?

I had a lot of similar discussions in my career. I think they are a symptom of a deeper problem. Implementing a feature can be measured: it is there. Quality is much harder to measure. How much quality do we need anyway? What for? Which metrics truly tell us the quality and which are not?

Different stakeholders have very different qualities in mind. Product people and users care about outer quality, developers are concerned with inner quality aspects, and finally, managers care most about process quality. These three qualities need very different sets of metrics.

Another problem is that some commonly used metrics are only proxies for the information that we need. E.g. code coverage tells us which lines of code get executed during a test, but sometimes we confuse this with how well the code is tested.

In this talk, I share some general motivations for measuring quality. I review various commonly used metrics that claim to measure quality. Based on my experience, I rate them regarding how they may be helpful or harmful to achieve actual goals and which side effects are to be expected. I give some examples of how the weaknesses of one metric might be countered by another one to create a beneficial system.