Shifting software testing left with operational definitions

What are operational definitions, and why should we use them? 

Refining requirements is an important part of building quality into your software. Tom Gilb wrote that “over 60 per cent of the bugs which will occur in your operational software will be there before the code is ever written” and that “we get more leverage if we go upstream and improve our design.” 

Operational definitions can be used as a tool to shift testing upstream to find gaps both in the process of defining requirements and in the requirements themselves. 

W. Edwards Deming, whose philosophy of continual quality improvement helped to rebuild Japan after World War II, wrote that “an operational definition puts communicable meaning into a concept." 

Nobel-prizewinning physicist Percy W. Bridgeman created operational definitions during his scientific research. While he was creating synthetic diamonds, his gauges kept breaking down under extreme pressure, and his work led him to examine how scientists define measurements. John Willis wrote that Bridgeman’s work inspired Walter Shewhart and Deming as well. 

Deming gave this example of an operational definition:

1. A specification test of a piece of metal or an assembly
2. A criterion (or criteria) for judgement
3. Decision: yes or no, the object or the material did or did not meet the criterion (or criteria)”

Another example of operational definition would be:

A specification, for example:

• As a Gmail user
• I want to log in to my Gmail account
• So that I can check my email

Criteria, for example:

• Acceptance criteria
• I can see emails sent to my Gmail address

A decision as to whether what is being tested meets the criteria, for example:

• The definition of "done" includes that all acceptance criteria will be tested

Can requirements such as user stories serve as operational definitions? 

The query “Do requirements constitute an operational definition?” can be used as a heuristic to help us understand the process of refining requirements. The Kindle dictionary says that a heuristic “enables a person to discover or learn something for themselves.” If the requirements are an operational definition, they must have the structure and clarity of an operational definition. 

No specification can ever be "complete" because some requirements will always have been overlooked or will emerge during development. Any process for defining work requirements should include the three components outlined in Deming’s example of an operational definition. This is because: 

1. Work needs to be specified
2. It needs criteria against which it can be tested
3. It needs to be tested

Use cases and user stories can be viewed as operational definitions if they are written and used properly. This is because they must have each of the three components of an operational definition. 

• A use case provides a way to capture the specification. The goal in the use case is a set of criteria. It is expected that the new feature will be tested against the goal. A use case has acceptance criteria, so it will be expected that the work will be tested against the acceptance criteria. 
• A user story contains a description that acts as a specification and has acceptance criteria, which are its criteria.

If a user story or use case is an operational definition, the feature being developed will be tested against the criteria. 

An example of a use case that is an operational definition:

A specification:

• As a Gmail user
• I want to delete an email in my inbox 
• So that I can remove unwanted emails

Criteria:

• Acceptance criteria
  • Click the select check box on an email
  • Click on the delete icon
  • The email is no longer in the inbox

A decision as to whether what is being tested meets the criteria:

• The definition of "done" includes that all acceptance criteria will be tested

An example of a use case that is not an operational definition would be:

A specification:

• As a Gmail user
• I want to delete an email in my inbox 
• So that I can remove unwanted emails

Criteria:

• Acceptance criteria
  • Click the select check box on an email
  • Click on the delete icon
  • The email is no longer in the inbox

The development team does not have a defined practice of testing acceptance criteria

If a user story or use case is not an operational definition, it has not been properly written or used. Checking whether a use case or user story is an operational definition gives a deeper understanding of the user story or use case.

Which techniques do NOT result in operational definitions? 

Not all techniques for describing the planned work can be seen as operational definitions. For example, sometimes testers are given Gherkin as a specification. The Cucumber Book says that Gherkin is a list of steps for a Cucumber test to work through, and so are the criteria used to test the work. 

However, testers also need the specification to understand the work that is to be tested. Gherkin is not an operational definition. This is not due to the “Given-When-Then“ syntax of Gherkin. Gherkin is not the complete specification from which tests are derived. Instead, it simply lays out the criteria that are used to run a test. An operational definition would include the specification.

This is why using Gherkin as a specification for testing is inadequate. Knowing that Gherkin is not an operational definition explains why using Gherkin as the basis for testing is awkward. This can bolster your case in asking for more information to support testing.

Tips for developing operational definitions

Operational definitions are particularly useful if you work with a team that defines requirements informally. This is because they provide a way of checking whether the feature being developed has a specification and criteria, and whether the feature will be tested against its criteria. If you are working with a team that has defined the requirements of its work informally, it can be challenging because it is not clear what the feature should do. The tester can ask their team to provide a specification and criteria so that the testing is more effective. If either the specification or the criteria are not provided, then mistakes will be made in testing. This will cost time for both the tester and the development team. 

A definition of the work to be done by a team needs to be clear. Operational definitions also help us shift testing upstream to identify where requirements are vague.

From time to time, planned work is described ambiguously. Operational definitions provide a guide as to whether the descriptions of the proposed work are unambiguous. Deming wrote that ”an operational definition of safe, sound, reliable or any other quality must be communicable, with the same meaning yesterday and today.” User stories, use cases, or any other method of describing proposed work can be unclear. A user story can be unclear, for example, if it starts with the phrase “as a user…”. It should be more specific and say which type of user. 

Operational definitions can be used to validate whether the description is clear. Deming wrote that an operational definition should have “the same meaning to the vendor as to purchaser, and the same meaning… to the .. worker.” If the description of the planned work is not clear to people in multiple departments, such as sales and engineering, it is not an operational definition. It will need to be revised to be made unambiguous. 

Sometimes, teams include adjectives in requirements for new features. The requirements may say that the new feature is “good,” “reliable,” “safe,” “better,” or “faster.” However, Deming wrote that these adjectives have “no communicable meaning until they are expressed in operational terms of sampling, test and criterion” as in an operational definition.

To wrap up

Testing should shift upstream because we can find issues before the code is written.  Operational definitions are a useful tool with which to test upstream because operational definitions provide clarity. Regardless of the method used to define requirements, a description of planned work should be an operational definition.

For more information 

• Finding software testing opportunities early with the Requirements Review Model, Hanisha Arora
• Testing against implicit requirements, Nicola Lindgren
• What's North of the North Pole?, John Willis
• Out of the Crisis, W. Edwards Deming (Chapter 9)