MindMap: Testing and Checking
The concept of Testing vs Checking was originally discussed and written about by Michael Bolton. It’s hard to believe it was back 2009. The topic seems to keep coming up in conversations, which is why I thought I’d attempt creating a MindMap on the topic.
I’ve called it ‘Testing AND Checking’, instead of ‘Testing Vs Checking’ as upon reading through blog posts and comments it felt right to me that checking is a part of testing and is not necessarily a separate discipline. As testers we all (probably) do testing and checking, but recognising when we are testing or checking is a useful exercise.
Perhaps others will think differently…let the conversation or debate continue!
MindMap is below, followed by a downloable ZIP file and a text list.
Enjoy!
Download - ZIP file includes PDF, png, text, Opml, Freemind and SimpleMind+ formats.
Testing and Checking
Checking
- Doesn’t mean the product works
- Help us know that the product works within expectations
- Can ‘pass’ but still have serious problems
- Automated tests are checks
- Need specifications
- Helps answer – does the system do what it is supposed to do?
- Great programmers do alot of checking
Is
- Confirmation
- Verification
- Validation
- Making sure the program doesn’t fail
- Verifying beliefs we expect to be true
Checks
Are machine decidable:
- Yes
- No
- True
- False
- Pass
- Fail
Testing
Is
- Exploration
- Learning
- Finding new information
- Discovery
- Investigation
- Evaluating
- Finding problems
- Searching for extents and limitations
- Driven by questions
- Finding problems not covered in checks
Tests
- Have open ended results
- Asks if there is a problem here?
- Can be about finding out if checks have been good enough
- Problems found from testing can be used to create checks to stop them appearing in the future
- Do not require specifications
- Involves some checking
The Debate
- Why do we need to define the difference?
- Does more terminology just cause more confusion?
- What problem is this distinction solving?
- Should it not be ‘versus’ it should be Testing AND Checking?
Useful Sources:
Websites we came across upon our research:
- http://www.developsense.com/blog/2009/08/testing-vs-checking/
- http://www.developsense.com/blog/2009/09/tests-vs-checks-motive-for/
- http://trishkhoo.com/2009/11/testing-vs-checking/
- http://girliemangalo.wordpress.com/2012/01/26/testing_versus_checking/
- http://www.stickyminds.com/sitewide.asp?ObjectId=16684&Function=edetail&ObjectType=COL
- http://jlottosen.wordpress.com/2012/04/02/testing-and-checking/
- http://www.satisfice.com/blog/archives/358
Why do we need to define the difference?
Because there is a difference to be defined, combined with a common confusion between the two.
Does more terminology just cause more confusion?
We make the terminology we need, otherwise we’d have no domain language at all. The terminology is to demarcate between the two so we can have a reasonable conversation about it and know what we’re talking about. It’s to raise our conciousness about the subject of exploratory, scripting and automation so when we mention them we carry with it a tacit understanding of what they mean.
What problem is this distinction solving?
The problem of replacing testing with limited checks and the problem of understanding the limitations of automation and scripts (and their power)
Should it not be ‘versus’ it should be Testing AND Checking?
Yes, if appropriate. Once people understand the difference between testing and checking they can make educated decisions about which to use, and when. I use both – automatic checking is fantastic for smoking potential testable problems quickly, and for checking the state of the database before and after important processes. But without knowing that “automated testing” very often equates to “automatic checking” how can testers, management and the business make good decisions about where to invest their time and money?
Very good mindmap! My only concern is the presentation of “absolutes” when concerned with testing. What you list here are some of the characteristics of -effective- testing.
For example, testing does not require specifications, but it does require a structured set of oracles to be effective. If the person performing the test does not have a thorough understanding of the oracles that apply in the context of the test, then it is not possible to report valid defects.
Another primary difference is that a check can be repeated with the expectation of getting the same result (under the same conditions). A test (e.g. based on a charter) may get different results based on a change in the tester’s inspiration, imagination, or understanding. That is why a tester may revisit a charter more than once during the course of a product development cycle based on new risks uncovered.
In a recent keynote at CAST 2012, Elisabeth Hendrickson revealed that it was actually she who first made this observation publicly (sort of). Just a bit of pedantry on my part, but a great talk: http://www.youtube.com/watch?v=hANmIC9JqUA&feature=g-user-u
Add: http://context-driven-testing.com/?p=69
The Insapience of Anti-Automationism by Cem Kaner