Observability In Automated Testing

João Proença's profile
João Proença

Quality Engineer

Challenge Description

In this activity, we will explore ways to evaluate how observable your test automation environments are. Each of our working contexts is different, so it’s important to understand what are the key characteristics of an observable system and what are the strengths/limitations of your own underneath those. Usually, we think about observability as being product (and production) related, but it is also very important in other areas of the Software Development Life Cycle (SDLC), namely in test automation, which we will focus on in this exercise.


Consider the following key characteristics, as coined by Abby Bangser and Charity Majors:

The only way to ask new questions is to keep the original raw data available and queryable

  • raw events
  • no pre-aggregation

Empower creative and shared exploration based on business context

  • oriented around the lifecycle of the request
  • batched up context
  • exploration over static dashboards

Make data easy to add details to and easy to query

  • structured data
  • arbitrarily wide events
  • schema-less-ness
  • high cardinality dimensions

1. Create a table like the one below for your test automation system:

Table headings: observability characteristics, strengths and limitiations. Row entries: Keeping original raw data, Creative and shared exploration, Easy to add details and query data

2. Fill out what are the key strengths of your system for each key characteristic.

3. Focus on the limitations for each characteristic. What are the main factors that are keeping you from fulfilling those ideals

4. Review your Strengths and Limitations and think about what are the top 3 small experiments you may run in your organisation to potentially overcome the limitations and turn them into strengths. Write them down!

What you’ll learn
  • Understand that observability is important throughout the whole SDLC, not just in production.
  • Identify experiments with automation data results to uncover the impact it has in understanding test failures and your feedback loop


Explore MoT
Test Exchange
A skills and knowledge exchange to enhance your testing, QA and quality engineering life
MoT Advanced Certificate in Test Automation
Ascend to leadership roles by mastering strategic skills in automation strategy creation, planning and execution
This Week in Testing
Debrief the week in Testing via a community radio show hosted by Simon Tomes and members of the community