In this activity, we will explore ways to evaluate how observable your test automation environments are. Each of our working contexts is different, so it’s important to understand what are the key characteristics of an observable system and what are the strengths/limitations of your own underneath those. Usually, we think about observability as being product (and production) related, but it is also very important in other areas of the Software Development Life Cycle (SDLC), namely in test automation, which we will focus on in this exercise.

Instructions:

Consider the following key characteristics, as coined by Abby Bangser and Charity Majors:

The only way to ask new questions is to keep the original raw data available and queryable

• raw events
• no pre-aggregation

Empower creative and shared exploration based on business context

• oriented around the lifecycle of the request
• batched up context
• exploration over static dashboards

Make data easy to add details to and easy to query

• structured data
• arbitrarily wide events
• schema-less-ness
• high cardinality dimensions

1. Create a table like the one below for your test automation system:

2. Fill out what are the key strengths of your system for each key characteristic.

3. Focus on the limitations for each characteristic. What are the main factors that are keeping you from fulfilling those ideals

4. Review your Strengths and Limitations and think about what are the top 3 small experiments you may run in your organisation to potentially overcome the limitations and turn them into strengths. Write them down!

What you’ll learn

• Observability: A 3-Year Retrospective - Charity Majors - https://bit.ly/mot-ob3y 

• A journey to truly understanding Observability - Abby Bangser - https://bit.ly/mot-unob