AMA Answer: How can we practice testability within context engineering?

Question by Rosie Sherry:  How can we practice testability within context engineering?

Response:

Fantastic question. Not many people talk about it (unfortunately).

Good news: We at MoT are now starting a discussion on this. It's a big area of work for QEs / test specialists/quality experts.

Most AI systems are hard to test because their context is invisible, uncontrolled, and constantly changing. I know of a few teams that version control/review/monitor their prompts or context, but that's still a minority considering the larger ecosystem.

Agentic AI can only scale if we make it testable enough. This means we have to create systems that can be:

AI is all about Context (explicit + implicit). 

Most AI systems fail because the implicitly supplied context is scattered in multiple places (memory, tools, history, skills, instructions, etc.). It lacks boundaries as well as ownership.

Here is how I see testability in context engineering (some ways to make it happen):
1. Make context sources known and explicit.
2. Store context in structured / defined formats (eg. markdown, yaml, json, etc.)
3. Version control context sources (wherever applicable). Treat it like code.
4. Break down the context into a hierarchy and see each level as a test dimension:

5.  Define strict rules for what context gets included (Eg, PDFs, JSONs, Text, Image) and what gets excluded.
6. Test the system that feeds AI.
7. Create contextual test cases.
Example:

GIVEN this "context."
WHEN the model responds
THEN it should:
- Follow Constraints
- Use "XYZ" Sources
- Ignore irrelevant Inputs.