Ever been writing automated tests with an AI code assistant, only to have it completely forget the framework you're using? Or start generating tests that don't match your page object patterns? Yeah, me too. Turns out we've all been using these AI tools wrong.
Anthropic just released a paper on "context engineering" that completely changed how I work with Claude for test automation. Here's what actually works.
The Real Problem
AI tools have a "context window"—basically short-term memory. When your conversation gets too long (discussing test frameworks, locator strategies, CI/CD integration), that memory fills up. Then the AI starts hallucinating and forgetting your automation patterns.
The fix isn't a better AI. It's using what we have smarter.
Fix #1: The Reset Button (Compaction)
Think of this like clearing your test data between test runs.
Compaction: summarizes your conversation, keeps the important stuff, tosses the rest. Fresh memory, but the AI still remembers your framework and patterns.
Just type the `compact` command when things get messy. I went from 92% memory used to 36% in one command.
Pro tip: Do this between major features—after setting up your framework, before adding new test suites, etc.
Fix #2: The claude.md File (Your Automation Standards Bible)
This one's a game-changer for test automation.
Create a `claude.md` file in your project with:
- Framework you're using (Playwright, Cypress, Selenium?)
- Your patterns (Page Object Model? screenplay pattern?)
- Locator strategies (data-testid preferred, no XPath)
- CI/CD setup and requirements
Now every new session, Claude already knows your automation standards. No more re-explaining "we use data-testid selectors" for the 50th time.
Fix #3: Let It Take Notes
Instead of making the AI remember everything, have it maintain files:
- test-progress.md – What's automated, what's left
- flaky-tests.md – Known flaky tests and why
- framework-decisions.md – Why you structured things certain ways
The AI can check these anytime without clogging its memory. It's like maintaining proper test documentation.
Tell Claude to update these after generating test code. Suddenly you have self-documenting automation coverage.
Fix #4: Multiple AI Agents (For Complex Frameworks)
This one's for the big automation projects.
Instead of one AI doing everything, have a team: one agent writes page objects, another creates test cases, another handles utilities and helpers. A "lead" agent coordinates them all.
I tried this setting up a new test framework. One prompt gave me proper page objects, reusable test data factories, AND CI/CD configuration—all properly structured.
Warning: This is slower. Only use it for comprehensive framework setup, not quick test additions.
Which One for Test Automation?
Quick guide:
- Long framework discussion? → Compaction
- Same automation project over weeks? → claude.md file
- Building entire test framework? → Multiple agents
- Daily test writing? → Note-taking files
Mix them. I keep a claude.md with our framework patterns, use note-taking for active sprints, and hit compact when context gets heavy.
Why Automation Engineers Should Care
Here's the thing: AI can massively speed up test automation, but only if it remembers your patterns and standards. Senior automation engineers know this—consistency is everything in test code.
These tricks make your AI remember your framework like a senior SDET, not a junior writing their first Selenium script.
Try It Tomorrow
Pick ONE of these for your next automation session:
- Create a simple claude.md with your framework and patterns
- Or just try the `compact` command when your chat gets long
You'll notice the difference immediately. The AI will actually maintain your page object structure instead of randomly mixing patterns or using terrible locators.
----------
The Paper: https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
