Don't Trust Claims. Test the Harness
A free event by MoT AI Chapter
An AI agent reviewing a refund request tells you, I have verified the manager’s approval, so we can go ahead with the refund. Should you trust that?
Most conversations about agent safety focus on permission: can the agent act, or is it blocked from doing so? That matters, but it is a problem we have been dealing with in software for years through authorization checks. The harder problem is different. An agent can claim that a required condition was verified when there is actually nothing confirming that it was. The system does not necessarily get bypassed. Instead, it trusts what the agent says and treats that claim as if it were evidence.
This session treats the code around the model, the harness, as the thing under test. I will run a small controlled experiment using the same model, prompt, and tools, with an approval record still marked pending rather than approved or rejected. One harness will rely on the agent’s claim, while the other will independently check the actual approval record before allowing the action. Only the harness changes.
The goal is not to prove that one harness is smarter. It is to show what the system catches when it requires evidence instead of trusting the model’s account of what it checked. We will look at where independent verification belongs in an agent workflow and how to turn that boundary into something we can actually test.
You will leave with a specific, repeatable approach: before a consequential action, do not just ask what the agent claims. Ask what evidence exists for that claim and whether the system can independently confirm it. This can be applied to workflows involving compliance checks, identity verification, deployment approvals, and other situations where an agent’s account of its own actions needs to be verified before something real happens.
Registrations for our other 2027 Chapter Events are live too.
Register for Jan 2027 Event: If AI is everywhere, what shall I do? | MoTaverse
Register for Feb 2027 Event: AI Governance - Whose headache is it anyway? | MoTaverse
Register for March 2027 Event: AMA on The Future of Testing in an AI Enabled Enterprise | MoTaverse
Most conversations about agent safety focus on permission: can the agent act, or is it blocked from doing so? That matters, but it is a problem we have been dealing with in software for years through authorization checks. The harder problem is different. An agent can claim that a required condition was verified when there is actually nothing confirming that it was. The system does not necessarily get bypassed. Instead, it trusts what the agent says and treats that claim as if it were evidence.
This session treats the code around the model, the harness, as the thing under test. I will run a small controlled experiment using the same model, prompt, and tools, with an approval record still marked pending rather than approved or rejected. One harness will rely on the agent’s claim, while the other will independently check the actual approval record before allowing the action. Only the harness changes.
The goal is not to prove that one harness is smarter. It is to show what the system catches when it requires evidence instead of trusting the model’s account of what it checked. We will look at where independent verification belongs in an agent workflow and how to turn that boundary into something we can actually test.
You will leave with a specific, repeatable approach: before a consequential action, do not just ask what the agent claims. Ask what evidence exists for that claim and whether the system can independently confirm it. This can be applied to workflows involving compliance checks, identity verification, deployment approvals, and other situations where an agent’s account of its own actions needs to be verified before something real happens.
Registrations for our other 2027 Chapter Events are live too.
Register for Jan 2027 Event: If AI is everywhere, what shall I do? | MoTaverse
Register for Feb 2027 Event: AI Governance - Whose headache is it anyway? | MoTaverse
Register for March 2027 Event: AMA on The Future of Testing in an AI Enabled Enterprise | MoTaverse
Thu, 22 Apr 2027
13:30 - 14:30 UTC
Location: Online
Host
Speaker
Take this 7-question assessment to see where your QA team stands and get a plan for what to automate next.
