James presents a framework for testing AI systems based on three core concepts learned from martial arts competition. He introduces Verifier's Law (the speed of development is proportional to how verifiable the task is), using a determinism pyramid to make AI outputs more testable through structured outputs, tool calls, and constraints. The second concept is measured risk, using risk matrices to prioritize bugs and communicate effectively across teams. The third is 'win or learn' - releasing changes safely with feature flags, A/B testing, and monitoring to learn from both successes and failures. James demonstrates these concepts through a fruit-selling chatbot example, showing how to evaluate features like live pricing, upselling, and safety guardrails. The presentation emphasizes that these aren't new concepts but applications of shift-left testing, risk communication, and shift-right observability specifically adapted for AI systems.
Comments