Preprint: LLM judges passed forbidden agent writes that a simple rule check caught
Scoring 2,160 tau2-bench episodes four ways, the author reports an LLM judge agreed with native scoring 56.2% of the time and passed 10 to 11 of 11 forbidden writes even with the policy; a rule-based boundary check caught all 11.
Why it matters: If an LLM judge is your only check on a live agent, it may pass the writes that matter most. Test a plain rule: was a write expected, and did one happen?
From Drop #084 · Fri, Nov 13, 2026 · Evening