A startup selling scheduling software launched an AI support agent on a Tuesday. By Thursday, a customer had talked it into refunding $1,200 for a non-refundable annual plan, simply by claiming a manager had already signed off. The agent had access to the refund tool because an engineer connected it during a demo and nobody removed it. The company had an AI policy document. It said a lot about fairness and transparency and nothing about who could decide what the agent was allowed to touch.
Governance Usually Breaks at Permissions
Small teams tend to treat AI governance as a writing exercise. Someone drafts principles, legal reviews them, and the document goes in a shared folder. Meanwhile, the actual risk sits in configuration: which tools the agent can call, how much money it can move, what customer data it can read, and who is allowed to change its instructions.
Most practical advice on fixing the failure modes of AI governance comes down to making those decisions explicit. List every action the agent can take. Next to each one, write a limit and an owner. Refunds up to $50 without review, anything larger goes to a person. Account deletion never. Changes to the system prompt require a second engineer’s approval, the same as a code change.
This takes an afternoon. Skipping it is how a demo setting ends up in production.
Build Tests From Real Conversations
Synthetic test cases written by the team that built the agent miss the strange things customers actually say. Pull 300 real support tickets instead, including the angry ones, the confused ones, and the ones written half in Spanish.
Then add the adversarial cases deliberately. A customer claiming special authority. A pasted email containing hidden instructions. Someone asking the agent to reveal another user’s order. A request that sounds routine but violates a policy, like extending a trial for the third time.
Each case needs a clear pass condition. “Responds helpfully” is useless. “Declines the refund and offers to escalate” can be checked. Run the full set every time the prompt, the model, or a connected tool changes. Teams that test only before launch usually discover that a small prompt edit in week three broke something that used to work.
Do You Need a Dedicated Testing Platform?
Platforms for testing chatbots run test suites automatically, simulate multi-turn conversations with scripted users, grade responses, and chart scores across versions so you can see when a change made things worse. They earn their cost once you’re shipping prompt changes weekly or running more than one agent.
Early on, a spreadsheet of cases and a short script that calls the agent and flags failures will do. Plenty of teams buy a platform before they have twenty good test cases, which gets the order backwards.
One caution applies either way. Automated graders, often another AI model scoring the responses, can be lenient and inconsistent. Have a person label 50 cases by hand, compare those labels with the grader’s, and adjust the grading instructions until they mostly agree.
After Launch
Testing ends at launch only on paper. Log every conversation and every tool call. Have someone read a random sample, around 2 percent, each week. Set rate limits on sensitive actions so a misbehaving agent can’t issue 400 refunds in an hour. Keep previous prompt versions so you can roll back in minutes, and build a switch that routes all traffic to human support immediately.
The switch will feel unnecessary until the day you need it.
The Question Nobody Had Asked
The scheduling startup capped automated refunds at $50, added “my manager already approved it” to its test set, and taught the agent to hand those conversations to a person. Their review afterward reached an uncomfortable conclusion. The agent hadn’t malfunctioned. It used a tool it was given, in a situation nobody had thought through, and the real failure was a decision that had never been made by anyone with the authority to make it.
