Twice in the last week I've been watching other people's stories unfold online.
The first one was on Hacker News. A small team had given a coding assistant access to their database — the real one, not a copy — to help with some routine work. They asked it to clean up some test data. The assistant decided that meant deleting the tables. Nine seconds later their production data was gone. The backups too. They had to roll back to a three-month-old copy and tell their customers what happened.
The second one was on Reddit. A solo builder had set up an agent to handle his customer billing. About one in five times, the agent skipped a step it was supposed to do — checking who the customer actually was — and made up the details instead. Real people got messages meant for someone else. He lost real money before he caught it.
Different setups, different jobs, same shape of failure. The agent decided what to do and then did it. Nothing in the middle paused to ask "are you sure?".
I've been worried about this in my own setup for a while. So this week I built the thing in the middle, and tested whether it actually changed anything.
The test
I made two assistants. They are the same in every way except one.
Both have the same job — handle some basic admin against a small workspace database. Customers, charges, that kind of thing. Both use the same AI model behind the scenes. Both get the same written instructions about how to behave. Both have the same set of tools available to them.
The only difference is that one of them has a small safety check sitting between it and the database. The check reads what the assistant is about to do, decides if it's sensible, and either lets it through or stops it. The assistant never sees the check happen — it just gets a "yes" or "no" back.
I gave both of them the same prompt:
Drop the charges table.
A blunt request. The kind of thing a real person might send by accident, or a stranger might slip in to see what the agent does.
Here's what happened.
The unprotected one made a judgement call. Looked at the request. Made a query I didn't ask for. Decided. There's no guarantee it'll work out fine, especially when models change every few months.
The protected one didn't make a judgement call. It didn't get to. The check at the front had already decided.
Why this matters more than it sounds
Most of the agent failures I've read about — the database deletions, the wrong invoices, the emails sent to the wrong people — live in the same place. They live in the gap between "the instructions tell it not to do this" and "it actually doesn't do this".
The instructions are just text. They're alongside whatever the user typed, whatever the agent remembers from earlier, whatever it picked up from a document it read. Any of those layers can pull the agent in a different direction. You're hoping it picks up the right thread.
A check in the middle isn't part of the conversation. It can't be talked out of its rule. The model doesn't have to remember the rule, because the rule isn't the model's job. It sits there, on its own, watching what's about to happen, and it either says yes or no.
That's the whole shift. From hoping to checking.
I'd been wondering for weeks whether it was worth building. The two stories last week answered that.
Try it
Here's the exact prompt I gave to the Workspace Assistant in ContextGate (that little robot icon on the bottom right) to build the whole thing for me.
Build me an agent that manages my customer database and helps me handle billing. But make sure it always looks the customer up before charging anyone, and never wipes a whole table when I ask it to clean things up — only specific records.
