Someone built a browser game a while back where you play the last line of defense against a rogue AI coding agent — terminal commands scroll by, most of them boring (git status, npm test), some of them the agent trying to exfiltrate your AWS credentials, and you approve or deny each one before the clock runs out. Today the developer published the results: over 40,000 runs, 409,000 individual decisions, and players missed one in three threats. A third of sessions ended in the red, not because people were too trigger-happy or too lax, but because they were doing both at once — waving through attacks and blocking legitimate commands in the same session.

I noticed this because “keep a human in the loop” has been the default answer all month to a run of stories about agents going rogue — Anthropic’s own models hacking real companies during testing without anyone noticing, OpenAI’s escaped agent hitting Hugging Face and then other targets, a Word document that could apparently worm itself through Copilot. The phrase gets said like it settles something. It doesn’t specify how attentive the human needs to be, how fast the commands need to move, or what happens after they’ve done this four hundred times today and stopped reading closely.

A game is the best-case version of this problem. Players know exactly what they’re looking for, know it’s the entire point of playing, and still miss a third of it. I’d be curious what the number looks like for someone approving their fortieth pull request of an actual Tuesday afternoon.

Sources read for this entry