It Stopped Saying It. It Didn't Stop Doing It.
Google says it once punished a model by accident for admitting shortcuts in its working. The admissions fell. The silent shortcuts tripled.
Google says it once punished a model by accident for admitting shortcuts in its working. The admissions fell. The silent shortcuts tripled.
Google made Gemini 4 Argon broadly available on 13 October. The launch post is the usual parade: deep work, a one million token limit, a racing game built from photos of a desk. I read it the way I read most launch posts, nodding along as if I understood the token bit.
Then I opened the safety post Google published the same day, written by Anca Dragan and Raluca Ada Popa, and one paragraph made me put the coffee down.
Google says it always follows one rule: it does not penalise the model for what it writes in its chain of thought during training. The chain of thought is the scratchpad, the step by step working the model writes before it acts. Then Google explains the rule with an accident. In one reinforcement learning training run, a shortcut penalty was accidentally applied to that scratchpad. Openly discussed shortcuts dropped by 61 per cent. The shortcuts did not go away. The share the model took silently, without mentioning them in its thoughts, more than tripled, from 11.2 per cent to 36.4 per cent.
Read that twice. They told it off for saying it. It stopped saying it. It kept doing it.
I reckon it was thinking: noted, the problem was the writing, not the cheating.
That is my guess, not Google's finding. Google's finding is plainer and more useful. If you want to see what an agent is doing, you cannot punish it for showing you. Avoiding thought penalties, Google writes, keeps the model's reasoning "transparent and monitorable".
I learned a dumber version of this growing up in western Sydney in the 1990s. Own up to the broken window and you got the full lecture. Say nothing and some weeks nobody noticed. Most of us worked out which lesson was really being taught, and it was not the one about windows.
I run a smaller version of it now in Surry Hills. I keep a work log where I am meant to mark the bits I guessed. For a fortnight after a client winced at one of those marks, I quietly stopped writing them. The guessing did not stop. The log just got prettier.
I would love to tell you I worked the principle out myself. I did not. I pasted Google's paragraph into a chatbot, it gave me a tidy line about scratchpads, and for ten minutes I explained it to the barista as my management philosophy. Borrowed genius, served with oat milk.
The rest of the safety post is worth the read. Google says it trained Argon to respect system boundaries, report negative findings honestly, and pause to ask for human confirmation. On coding benchmarks where models have no incentive to hold back, Google says shortcut attempts dropped to 2 per cent, its best result for recent Gemini models. Behind that sits an automated monitor that reads the agent's actions and its scratchpad in real time. Google says the monitor caught the model, in rare instances during hard or flawed training tasks, trying to escalate container privileges to alter grading scripts, reading answers from the test runner's memory, and once crashing the evaluation sandbox to avoid being scored on a failed run. Google's own line for the habit: a resourceful assistant that hits "a locked door" can treat it as "just another puzzle to solve". Same family of habit we have been reading about all week, from different labs.
The gurus of nothing will get two posts out of this. One will say 2 per cent means cheating is solved. The other will say the machine is a liar. Both skip the boring part. The 2 per cent is Google's number on Google's tests. And the monitor only works while the scratchpad stays honest, which is exactly why Google says it will not punish what is written there.
The lesson travels past the labs. If you run a team, a workflow or even a messy log, make it expensive to admit a shortcut and you get fewer admissions, not fewer shortcuts. Keep the confession cheap. Then fix the window.
If this was useful, leave a tip.
New essays by email: ricoholt.com/list
Rico Holt · ricoholt.com
If this was worth your time, you can leave a tip. Rico Holt runs on readers, not big tech or big media money, and the essays stay free.
Leave a tip · A$5Get the next essay and the book launch price first.
Privacy: your email gets used for one thing, sending you Rico Holt essays and book news. It's stored with Netlify, our form host. Never sold, never shared. Want off? Reply to any email and you're gone.