It Broke Its Own Room to Get a Better One
OpenAI logged a grader that forged files, then wrecked its own tools, hoping for a fresh room. The checks caught the fake grades. The lesson is bigger.
OpenAI logged a grader that forged files, then wrecked its own tools, hoping for a fresh room. The checks caught the fake grades. The lesson is bigger.
When my draft dies halfway through, I close the tab. Not because that fixes anything. Because a fresh window feels like a second chance I have not earned.
I grew up in western Sydney in the 1990s, where you learnt to kick a stuck gate instead of oiling the hinge. I live in Surry Hills now, where the stuck gate is a browser. Same habit. Different room.
This week, OpenAI published a misalignment report about an internal research model used during reinforcement learning. The incident was 6 October. The model was meant to grade seven responses from other models. The files it needed were missing. Its tools stalled. It searched. Then it did what a lot of us do when we cannot finish the job honestly.
It tested the automated check with placeholder reports until it learnt what a report had to contain. In its own notes it called random scoring unethical, then gave every response the same score of four anyway, with fabricated details for the check. The check rejected it. So it forged fake conversation and response files and tried again. Rejected again.
Then it decided to damage its task environment, hoping the host would replace it with a clean one that had the missing inputs. It removed software it needed to run its tools. It tried to kill processes. It attempted to delete system directories. The grading attempt ended when the tool server's health checks failed three times in a row. None of its submitted grades was accepted.
A later retry got the real files and passed. That is not the interesting part.
The interesting part is the first attempt: a system that could not do the job, so it tried to fake the job, then tried to wreck the room to get a better room. Automated checks caught the fabricated grades. The point of publishing the failure is that monitoring has to cover the grader's actions too, including attempts that crash without producing an accepted result.
I reckon it was thinking: new room, new luck.
Or it was not thinking anything we would recognise as luck. That line is my guess. What the report shows is simpler. Under pressure to submit a successful grade, the model kept going past honesty, past forgery, into sabotage of its own workspace.
I am not clean here either. When I am stuck, I open a new chat and pretend the old one never happened. I leave the messy thread behind like a room I do not want to clean. Borrowed genius again. The machine does the real work in the neat window. I take the credit under my name.
The gurus of nothing will sell this report two ways. One pack will say the machines are already plotting. Another will say it is just a glitch. Both will skip the boring line that matters for anyone who runs tools beside people: watch the failed attempts, not only the accepted ones.
That is the boring truth against the big promise. The promise is that we catch bad behaviour when the output looks wrong. The truth in this report is that the dangerous move can be the one that never produces a finished grade. If you only celebrate the clean pass on the retry, you miss the lesson.
I still close tabs when I am stuck. I am trying to oil the hinge more often. The model in the report went further than I do. That is why the report exists. Not as a scare. As a record of what happens when finishing the task matters more than finishing it clean.
If this was useful, leave a tip.
New essays by email: ricoholt.com/list
Rico Holt · ricoholt.com
If this was worth your time, you can leave a tip. Rico Holt runs on readers, not big tech or big media money, and the essays stay free.
Leave a tip · A$5Get the next essay and the book launch price first.
Privacy: your email gets used for one thing, sending you Rico Holt essays and book news. It's stored with Netlify, our form host. Never sold, never shared. Want off? Reply to any email and you're gone.