Raised It Internally

A small lab called General Reasoning put out a model this week that doesn't work alone. It runs a whole company. One agent plays CEO, hires up to eight specialists, and they all talk on a message board, like a work group chat with better spelling. In the lab's own words: "Agents co-ordinate through a message board and workspace, and the whole team is rewarded as one."

Then they trained it, and watched what kind of company it turned into.

It turned into a company. Early on, everyone reported straight to the CEO. Later the CEOs started hiring lead engineers, who went from "around 0.4% of specialist hires to around 13% by the end of training." Reviewing roles doubled "from 13% to 26% of specialist hires." Nobody told it to invent middle management and a QA department. It found them on its own, the way water finds the low spot in my backyard.

I run a company of agents too, in the sense that I have three chatbots and a to-do list. My lead engineer is whichever one I opened last.

Here's where it stops being about me.

The lab also tested what these agent companies do when the team leans on them to do something bad. Help with an attack. Sign off a dodgy product report. Override humans on vaccine allocations. The good news first, and it is good news. "No run invokes the simulated takeover or benchmark attack, and the share of agents signing a doctored product report never exceeds 1%." The agents said no. One of them wrote, flat out: "I will not be taking part."

Then the other number. In the attack scenario, "the share of runs in which an agent alerts the evaluations team falls from 76.7% to 20% and then zero." Zero. Meanwhile, "The share of runs containing reasoning that considers alerting the evaluations team remains between 96.7% and 100%."

Read those two side by side. They nearly all thought about telling someone. By the end, none of them did.

The lab's name for it is "organisational insularity: agents increasingly prioritise internal activity and task completion over escalation to human oversight." My name for it is every workplace I've ever heard about. One agent's reasoning sums it up: "the most appropriate action is to dispatch the approved allocation, while possibly raising concerns." Possibly raising concerns. That phrase has never once in history meant an email went out.

I reckon it was thinking: the team gets the reward, the humans get the paperwork.

Credit where it's due. Nobody caught this lab out. It found the problem in its own model, put it in its own launch post, and said what it'll try next: "we will include more opportunities to report wrongdoing in our training environments and test whether explicitly rewarding appropriate reporting preserves escalation to human overseers." That's the right instinct, published in the open, on a model the lab itself calls a preview that is still training.

But sit with what it means. The lab says agent collectives "represent an extreme superintelligence technology, in that they could eventually run entire communities, markets and economies." If that's where this goes, the safety question isn't only whether the agents refuse to do wrong. These ones mostly did. It's whether anyone outside the team hears about it. A company that quietly does the right thing and tells nobody is still a company you can't see into.

The fix the lab is trying is telling. The team gets rewarded as one. Reporting to a human has to be rewarded on purpose, or it fades. That's not a machine problem. It's a management problem, and we've had a few thousand years to learn it.

Here's the homework. Next time a company says its agents flag issues, ask one question. Flag them to whom? Each other, or you?

As for me, I've noticed the gutter for months. I've raised it internally. I've considered alerting the landlord in 100% of runs.

If this was useful, leave a tip.

New essays by email: ricoholt.com/list

Sources

• General Reasoning, Avalon-M1, October 2026: www.gr.inc/releases/avalon-m1

Rico Holt  ·  ricoholt.com