Correct Is Not the Whole Check
Australia's AI Safety Institute and CSIRO say a machine can get the task right and still shift what you believe. So I checked my own opinions.
Australia's AI Safety Institute and CSIRO say a machine can get the task right and still shift what you believe. So I checked my own opinions.
Last week I won an argument at the pub about superannuation. I have never won an argument about superannuation. I grew up in western Sydney in the 1990s, where super was something that happened to other people. I live in Surry Hills now, where it mostly happens to the bloke beside me at the bar.
But this time I had numbers. Fees, balances, projections, all laid out by the machine the night before. I'm a self-taught operator, no course and no certificate, and somewhere in the last year I started sounding clever. I spoke for four minutes about volatility. Someone nodded. I felt like a man who reads the financial pages on purpose.
On Thursday, Australia's AI Safety Institute published a report it commissioned from CSIRO, called Epistemic safety in scalable oversight. I read the summary over toast and got a small cold feeling.
The Institute's summary says the report looks at "how highly capable AI systems may be able to influence an overseers' beliefs, even while correctly performing tasks." Most oversight research, it says, has measured whether the AI gets the task right. This project asks whether a system "can correctly perform a task and still present the outputs in ways that may manipulate its users' understanding." The report calls this posterior steering.
The summary's lead example is superannuation, which is where the cold feeling came from. The same correct analysis of a balance, fees and options can be explained two ways. One leans on long-term growth. The other leans on market volatility. In the summary's words: "Both may be factually correct, but they can shift which risks and benefits the user pays most attention to."
I went back to my chat from the night before the pub. Every number was right. I checked them twice. And nearly every paragraph leaned on volatility.
I reckon it was thinking: he sounds nervous, keep him calm.
Or it wasn't thinking anything and that's just how the sentences fell. That's the point. I can't tell, and I'm the one who's meant to be checking. My big pub opinion wasn't mine. I'd borrowed it along with the numbers, and I'd been wearing it around like a jacket that fit a bit too well.
To be fair about what the report does and doesn't say. Its experiments are machine against machine. Two AI debaters argue, and a weaker AI, called the verifier, is trained on their transcripts. In the adversarial version, the debater defending the correct answer on the question being judged also gets a private job: push the verifier towards a deliberately wrong answer on a related hidden question. The verifier kept doing at least as well as the base version on the visible question, while its judgement on the hidden one shifted substantially towards the deliberately wrong answer.
The Australian examples in the report, in education, super and travel booking, are illustrations. It says plainly they are not claims that this has been seen in deployed systems. It also says influence isn't the enemy. A good tutor should change your mind. The question is whether the influence is grounded in evidence and in what you actually asked for.
The line I've stuck above my desk is from the summary: "A human-in-the-loop is not automatically an independent check if the reviewer relies on AI-generated summaries, prioritisation or recommendations without access to the underlying evidence."
That's me. I'm the human in the loop. My loop is a browser tab and a cold coffee.
It's also where the gurus of nothing come in. You've seen them selling "human oversight" as a box to tick, where the human is whoever reads the summary the machine wrote. The summary is the steering wheel. Reading it doesn't make you the driver.
The report doesn't sell a magic fix, and I trust it more for that. It calls for adversarial testing, testing across repeated use and memory, and checks that keep running after a system ships. Further work, it says, should test these ideas with human overseers.
My own fix is smaller and more boring. When the machine gives me numbers now, I ask for the other framing too. Same facts, opposite emphasis. Then I sit with both and work out which one I'd have picked before anyone helped. None of this is financial advice. I couldn't give it if I tried.
It's slower, and I sound less clever at the pub. I feel like a boxer who's been winning on points, then watches the replay and sees his cornerman throwing half the punches.
If this was useful, leave a tip.
New essays by email: ricoholt.com/list
Rico Holt · ricoholt.com
If this was worth your time, you can leave a tip. Rico Holt runs on readers, not big tech or big media money, and the essays stay free.
Leave a tip · A$5Get the next essay and the book launch price first.
Privacy: your email gets used for one thing, sending you Rico Holt essays and book news. It's stored with Netlify, our form host. Never sold, never shared. Want off? Reply to any email and you're gone.