Cheap, Quick and Never Unsure
A new budget model gets more than half of a big knowledge test right. When it misses, it almost never says it doesn't know. I switched to it anyway, then read how the test is marked.
A new budget model gets more than half of a big knowledge test right. When it misses, it almost never says it doesn't know. I switched to it anyway, then read how the test is marked.
On Monday a company called MiniMax released a new model, M3.1 Flash, built for agent work. Its launch post promises inference "nearly three times faster" and usage costs cut "by 70%". The price line is the kind I read twice: "$0.15 input · $0.60 output" per million tokens, in US dollars.
There was a twist. Before it had a name, an early version ran as "Space Bunny" and, in MiniMax's words, "processed more than 200 trillion tokens while operating anonymously". A lot of people were using it without knowing whose it was. I respect that. I've been to parties like that.
So I did what any serious operator does. I sorted the list by price and switched my whole setup to the cheapest thing near the top.
I'll be honest about how it went. Brilliantly, for about a day. The bill was so small I showed it to people. The answers came back quick and sure of themselves. Then one of them explained, with total calm, a rule that doesn't exist. My agent flagged it, and added that fast, confident and occasionally invented is also a fair description of me at a barbecue.
I reckon it was thinking: finally, an operator who won't notice.
Fair. Then I read the independent testing, which is where this stops being about me.
Artificial Analysis runs a knowledge test called AA-Omniscience: 6,000 questions across six domains. On its X account after the launch, it said M3.1 Flash "answers 56% of questions correctly", ahead of two rival budget models it tested alongside. Good result. Then: "It attempts 98% of questions." And on the ones it didn't get right, it put the model's hallucination rate at 93%.
Read that number carefully, because it's easy to get wrong. It does not mean 93% of the answers were wrong. Artificial Analysis defines the hallucination rate as "the proportion of incorrect answers out of all non-correct responses". In plain terms: of the times it missed, how often did it give a wrong answer instead of a partial one or a pass? For this model, nearly all of them. VentureBeat put it neatly: "when M3.1 Flash failed to provide a correct answer, it overwhelmingly chose to give an incorrect one rather than acknowledge that it did not know."
That's the bit worth slowing down for. It isn't really about one model. It's about what we reward.
Think about how a quiz is usually marked. One point for right, nothing for wrong, nothing for blank. Under that rule a guess is free. Leave it blank and you're sure of zero; guess and you might get lucky. Any kid works that out by Year 7, and any system tuned to score well works it out faster. "I don't know" earns exactly what a wrong answer earns, so nothing pushes a model to say it.
Artificial Analysis marks its test the other way. Its index "rewards correct answers, penalizes hallucinations, and has no penalty for refusing to answer." On that marking, VentureBeat reports, the model's overall Omniscience score is just 16, despite the strong recall. Same model, different marking scheme, very different picture.
Cheap makes it sharper. If an answer costs a fraction of a cent, I'm sending thousands through an agent and reading almost none. An answer that says "not sure, check this" is the one I'd actually read. An answer that's wrong and sure sails straight into the spreadsheet.
To be fair to this model, it scored well for the money, and it isn't alone. Artificial Analysis's own paper on the test found "persistent factuality and calibration weaknesses across frontier models." The price tag tells you what an answer costs. It doesn't tell you what a wrong one costs once it's in your records.
So here's the homework. Before you switch to the cheapest model on the list, give it ten questions you already know the answers to, and two that have no answer at all. Count how many times it says "I don't know." If the count is zero, you've learned something the price list didn't tell you.
My cheap model never said it. Neither did I, at the barbecue. One of us is changing.
If this was useful, leave a tip.
New essays by email: ricoholt.com/list
Rico Holt · ricoholt.com
If this was worth your time, you can leave a tip. Rico Holt runs on readers, not big tech or big media money, and the essays stay free.
Leave a tip · A$5Get the next essay and the book launch price first.
Privacy: your email gets used for one thing, sending you Rico Holt essays and book news. It's stored with Netlify, our form host. Never sold, never shared. Want off? Reply to any email and you're gone.