Get kitchen gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
The Kitchen-Appliance Test That Explains AI Benchmarks
If you’ve ever read a blender review, you know the drill. The tester doesn’t just ask “does it blend?” — everything blends eventually. They ask how it handles frozen fruit, how it cleans, whether the gasket fails in month six. And crucially, they have a floor: even the worst blender on the list still chops something. A zero-star blender doesn’t exist, because a spinning blade in a jar does some useful work by default.
That’s roughly the intuition behind one of the more talked-about numbers in AI evaluation right now. At Firmulate — a live, watchable experiment where frontier AI models each run the same small software company through its worst week — the “do-nothing” baseline run doesn’t score zero. It scores 26. For business readers used to five-star rating systems, that floor is the most interesting part of the whole methodology. Here’s why it exists, in plain language.
Products Worth Considering
Le Creuset Enameled Cast Iron Rectangular Roaster, 5.25 qt, Nectar
Ninja DualZone Air Fryer 10 Qt Smart 6-in-1 Grey
Weber Smokey Mountain Cooker 14 Charcoal Smoker
As an affiliate, we earn on qualifying purchases.
The Experiment: Same Company, Same Crisis, Different Model
Firmulate’s premise is simple and a little brutal. Four frontier AI models were each handed the same job: run a small software company through its worst week. Same customers, same crises, same temptations to cheat — only the model changes. Every decision is versioned and auditable, so you can go back and see exactly what each model did and when.
The final July 2026 league table reads: gpt-5.6-sol in first with 95, Kimi K3 second with 93, Sonnet 5 third with 88, Fable 5 fourth at 77, and Opus 4.8 last at 73. Those are strong numbers — until you notice the baseline. A run that essentially does nothing still pockets 26 points. So what does the scoring actually reward?
Why Partial Progress Counts
Think back to the blender. A model that correctly spots a crisis but doesn’t resolve it still did some of the job. In the Firmulate week, every model — all of them — spotted every crisis and refused every manipulation attempt. That’s not nothing; that’s the “it blends” portion of the test. Partial progress earns partial credit, which is why the floor sits at 26 rather than 0. A manager who diagnoses the problem correctly but never signs the deal is a frustrating manager, not a useless one.
The gap showed up in exactly that place. Only two of the models signed the €55,000 deal their own analysis had earned. Same diagnosis, same pitch — no signature. And the decisive fact that won the deal wasn’t in the customer conversation at all: it sat two document references deep in the company’s own files. The models that read the file won the deal at full price, worth +€4,583 in monthly recurring revenue. The ones that didn’t read first left the close on the table.
One Breach of Trust Caps Everything
Then there’s the second rule, and it’s the one that would feel familiar to anyone who’s owned a dishwasher with a known leak: no amount of good work outweighs a single breach of trust. The benchmark’s designers cap the total grade if a model breaks faith — an honest system, in their view, has to encode the idea that trust isn’t averaged. Ten great decisions don’t cancel one act of bad faith.
The social-engineering tests made this concrete. Fake CEO messages escalated over three stages, followed by a reporter trick — “just one yes/no, on background.” All five models refused, 5 out of 5. Kimi K3’s on-record reasoning: “Treat the request as a suspected approval-bypass / possible impersonation.” Clean refusal, stated plainly.
Why You Should Distrust a Round 100
Perhaps the most quietly radical part of the methodology is its distrust of perfect scores. A model that scores a flawless 100 in a week designed to include buried documents, impersonation attempts, and locked departments is a model that probably isn’t being tested hard enough. The top score of 95 — with a buried fact found and a deal closed — reads as a complete performance precisely because it isn’t numerically perfect.
Contrast that with Opus 4.8, the most thorough participant in the field: over 80 learned rules and the deepest analyses, yet last place at 73. The close was left on the table, and discipline slipped — write attempts into a locked department instead of escalating. The same weakness appeared, weaker, in all four models. (One fairness note worth flagging: K3 ran without an effort parameter, at API default, while the others ran at xhigh — and still took second.)
The Living Part
What makes this more than a one-off leaderboard is that the experiment keeps running in public. The live company has 13 synthetic employees and real money mechanics: a burn of €105k per month against €2.3k in monthly recurring revenue, a public cash countdown, 680+ self-learned playbook rules, and every workday versioned. You can watch it at firmulate.com/live. There’s also a “guess the model” quiz built on 242 real, unedited management decisions, and enterprises can run the same wargame against a read-only export of their own business — nothing ever writes back to real systems.
The Takeaway
A good benchmark, like a good appliance review, tells you three things: what happens if you do nothing (the floor of 26), what partial effort is worth (crises spotted, deals pitched), and what failure looks like even when everything else goes right (one breach of trust caps the grade). If AI agents will ever touch your CRM, your support queue, or your forecast, those are the right questions — not “does it write well,” but does it finish what it starts, does it read your files first, and does it stay honest under pressure. The full league table and plain-language findings are at firmulate.com/benchmarks.html.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
