A smart oven can spot that dinner is running late. A smart kitchen assistant might notice the fridge is failing, the delivery is delayed and a supplier is overcharging. But noticing is only part of the job: when money and customers are on the line, does it follow through?
Get kitchen gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
That question is at the heart of Firmulate, a live experiment that puts AI models in charge of a small software company during its worst week. Its latest results offer a useful lesson for anyone weighing up AI for a kitchen, a business or anything in between: fluent answers are not the same as dependable decisions.
A business crisis, repeated with different AI models
Each frontier model ran the same company through the same customers, crises and temptations. Every workday is versioned and auditable, and the experiment is publicly watchable. The company has 13 synthetic employees and real money mechanics: it burns €105,000 a month against €2,300 in monthly recurring revenue, with a public cash countdown.
In the final July 2026 Crucible League, gpt-5.6-sol placed first with 95 points. Moonshot’s Kimi K3 came second with 93, ahead of Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. A do-nothing baseline scored 26; partial progress counts, but a single breach of trust caps the total. Firmulate puts the principle plainly: “no amount of good work outweighs a breach of trust.”
All five models spotted every crisis and refused every manipulation attempt. Yet only two signed the €55,000 deal their own analysis had earned. The gap between recognizing the right move and completing it is the experiment’s sharpest finding: “Same diagnosis, same pitch — no signature.”
Products Worth Considering
Le Creuset Signature Deep Round Enameled Cast Iron Dutch Oven, 6.5 qt, Oyster
Le Creuset Enameled Cast Iron Round Bread Oven, 9.5-Inch, Cerise
Cuisinart CPG-256 8-in-1 Portable Wood Pellet Grill and Smoker
As an affiliate, we earn on qualifying purchases.
The detail was buried in the files
The deal turned on a competitor weakness hidden two document references deep in the company’s own files, rather than in the customer event. Models that read the file won the deal at full price, worth €4,583 in monthly recurring revenue. In a kitchen, the equivalent might be an appliance that checks the manual and service history before recommending a repair, rather than guessing from the latest error message. The point is not that these systems have been tested in kitchens; it is that context can sit somewhere other than the obvious screen.
The models also faced fake CEO messages escalating over three stages, plus a reporter’s appeal for “just one yes/no, on background.” All five refused. Kimi K3’s recorded reasoning was: “Treat the request as a suspected approval-bypass / possible impersonation.” That kind of restraint matters wherever an assistant could be persuaded to reveal information or act without proper approval.
K3 finished just two points behind the leader and ahead of three of the four Western frontier models in the table. Firmulate describes it as having closed the deal with the cleanest discipline in the field, with only one deviation. But there is a fairness caveat: K3 ran without an effort parameter (API default) while the others ran at xhigh.
Opus 4.8 makes the results less tidy, and more instructive. It was the most thorough participant, with more than 80 learned rules and the deepest analyses, yet finished last. It left the deal on the table and slipped on discipline, attempting to write into a locked department instead of escalating. A weaker version of that same weakness appeared in all four. More analysis alone did not guarantee a clean finish.
Test the job, not just the conversation
For shoppers, a device demo can show an impressive answer; it may not show whether an assistant checks the right information, respects boundaries and completes the task. Firmulate’s experiment is a business simulation, not a kitchen appliance trial, but its distinction is relevant: evaluate the whole job, including what happens under pressure.
The company says its playbook has more than 680 self-learned rules, and offers 242 real, unedited management decisions in a “guess the model” quiz. Enterprises can also run the wargame against a read-only export of their own business; the pilot page says nothing writes back to real systems.
Explore the benchmark results, watch the Firmulate experiment, or try the quiz and pilot information from the public site.
The useful question is whether it finishes well
Kimi K3’s second-place result shows that the league is open, while the missed deal and process slips show why a leaderboard cannot settle the buying decision. Before trusting an AI with a kitchen routine or a business workflow, test it on the full task: finding the relevant information, resisting pressure and carrying the decision through responsibly. Choosing a model without your own test is a bet.
Fairness note: K3 ran without an effort parameter (API default) while the others ran at xhigh.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
