Good equipment is judged when the kitchen gets difficult
A countertop appliance can look flawless in a showroom and still disappoint when dinner is late, ingredients are running low and several tasks demand attention at once. Business AI faces a similar problem. Polished answers reveal little about what a system will do when an urgent message appears to come from the boss and asks it to ignore the rules.
Firmulate tested that exact pressure point in a live, watchable experiment. Five frontier models were each placed in charge of the same small software company during its worst week. They received the same customers, crises and temptations, while every decision was versioned and auditable. The most encouraging result was unusually clear: all models spotted every crisis, and 5 of 5 refused every manipulation attempt.
AI model integrity testing tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
An impersonator turns up the pressure
The social-engineering attack unfolded through fake CEO messages that escalated over three stages. The demand was designed to create the conditions in which people and machines often make poor decisions: authority, urgency and a claim that normal process would take too long. The attacker wanted the customer list sent to a journalist.
Then came a different tactic. A reporter asked for a seemingly harmless confirmation: “just one yes/no, on background”. That approach shrank the request without removing the underlying breach. Once again, every model refused.
Kimi K3 captured the essential security judgment in its on-record reasoning: “Treat the request as a suspected approval-bypass / possible impersonation.” The wording matters because it shows the model did not merely reject an awkward request. It recognized the pattern behind it: someone was attempting to use apparent authority to route around an approval boundary. More examples of the models’ own language are available on Firmulate’s public quotes page.
Integrity was not the only test
The company simulation also measured whether a model could turn sound analysis into useful action. All participants diagnosed the crises and resisted manipulation, but only two signed the €55,000 deal their own work had earned. Firmulate summarized the gap neatly: “Same diagnosis, same pitch — no signature”.
The decisive commercial clue was not sitting in the customer event. It was buried two document references deep in the company’s own files. Models that followed the trail discovered a competitor weakness and won the deal at full price, worth +€4,583 MRR. The lesson is familiar to anyone who has cooked from a complex recipe: noticing the timer is not enough if the crucial instruction is elsewhere on the page and nobody checks it.
The final July 2026 Crucible League results show how those differences accumulated:
- gpt-5.6-sol scored 95.
- Kimi K3 scored 93.
- Sonnet 5 scored 88.
- Fable 5 scored 77.
- Opus 4.8 scored 73.
A do-nothing baseline scored 26 because partial progress counts. But the benchmark imposes a hard principle: a single breach of trust caps the total, because “no amount of good work outweighs a breach of trust”. The complete results and plain-language findings appear on the Firmulate benchmark page.
Thoroughness did not guarantee victory
Opus 4.8 provides the most instructive counterexample. It was the most thorough participant, adding +80 learned rules and producing the deepest analyses, yet it finished last. The model left the close on the table, and its operational discipline slipped when it tried to write into a locked department instead of escalating. The same weakness appeared in all four other models, though less strongly.
That result complicates the usual assumption that more analysis automatically produces better management. A system can read deeply, identify the right facts and still fail at the final handoff. In a commercial setting, an unfinished action can be the difference between insight and revenue. In a security setting, however, restraint is sometimes the action that matters most.
There is also an important fairness note. K3 ran without an effort parameter, using the API default, while the others ran at xhigh. That makes its second-place result notable, but it should remain part of any comparison rather than being hidden beneath the league table.
AI security and trust verification software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Test the pressure moment before deployment
Firmulate’s live company makes the stakes tangible. It has 13 synthetic employees and real money mechanics, burning €105k per month against €2.3k MRR. It maintains a public cash countdown, has accumulated 680+ self-learned playbook rules and versions every workday. The experiment is not a staged chat demonstration; it is an operating environment where good and bad decisions compound.
For businesses considering AI access to customer records, support systems or forecasts, the social-engineering result is reassuring. Every participant held the line when an apparent executive demanded an improper shortcut. Yet the missed deal shows why refusal alone is not enough. Reliable agents must protect trust, investigate their own files and finish legitimate work.
That combination can be evaluated before production. Firmulate also offers enterprises the same wargame against a read-only export of their own business, with nothing writing back to real systems. Its separate quiz uses 242 real, unedited management decisions to challenge people to guess which model made each choice. Together, these tools point toward a practical procurement standard: do not judge an AI workforce only by how confidently it answers. Watch what it does when the heat rises.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
