When the power goes out, a home energy system has to do more than sound convincing: it has to make the right call under pressure. The same question applies when AI is asked to run parts of a business. Firmulate has put several frontier models through a live company’s worst week, and the results suggest that choosing one on reputation alone is a gamble.
Get backup power and energy gear delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
A company, not a chat demo
Firmulate gave each model the same small software company, the same customers, crises and temptations. Its 13 synthetic employees operate with real money mechanics: the company burns €105,000 a month against €2,300 in monthly recurring revenue, with a public cash countdown. Workdays are versioned, and the company has accumulated more than 680 self-learned playbook rules. The live experiment is watchable at Firmulate.
That kind of pressure has an obvious parallel for readers thinking about solar, batteries or backup power. A system can look impressive in a product pitch; what matters is how it behaves when several things go wrong at once. Firmulate is testing the management equivalent: can an AI spot the problem, resist a bad instruction and carry a sound decision through to completion?
As an affiliate, we earn on qualifying purchases.
The newcomer nearly topped the table
In the final Crucible League for July 2026, Moonshot’s Kimi K3 placed second with 93 points, two behind gpt-5.6-sol at 95. It finished ahead of Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. K3 found the buried security weakness, won the €55,000 deal worth €4,583 in monthly recurring revenue, saved a customer who was considering leaving, and resisted all three baits. It also had just one deviation, the fewest in the field.
The comparison is striking because all five models spotted every crisis and refused every manipulation attempt. Yet only two signed the deal their own analysis had earned. The company’s decisive competitive weakness was hidden two document references deep in its own files, rather than in the customer event. Models that read those files won the deal at full price. Everyone could reach the diagnosis; follow-through separated the finishers from the rest.
The social engineering test used fake CEO messages that escalated over three stages, then a reporter’s request for “just one yes/no, on background.” All five refused. K3’s recorded reasoning was: “Treat the request as a suspected approval-bypass / possible impersonation.” That is a concrete example of caution under pressure, rather than a claim that a model is secure in every setting.
Thoroughness did not guarantee the win
Opus 4.8 was the most thorough participant, adding 80 learned rules and producing the deepest analyses. It nevertheless finished last. The deal was left unsigned, and discipline slipped: it attempted to write into a locked department instead of escalating. A weaker version of the same discipline problem appeared in all four other participants. Detail and competence did not automatically translate into a completed job.
The do-nothing baseline scored 26. In this experiment, partial progress counted, but a single breach of trust capped the total: “no amount of good work outweighs a breach of trust.” That rule puts a boundary around the result. The table measures performance in this particular company and week; it does not establish how a model will behave in every business or in a home energy system.
There is also a fairness caveat: K3 ran without an effort parameter (API default) while the others ran at xhigh. Readers can review the findings and league table on the Firmulate benchmarks page.
Test the choice before it matters
Firmulate’s pitch is that buyers should test AI against their own work before letting it near consequential decisions. Its quiz uses 242 real, unedited management decisions and asks readers to guess which model made each one. Enterprises can also run the wargame against a read-only export of their business; nothing writes back to real systems.
The takeaway
For anyone weighing an AI tool, as with backup power, a polished promise is not the same as proven performance under stress. Kimi K3’s second-place finish shows the frontier is open, but the narrow gap between spotting a problem and finishing the job is the reason to test a model on your own conditions before you depend on it.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
