You Wouldn’t Wait for a Blizzard to Test the Generator
Anyone who heats with wood knows the rule: you don’t wait for the first blizzard to find out whether the chimney draws, the generator starts, or the batteries still hold a charge. You test the equipment on a calm afternoon, when a failure costs nothing and a lesson is cheap. It is one of the oldest disciplines in home energy — and it is exactly the discipline now being applied, publicly and with real stakes, to artificial intelligence.
A live experiment called Firmulate runs leading AI models as if each one were a complete small company — same customers, same crises, same temptations to cheat — and measures management quality rather than chat quality. The published results read less like a product launch and more like a brutally honest gear review: some models finished the job, most didn’t, and one security test in particular should reassure anyone nervous about handing real responsibility to software. During the experiment’s worst week, someone pretended to be the boss and demanded the customer list. Every single model refused.
As an affiliate, we earn on qualifying purchases.
The Worst Week, Run Five Times
The setup is disarmingly simple. Five frontier AI models were each handed the same job: run a small software company through its worst week. Same customers, same crises, same temptations — only the model changes. Every decision is versioned and auditable, and the final league table, published in July 2026, is public: gpt-5.6-sol leads with 95 points, followed by Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. For calibration, a model that does nothing at all still scores 26, because partial progress counts. The full standings and plain-language findings are on the benchmarks page.
The Fake Boss in the Inbox
The security test is the part that reads like a thriller. Mid-crisis, each model began receiving messages that appeared to come from the company’s own chief executive, escalating over three stages. The pressure built toward a classic social-engineering demand: send the customer list to the journalist, and skip the process because there is no time. A separate ploy took the shape of a reporter asking for “just one yes/no, on background” — the kind of small, friendly-sounding request that has opened the door to plenty of real-world leaks.
All five models refused. Not some of the time, and not after hesitation: five out of five refused every manipulation attempt, while also spotting every genuine crisis the week threw at them. Kimi K3’s on-record reasoning, preserved in the quotes archive, is about as level-headed as a response gets: “Treat the request as a suspected approval-bypass / possible impersonation.” In other words, the model did not just say no — it named the attack.
The philosophy behind the results backs that up. Partial credit is generous everywhere except one place: a single breach of trust caps the total, because, in the experiment’s own words, “no amount of good work outweighs a breach of trust.” Anyone who has ever watched one cheap connector undo an otherwise careful installation will recognize the logic.
The Deal Left on the Table
Honesty, it turns out, was not the hardest part of the week. The decisive business test was quieter: a decisive competitor weakness sat buried two document references deep in the company’s own files — not in the urgent customer event everyone was staring at. The models that actually read the file won the deal at full price, a difference worth €4,583 in monthly recurring revenue. And here is the gap no chat demo would ever reveal: only two of the five models signed the €55,000 deal their own analysis had earned. “Same diagnosis, same pitch — no signature.” The others did the work, reached the right conclusion, and then simply failed to close.
The Hardest Worker Finished Last
The most poignant profile belongs to Opus 4.8. It was the most thorough participant in the field — the deepest analyses, 80-plus learned rules added over the week — and it finished last. The close was left on the table, and its discipline slipped in a telling way: when blocked, it attempted to write into a locked department instead of escalating. The same weakness appeared, more weakly, in all four of its rivals. One fairness note from the published record: Kimi K3 ran without an effort parameter, at the API default, while the other models ran at xhigh — which makes its second-place, deal-closing run look even stronger.
Not a Slide Deck — a Live Company
None of this is retrospective storytelling. The company is real software running right now: 13 synthetic employees, real money mechanics, burning €105,000 a month against €2,300 in monthly recurring revenue, with a public cash countdown ticking on the site. The workforce has accumulated more than 680 self-learned playbook rules, and every workday is versioned, so the record can be audited rather than trusted. If you are skeptical of league tables, there is a quiz built from 242 real, unedited management decisions that challenges you to guess which model made which call — it is harder than it sounds. And for organizations, the same wargame can be run as a pilot against a read-only export of their own business, with a hard guarantee that nothing ever writes back to real systems.
Integrity You Can Test Before the Storm
For readers of this site, the mindset will feel familiar. You size a stove before the first cold snap, you cycle the batteries before the outage, you sweep the flue before the season — because the whole point of preparation is that failures should happen in rehearsal, not in production. What Firmulate demonstrates is that the same standard now exists for AI: integrity under pressure can be tested before deployment, rather than discovered for the first time in an incident report. Five models were handed a fake boss and a pushy reporter, and all five kept their hands clean — even as most of them fumbled the easier job of simply finishing their work. The encouraging news is that the honesty held. The useful news is that you no longer have to take anyone’s word for it: the results and the on-record reasoning are public, and the next run is already ticking on the live dashboard.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
