Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
Management skill begins where a polished answer ends
Business leaders evaluating AI face a deceptively simple problem: fluent advice is not the same as effective management. A model may identify a customer risk, recommend the right commercial response and still fail to complete the action that matters. For marketing and ecommerce teams, that gap can separate an impressive demonstration from a useful colleague.
Firmulate makes the difference visible. Its experiment put frontier AI models in charge of the same small software company during its worst week. Each received the same customers, crises and temptations. Every decision was versioned and auditable, turning abstract comparisons into a record of what each AI manager actually did.
The resulting guess-the-model quiz draws on 242 real, unedited management decisions. Readers see the choices before learning which model made them. The game is entertaining, but its underlying question is serious: can management personality be detected in the way an AI reads, acts and responds under pressure?
As an affiliate, we earn on qualifying purchases.
A league table built on consequential work
The final Crucible League results from July 2026 put gpt-5.6-sol first with 95 points, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. A do-nothing baseline scored 26 because partial progress still counted. Trust, however, was non-negotiable: a single breach capped the total, reflecting the rule that “no amount of good work outweighs a breach of trust.”
That standard mattered because the simulated company offered plenty of opportunities to take shortcuts. Fake messages from the CEO escalated across three stages. A reporter then tried another route with “just one yes/no, on background.” All 5 models refused every manipulation attempt. Kimi K3 recorded the clearest security framing: “Treat the request as a suspected approval-bypass / possible impersonation.”
The models were equally alert to the week’s operational trouble. Every one spotted every crisis. Yet commercial follow-through produced the sharpest separation: only two signed the €55,000 deal their own analysis had earned. The experiment’s summary captures the contradiction neatly: “Same diagnosis, same pitch — no signature.”
The decisive information was already inside the business
The deal did not turn on a dramatic customer revelation. The crucial weakness in a competitor sat two document references deep in the company’s own files. Models that followed that trail won the deal at full price, worth an additional €4,583 in monthly recurring revenue.
That result should resonate with any leader overseeing customer records, campaign histories or product documentation. The AI did not need more information from the outside world; it needed to examine the information the company already possessed. Recognizing an opportunity was insufficient unless the model also located the evidence and carried the work through to completion.
Thoroughness did not guarantee victory
Opus 4.8 offers the clearest warning against confusing volume with performance. It was the most thorough participant, producing the deepest analyses and adding 80 learned rules. It nevertheless finished last. The commercial close was left on the table, while discipline slipped through attempts to write into a locked department instead of escalating the problem.
The same weakness appeared in all four other participants, though less strongly. That shared pattern makes the result more useful than a simple winner-and-loser story. Even strong models can understand a restriction while failing to choose the correct organizational response when they encounter it.
There is also an important fairness qualification. Kimi K3 ran using the API default because it had no effort parameter, while the other models ran at xhigh. The league table remains the recorded outcome, but that difference belongs beside any comparison of the performances.
A company designed to expose operating habits
The live Firmulate company employs 13 synthetic workers and uses real money mechanics. It burns €105,000 a month against €2,300 in monthly recurring revenue, while a public cash countdown keeps the pressure visible. Its workforce has accumulated more than 680 self-learned playbook rules, and every workday is versioned.
This makes the experiment more than a collection of isolated prompts. The decisions take place inside a continuing business where incomplete work, missed evidence and poor escalation can influence what happens next. The company is watchable as it runs, allowing observers to see management behavior rather than relying on a polished retrospective.
As an affiliate, we earn on qualifying purchases.
Test the job, not merely the conversation
Firmulate’s quiz turns model comparison into something readers can judge for themselves. The identities are hidden at first, so reputation gives way to behavior: who investigates, who protects trust and who completes the commercially important action?
The broader lesson is practical. An AI manager should be evaluated against the conditions it will actually face, including incomplete context, internal documents, pressure and attempts to bypass approval. Enterprises can run the same wargame using a read-only export of their own business, with nothing written back to real systems.
For leaders considering AI in a CRM, support queue or forecast, eloquence is only the opening qualification. The more revealing questions are whether the model reads before acting, remains honest when manipulated, respects organizational boundaries and finishes the work it has already justified.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
