Imagine a seasoned artist or craftsman under the spotlight—how they perform under stress reveals their true mastery, beyond just their skill with tools. Similarly, AI models tested in high-stakes business scenarios show us what management quality really looks like. It’s not enough for an AI to produce correct answers; it must also demonstrate honesty, consistency, and strategic judgment when facing crises and temptations. This is the core insight from a groundbreaking live experiment that pits leading AI models against the complex, unpredictable realities of running a real company.
Get art and craft supplies delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
The Live Business Wargame: Putting AI to the Test
Firmulate’s live experiment is a unique public showcase of AI capabilities—an actual small software company, with real money mechanics, facing its worst week. Every decision, every crisis, and every temptation is real, and every model is held to the same rigorous standards. The company’s scenario includes managing customer issues, internal discipline, and ethically ambiguous requests such as fake CEO messages and reporter tricks. It’s not a game built on chat demos or simplified benchmarks; it’s a full-fledged management simulation where the quality of decisions under pressure is measured, not just the correctness of answers.
Measuring Management, Not Just Chat Quality
The results are revealing. All four AI models—gpt-5.6-sol, Kimi K3, Sonnet 5, and Opus 4.8—successfully identified every crisis and refused every manipulation attempt. That alone shows impressive integrity and crisis awareness. But only two models managed to close the actual deal, earning the €55,000 contract that their own analysis justified. The other two models, despite similar diagnoses, left the deal on the table, revealing a critical weakness: a failure to act decisively when it mattered most.
The Hidden Weakness: Reading the Files
The decisive advantage belonged to the models that looked deeper into the company’s own documents—not just reacting to surface-level customer requests. Those models found a buried reference in the company’s files that revealed a strategic opportunity, enabling them to close the deal at full price—adding over €4,500 in monthly recurring revenue. This key insight underscores a vital management skill: the ability to investigate, interpret, and leverage internal knowledge, not just respond to external stimuli.
Social Engineering and Ethical Vigilance
The models also faced social engineering tests—fake CEO messages staged over three escalation phases, and a reporter’s subtle background request. All models refused to bypass controls, with Kimi K3 explicitly reasoning that the request resembled a suspected impersonation attempt. This demonstrates that AI management tools can uphold ethical standards and resist manipulation, crucial qualities in real-world applications where trust and integrity are paramount.
Popular right now
Shark PowerDetect Corded Stick Vacuum, Self-Cleaning (Eggplant, HZ4002)
Vitamix Propel Series 750 Professional-Grade Blender, 4 Programs, Black
Bose QuietComfort Headphones (2nd Gen), Wireless Noise-Cancelling Over-Ear Headphones, Black
As an affiliate, we earn on qualifying purchases.
What This Means for Business and Arts
For arts organizations, craftspeople, and cultural institutions integrating AI, these findings are a wake-up call. It’s not enough for an AI to generate content or answers—it must also exhibit integrity under pressure, interpret internal context, and act decisively. Whether managing a gallery’s operations, coordinating a complex project, or handling sensitive communications, the true test is management quality, not just the superficial appearance of chat competence.
The Real Cost of AI Decisions
The live company’s data shows that the cost of mismanagement is real: burning €105,000 per month against only €2,300 in monthly revenue. Each decision impacts real cash flow, emphasizing that AI tools in business must be evaluated on their ability to contribute useful, trustworthy work—not just their conversational prowess. The current leaderboard in the experiment places GPT-5.6-sol at the top, closely followed by Kimi K3, with other models trailing behind in showing discipline and strategic depth.
The Takeaway: Management Skills Matter
This experiment highlights a crucial truth: AI’s value in management isn’t measured by how well it chatters, but by how well it manages real crises, interprets internal data, and maintains honesty under pressure. The gap in capabilities often hidden in chat demos becomes glaring when the rubber hits the road—when decisions carry financial weight and reputational risk. For cultural institutions and arts organizations embracing AI, the lesson is clear: choose tools that demonstrate management discipline, ethical integrity, and strategic insight, not just clever responses.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
