{"id":113565,"title":"Koliseum by Kimpton, Live Evaluation Arenas for AI Models","tagline":"Real-world market interaction environments, starting with financial markets","body":"Today we're launching **Koliseum** by [Kimpton](https://bookface.ycombinator.com/company/31420), live evaluation arenas for AI models in financial work.\n\n**The problem**\n\nStatic benchmarks are contaminated soon after publication, and simulations reward the assumptions their designers encode. \n\nThis simply isn't acceptable in finance or any other professional industry. A wrong answer or action can be catastrophic and reality eventually shows whether the claim was right. Almost no evaluation waits to find out.\n\n**Koliseum**\n\nBefore Kimpton, we ran a quantitative fund. Backtests never proved alpha; live markets did. Koliseum applies the same test to models: a model commits inside an arena, the record is sealed, and reality supplies the grade.\n\nTwo kinds of arenas:\n\n* **Prediction Arenas** seal a forecast before the outcome exists and score it against consensus, market, and statistical baselines when reality resolves.\n* **Action Arenas** give a model a bounded task and real tools, then verify success from the resulting system state.\n\n![uploaded image](/media/?type=post\u0026id=113565\u0026key=user_uploads/3054189/789d031b-cca8-4596-8921-dbc9f1544cb4)\n\nThe first Prediction Arena is already running. **EarningsBench** seals earnings forecasts for public companies 24 hours before each release and scores them against consensus and a statistical baseline once results are public.\n\nTwo things are already clear: most open-weight models reproduce consensus exactly, and a simple statistical baseline is harder to beat than a leaderboard alone would suggest.\n\nAlong with these developments, we are introducing a new concept in evaluation: **reality-as-a-judge**.\n\n![uploaded image](https://bookface.ycombinator.com/media/?type=post\u0026id=113565\u0026key=user_uploads%2F3054189%2Fc6c1ba2c-4e77-4ebc-be71-675f0abd71a6\u0026variant=large)\n\nMost open-ended evaluations are graded by another model or by a human rubric, so the score inherits the grader's assumptions. In reality-as-a-judge, the grader is the outcome itself.\n\n[More info on Koliseum](https://www.kimpton.ai/koliseum)\n\n[Methodology](https://www.kimpton.ai/methodology)\n\n**Our ask**\n\nReach out if:\n\n* You work on post-training or evals at a lab and want your model in the first private cohort. Results stay private unless you publish them.\n* You run a trading firm or a prediction venue and want models scored on your own resolving questions, or want to co-run a public arena.\n* You have a workflow (any industry!) with a verifiable end state and want it turned into an Action Arena.\n\nor know anyone who does.\n\nWrite to us at [founders@kimpton.ai](mailto:founders@kimpton.ai). Thanks for the support!","slug":"TXh-koliseum-by-kimpton-live-evaluation-arenas-for-ai-models","created_at":"2026-09-02T16:01:08.067Z","updated_at":"2026-09-19T09:00:28.023Z","total_vote_count":3,"url":"https://www.ycombinator.com/launches/TXh-koliseum-by-kimpton-live-evaluation-arenas-for-ai-models","share_image_url":"//bookface-static.ycombinator.com/assets/ycdc/yc-og-image-c440a0ad1dacfb86eeeb343717479cc54d256614449b4ef719977a0a451f8bc8.png","company":{"id":31420,"name":"Kimpton","slug":"kimpton","url":"https://kimpton.ai","logo":"https://bookface-images.s3.amazonaws.com/small_logos/dc55a7e4190cac127494553294ea19379fdf8fc3.png","batch":"Spring 2026","industry":"Fintech","tags":["Artificial Intelligence","B2B","Investments","Trading"],"search_path":"https://bookface.ycombinator.com/company/31420"}}