{"id":116908,"title":"Biopharma Bench 0.1 – Evaluating AI on Autonomous Drug Development","tagline":"Frontier agents struggle in drug development; open models crush on cost","body":"**TL;DR:** Everyone is trying to use AI for drug _discovery_ (finding molecules), but 90% of actual pharma time and money gets burned in drug _development_ - people digging through messy SharePoint, Veeva, fixing conflicting batch records (often scanned), updating clinical protocols, and answering FDA audit letters with 20 redline versions and siloed email threads with colleagues.\n\n![uploaded image](/media/?type=post&id=116908&key=user_uploads/299659/82cec064-a13e-4bfc-b527-31b46388e50c)\n\nWe built **Biopharma Bench 0.1**: 12 complete, reconstructed drug companies, 71 multi-hour tasks, and 752 ground-truth checks audited by industry veterans to see what happens when you actually put frontier AI into an employee’s seat.\n\n> One interesting finding: DeepSeek V4.1 Flash matched GPT-6 Sol on the actual work (57.0% vs 54.3%) at a sixth of the cost ($0.11 vs $0.64 per task). Meanwhile Claude Opus 5, #2 overall at 63.8%, didn't fully pass a single task.\n\nCheck out the interactive traces &amp; leaderboard: <https://eval.raycaster.ai/benchmarks/biopharma-bench>\\\nRead the deep-dive blog post: <https://www.raycaster.ai/blog/biopharma-bench>\\\nJoin the discussion on Hacker News: <https://news.ycombinator.com/item?id=49845891>\\\nX: <https://x.com/dubrovnik07/status/2103543413933064530>\\\nLinkedIn: <https://lnkd.in/p/g_PD7jYr>\n\n### Backstory\n\nHey everyone! We’re the team behind Raycaster (<https://raycaster.ai>).\n\nWe first started building AI agents for regulated industries like biopharma because we see the inevitable wave of knowledge work getting handled by AI systems with human oversight (definitely not fear-mongering, but in a super exciting way!).\\\n\\\nOver the last few months, our major focus has been helping frontier AI labs globally hillclimb capabilities in underserved industries such as biopharma, energy, data centers, among others.\n\nVery quickly, we noticed something broken: almost all bio AI benchmarks test multiple-choice questions or molecular physics. Nobody was testing whether an AI agent could actually do the drug development work, open a folder of 50 disorganized PDFs and spreadsheets, find where two lab reports contradict each other, and draft an FDA-defensible document.\n\nSo, we spent the last few months working directly with former pharma directors and regulatory specialists to build an evaluation that tests what real work actually looks like.\n\n---\n\n### The Problem: The \"Fluency Trap\"\n\nIt takes 8–10 years and over $1B to get a drug to patients. Most of that delay is paperwork and coordination mistakes:\n\n* Protocol Drift: \\~75% of clinical trials have to amend their protocols mid-flight. Each amendment costs hundreds of thousands of dollars and months of lost time.\n* Messy Records: A tiny discrepancy between a stability test memo from last Tuesday and an executed batch record can trigger FDA warning letters (Form 483s) or clinical holds.\n\nWhen people play with LLMs for this, they fall into what we call the fluency trap: Frontier models write seemingly true prose. If you just read the output, the AI sounds like a 20-year VP of Regulatory Affairs who knows everything.\n\nBut when you actually grade the facts against ground truth, the model cited an outdated 2023 draft instead of the approved amendment, or hallucinated a purity threshold. In biopharma, being 95% right with one made-up number is an instant rejection.\n\n![uploaded image](/media/?type=post&id=116908&key=user_uploads/299659/46c7d62a-3842-4e51-8680-ff7f80f37255)\n\n### How Biopharma Bench Works\n\nWe didn’t give the models clean, cherry-picked prompts. We dropped them into messy environments:\n\n* 12 Companies: Real reconstructed company drives packed with thousands of messy files and software states - unredacted study reports, supplier quotes, assay validations, and random internal memos.\n* 71 Long-Horizon Tasks: Multi-hour workflows across Preclinical, CMC/Manufacturing, Clinical Ops, and Regulatory Affairs (like investigating an out-of-spec stability result or drafting a protocol amendment).\n* 752 Ground-Truth Checks: Graded against falsifiable rubrics created using real documents along with former pharma directors.\n\n![uploaded image](/media/?type=post&id=116908&key=user_uploads/299659/36d4376d-59ed-4eba-9ede-d93930299e98)\n\n---\n\n### What Shocked Us from the Runs\n\nWe ran frontier reasoning models across multiple popular harnesses (Codex, Pi, Claude Code). Here are the 3 biggest takeaways:\n\n1. **DeepSeek matched GPT-6 Sol at 1/6 the cost.** \n2. **Almost nobody finishes a full task cleanly.** Top systems get 45–70% of checks, but full passes are rare: GPT-6 Astra 8/71, Grok 4.6 2/71, Kimi K3 and GPT-5.6 Sol 1 each. Claude Opus 5, DeepSeek, GPT-6 Sol, GLM, and Gemini got 0.\n3. **The writing is fine; the sourcing isn't.** In one public task, Claude Opus 5 did everything right on the surface: it navigated the folders, read scanned protocols page by page, and wrote a clean brief. But a secondary report claimed a \"protocol-defined 30-minute window\" that doesn't exist in the controlling protocol. Opus repeated it as fact in the brief for the inspector. An experienced engineer would have flagged it.\n\n---\n\n### See the Traces\n\n![uploaded image](/media/?type=post&id=116908&key=user_uploads/299659/10059154-ce98-41cb-b0ed-f23ac8942580)\n\n---\n\n### How We Can Help\n\n* Naturally as a data / research lab, we have tons of insights into benchmark design - we recently became a proud sponsor of [evalscience.org](https://evalscience.org), a 501(c)(3) nonprofit focused on scientific agent evaluation methods and independent benchmark verification. If there’s interest, happy to share more regularly on BookFace\n* We work with / fund academics and researchers and have landed joint papers in NeurIPS, NAACL and all the big conferences. Bring your idea to us and see what can come out of it\n* Finally, if you’re a neo-lab reading this, we have lots of ready to license data, environments and evals on some of the most interesting yet underserved industries 😄 , talk to us at [team@raycaster.ai](mailto:team@raycaster.ai)","slug":"UPc-biopharma-bench-0-1-evaluating-ai-on-autonomous-drug-development","created_at":"2026-09-25T17:16:44.511Z","updated_at":"2026-11-18T05:11:02.820Z","total_vote_count":15,"url":"https://www.ycombinator.com/launches/UPc-biopharma-bench-0-1-evaluating-ai-on-autonomous-drug-development","share_image_url":"https://www.ycombinator.com/media/?type=post&id=116908&key=user_uploads/299659/82cec064-a13e-4bfc-b527-31b46388e50c","company":{"id":30039,"name":"Raycaster","slug":"raycaster","url":"https://eval.raycaster.ai","logo":"https://bookface-images.s3.amazonaws.com/small_logos/8e3854e63389e5cabd44e4b6563fefc3bd539147.png","batch":"Fall 2024","industry":"Healthcare","tags":["Biotech","Data Labeling","Enterprise Software","AI"],"search_path":"https://bookface.ycombinator.com/company/30039"}}