
TL;DR: Everyone is trying to use AI for drug discovery (finding molecules), but 90% of actual pharma time and money gets burned in drug development - people digging through messy SharePoint, Veeva, fixing conflicting batch records (often scanned), updating clinical protocols, and answering FDA audit letters with 20 redline versions and siloed email threads with colleagues.
We built Biopharma Bench 0.1: 12 complete, reconstructed drug companies, 71 multi-hour tasks, and 752 ground-truth checks audited by industry veterans to see what happens when you actually put frontier AI into an employee’s seat.
One interesting finding: DeepSeek V4.1 Flash matched GPT-6 Sol on the actual work (57.0% vs 54.3%) at a sixth of the cost ($0.11 vs $0.64 per task). Meanwhile Claude Opus 5, #2 overall at 63.8%, didn't fully pass a single task.
Check out the interactive traces & leaderboard: https://eval.raycaster.ai/benchmarks/biopharma-bench
Read the deep-dive blog post: https://www.raycaster.ai/blog/biopharma-bench
Join the discussion on Hacker News: https://news.ycombinator.com/item?id=49845891
X: https://x.com/dubrovnik07/status/2103543413933064530
LinkedIn: https://lnkd.in/p/g_PD7jYr
Hey everyone! We’re the team behind Raycaster (https://raycaster.ai).
We first started building AI agents for regulated industries like biopharma because we see the inevitable wave of knowledge work getting handled by AI systems with human oversight (definitely not fear-mongering, but in a super exciting way!).
Over the last few months, our major focus has been helping frontier AI labs globally hillclimb capabilities in underserved industries such as biopharma, energy, data centers, among others.
Very quickly, we noticed something broken: almost all bio AI benchmarks test multiple-choice questions or molecular physics. Nobody was testing whether an AI agent could actually do the drug development work, open a folder of 50 disorganized PDFs and spreadsheets, find where two lab reports contradict each other, and draft an FDA-defensible document.
So, we spent the last few months working directly with former pharma directors and regulatory specialists to build an evaluation that tests what real work actually looks like.
It takes 8–10 years and over $1B to get a drug to patients. Most of that delay is paperwork and coordination mistakes:
When people play with LLMs for this, they fall into what we call the fluency trap: Frontier models write seemingly true prose. If you just read the output, the AI sounds like a 20-year VP of Regulatory Affairs who knows everything.
But when you actually grade the facts against ground truth, the model cited an outdated 2023 draft instead of the approved amendment, or hallucinated a purity threshold. In biopharma, being 95% right with one made-up number is an instant rejection.
We didn’t give the models clean, cherry-picked prompts. We dropped them into messy environments:
We ran frontier reasoning models across multiple popular harnesses (Codex, Pi, Claude Code). Here are the 3 biggest takeaways: