
TL;DR: OneTriangle is building the cheapest, fastest inference. We cut inference prefill costs by 20% and time-to-first-token by 40%, letting you get large-model quality at small-model cost.
Video: https://youtu.be/HKe2EZUdNHo
Team
We’re Hannah (CEO) and Medha (CTO), MIT CSAIL grads building with a team of 4 MIT engineers spanning ex-DeepMind and Jane Street, Physics and Astronomy Olympiad medalists, and NeurIPS/ICML authors.
The Problem
Inference is expensive.
• You’re forced into a bad tradeoff. You need the decoding power of large models, but you have the budget for small ones.
• Prefill is the hidden tax. For long-context and agentic workloads, prefill dominates cost and latency, and everyone just eats it.
Our Solution
OneTriangle transfers the KV cache between models, something that has never been done in production. We prefill on a small model, strip the positional encoding, map the cache into the large model’s space, and restore relative importances. The result:
• 20% lower prefill costs
• 40% faster TTFT
• Large-model quality at small-model cost
We beat out NVIDIA’s KV cache transfer paper by getting a lower KL divergence score in 2 weeks.
No API changes, no quality cliff, and we’re upstreaming it into vLLM so the fastest path to cheap inference is the infrastructure you already run.
Why Now
Inference has surpassed training as AI’s largest compute cost, and it’s accelerating with demand for agents that generate 10x the tokens a chat session does. Inference already eats up to 90% of total compute cost, over $600 billion is going into AI data centers this year alone. Every team pushing the frontier is bottlenecked by the same thing: serving costs.
The Future
Our bet is that model-mixing becomes the default serving pattern: small models handle prefill and routine tokens, large models handle the hard ones, and the cache moves freely between them. Whoever owns the transfer layer owns the economics of inference. We are the future of lightweight, fast inference.
OneTriangle is built by a group of five MIT engineers and researchers with experience at Google DeepMind, Jane Street, SpaceX, MIT Lincoln Laboratory, and CSAIL. The team includes International Physics and Astronomy Olympiad medalists and has published at NeurIPS and ICML.
Advised by vLLM lead at Red Hat AI, Robert Shaw, and a Google Distinguished Engineer in AI Infrastructure.
Make computed context transferable across the open-weight ecosystem. We build infrastructure that transfers state between models, so teams can use the best model for each task without paying the prefill cost twice. This way, we can provide inference at a fraction of the cost and TTFT.