TL;DR: OneTriangle is building the cheapest, fastest inference. We cut inference prefill costs by 20% and time-to-first-token by 40%, letting you get large-model quality at small-model cost.

Video: https://youtu.be/HKe2EZUdNHo

Team

We’re Hannah (CEO) and Medha (CTO), MIT CSAIL grads building with a team of 4 MIT engineers spanning ex-DeepMind and Jane Street, Physics and Astronomy Olympiad medalists, and NeurIPS/ICML authors. 

The Problem

Inference is expensive. 

• You’re forced into a bad tradeoff. You need the decoding power of large models, but you have the budget for small ones.

• Prefill is the hidden tax. For long-context and agentic workloads, prefill dominates cost and latency, and everyone just eats it.

Our Solution

OneTriangle transfers the KV cache between models, something that has never been done in production. We prefill on a small model, strip the positional encoding, map the cache into the large model’s space, and restore relative importances. The result:

• 20% lower prefill costs

• 40% faster TTFT

• Large-model quality at small-model cost

We beat out NVIDIA’s KV cache transfer paper by getting a lower KL divergence score in 2 weeks.

No API changes, no quality cliff, and we’re upstreaming it into vLLM so the fastest path to cheap inference is the infrastructure you already run.

Why Now

Inference has surpassed training as AI’s largest compute cost, and it’s accelerating with demand for agents that generate 10x the tokens a chat session does. Inference already eats up to 90% of total compute cost, over $600 billion is going into AI data centers this year alone. Every team pushing the frontier is bottlenecked by the same thing: serving costs.

The Future

Our bet is that model-mixing becomes the default serving pattern: small models handle prefill and routine tokens, large models handle the hard ones, and the cache moves freely between them. Whoever owns the transfer layer owns the economics of inference. We are the future of lightweight, fast inference.