Your agent logs aren’t just for debugging. They’re your training data.
Hey YC! We’re Shubham and Parth, childhood friends and founders of Agnost AI.

TL;DR: Agnost AI uses your agent’s production traces to train a model specialized for your exact workload. Our first custom model for our customer delivered them a:

• 22.9% higher task success,
• 90.2% lower median latency, and
• 94.5% lower cost than Opus 4.8.

Demo: https://youtu.be/R3NyWNYXHu4

What we found

A few weeks ago, we launched Agnost AI as product analytics for AI agents. We read conversations between users and agents to uncover failures, behavior drift, hallucinated links, frustration, and churn signals.

We expected to find bugs. Surprisingly, we also found a lot of infrastructure waste. Many customer-facing agents use frontier models for narrow, repetitive jobs:

• Route a request to the correct tool
• Have a narrow system prompt with the given inputs
• Retrieve the relevant data from internal tool calls
• Parse & format the final response

The underlying frontier model can write code, solve advanced mathematics, understand biology, and reason across hundreds of domains. Your agent uses a small, repeatable slice of that capability while paying frontier-model prices for every request.

We trained a specialist

We used historical production traces to train our first workload-specific model for an early customer: agnost-<redacted>-0.1

We evaluated it against Opus 4.8 on 780 held-out customer traces for an Identifier Extraction Agent. The results:

• Task success: 71.5% → 87.9%, 22.9% relative improvement
• Median latency: 4.80s → 0.47s, 90.2% lower
• Tail latency: 11.20s → 1.35s, 87.9% lower
• Cost per 1,000 completed tasks: $42.00 → $2.30, 94.5% lower

This is one model for one workload, not a claim that specialist models should replace frontier models everywhere.

Frontier models remain the right choice for open-ended work. But once an agent’s production job becomes predictable, paying for general intelligence on every request stops making sense.

How it works

1. Send your existing traces to Agnost through OpenTelemetry or our SDK.
2. We identify the recurring workflows your agent performs.
3. We create an eval set from held-out production traces.
4. We train a specialist model on the historical workload.
5. We benchmark it against your current model on success, latency, and cost.

You only consider switching when the specialist wins on your own workload. Your logs go from debugging data to evaluation data and finally to training data recursively.

Who we are

We’re childhood friends, eight years and counting.

• Shubham: Youngest engineer on Cisco’s analytics team, first hire at Formbricks, and contributor to Bitcoin, OWASP, and the Linux Foundation.
• Parth: IIT Madras CS, ranked 159 out of 1.5 million. Built Spark pipelines at Microsoft and led GPU technology at Infurnia.

Our ask

If you have meaningful daily traffic, send us an anonymized trace export. We’ll analyze the workload, build a held-out evaluation set, and tell you whether a specialist model is worth training. If it is, we’ll benchmark it against your current frontier model.

Email me at shubham@agnost.ai or book a call at call.agnost.ai

More about us at: agnost.ai