Add blog post announcing the Agent LLM Traces dataset (10K agent runs, 6 domains) - #22
Add blog post announcing the Agent LLM Traces dataset (10K agent runs, 6 domains)#22elronbandel wants to merge 6 commits into
Conversation
Announces the Exgentic Agent LLM Traces dataset release (10K real agent runs across six domains, one standard format). Adds the blog source and the blog card on the home page.
elronbandel
changed the title
- Add "A closer look" section with three tables: benchmark complexity, per-model token cost, and per-agent prefix retention - Add "Inference benchmarking: an ongoing example" section covering inference-perf (kubernetes-sigs/inference-perf) and its OTel trace replay - Add load-testing bullet to "What you can do with it" - Fix "sessions" → "runs" throughout - Add inference-perf source link Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
||
| AI agents now book our travel, write our code, and answer our customers. Two can finish the same task and look identical on paper — one taking three clean steps, the other opening a file it shouldn't, crashing a service, and recovering by luck. We already know whether they succeed and what they cost. It's time we saw how they get there. | ||
|
|
||
| Today we're releasing [Exgentic Agent LLM Traces](https://huggingface.co/datasets/Exgentic/agent-llm-traces-v2): 10K real agent runs, across five leading models and six kinds of task, with every model call preserved in one standard format. It's the part of the run that normally gets thrown away. |
There was a problem hiding this comment.
Doesn't make clear we test different agents, not just models × tasks. The agent-design axis (each run through up to five agent designs) is only revealed later in "What's in it". Consider surfacing it here as a third dimension, e.g.: "10K real agent runs — five leading models, five agent designs, six kinds of task — with every model call preserved in one standard format."
|
|
||
| Today we're releasing [Exgentic Agent LLM Traces](https://huggingface.co/datasets/Exgentic/agent-llm-traces-v2): 10K real agent runs, across five leading models and six kinds of task, with every model call preserved in one standard format. It's the part of the run that normally gets thrown away. | ||
|
|
||
| > We've always known whether agents succeed or failed. Now we can see how. |
There was a problem hiding this comment.
Reconsider wording — we release the data, we're not doing the analysis/"seeing" ourselves. Reframe to the enabling act, e.g. "We've always known whether agents succeed. Now the record of how is public."
|
|
||
| That leaves a gap. We study *models* on mountains of public data. We study *agents* — the multi-step, tool-using, failing-and-recovering systems people actually deploy — on almost nothing public. The behavior that decides whether an agent is worth shipping lives in the full record of what it did, and that record rarely leaves the lab that produced it. | ||
|
|
||
| The gap exists for a concrete reason: producing real agent runs at scale is slow, costly, and fragmented. It means running the strongest available models across many different environments, thousands of times over, and paying for every token they generate. And even once you have the runs, every tool records them in its own shape, so what data exists stays private and mutually incompatible. There has been no large, uniform, public record of what capable agents actually do. |
There was a problem hiding this comment.
"the strongest available models" → "frontier models"? (matches wording used later, avoids repetition).
Also: our strongest, most specific claim — the shipping-relevant behavior lives in the full record, which is normally discarded — is scattered across the opener and here but never stated crisply. Consider one tight paragraph early saying what a single trace is (the full call sequence: inputs, tools, outputs, errors, per step) and why the sequence, not the final number, is the unit of value nobody else publishes. That is the whole "why traces" thesis; right now it's implied, not stated.
There was a problem hiding this comment.
rephrased the sentence about the available models + added a new paragraph that describes the trace, as you suggested.
|
|
||
| - **Train and fine-tune agents on real runs.** Long, tool-using runs from strong models are the kind of data that is hard to obtain and expensive to generate. | ||
| - **Build tools that evaluate agents, using real failures.** Study where agents actually break instead of on made-up examples. | ||
| - **Replay a recorded moment against a new model.** Feed the exact situation an agent faced to a different model and compare, without re-running the whole evaluation. |
There was a problem hiding this comment.
"recorded moment" is unclear. Make the unit concrete, e.g.: "Replay a recorded situation against a new model. Take the exact context an agent faced at one step — its history, available tools, and state — and feed it to a different model to see how it would have acted, without re-running the whole benchmark."
| - **Build tools that evaluate agents, using real failures.** Study where agents actually break instead of on made-up examples. | ||
| - **Replay a recorded moment against a new model.** Feed the exact situation an agent faced to a different model and compare, without re-running the whole evaluation. | ||
| - **Debug your own agent against a reference set.** Compare its behavior to how the strongest models handle the same kinds of tasks. | ||
| - **Load-test inference infrastructure against realistic traffic.** Standard load tests send independent requests; real agent runs are dependency chains where the output of one call shapes the input of the next. Because these traces are already in OTel format, they work directly with tools that replay recorded workloads against live endpoints. |
There was a problem hiding this comment.
The point here isn't to explain what's currently supported (nor to introduce OTel — it's introduced later in "What's in it"). Focus on the capability: replay these traces to evaluate an inference platform under real agentic load. Suggested: "Evaluate inference infrastructure under realistic load. Real agent runs are dependency chains — one call's output shapes the next — not the independent requests standard load tests send. These runs can be replayed against a live endpoint to measure how a serving configuration behaves under true agentic traffic."
Also (list-level): consider reordering — if the thesis is "study how agents behave," lead with the behavioral uses (debug against a reference set, study real failures, replay a situation) and move train/fine-tune lower.
| | Customer service | τ²-bench Retail | 1.8K | 198K | | ||
| | Customer service | τ²-bench Airline | 1K | 255K | | ||
|
|
||
| Five frontier models are represented in comparable volume — DeepSeek-V3.2 (2.3K runs), Kimi-K2.5 (2.3K), GPT-5.2 (2.1K), Claude Opus 4.5 (1.9K), and Gemini 3 Pro (1.4K) — each run through up to five different agent designs. Because the format is identical across all of them, comparing how two models handled the same task, or how one model's behavior shifts between agent designs, is a filter, not a data-cleaning project. |
There was a problem hiding this comment.
Last sentence drifts toward comparing scores/tokens of implementations — the focus should stay on the traces. Refocus on why the uniform format is the contribution, e.g.: "Because the format is identical across all of them, the runs line up directly — the same task across models, or one model across agent designs, sits side by side with no per-tool reconciliation."
Also (E): "Avg tokens / run" differs between the domain table above (SWE-bench 2.1M) and the per-model table below (Opus 2.4M). Fine — different groupings — but add a clause noting the tables group by different axes so readers don't trip.
|
|
||
| ## A closer look | ||
|
|
||
| Three views of the data, each answering a different question. |
There was a problem hiding this comment.
We're not publishing analysis here — just showing the kinds of insights the traces enable. Frame each table as a question the data lets you answer, not a conclusion we're drawing. Lead-in e.g.: "These aren't our findings — they're examples of what the traces let you ask."
There was a problem hiding this comment.
I've made some changes here, I'm not sure its what you meant.
|
|
||
| One use of this data is already underway. A team working on LLM inference infrastructure is using these traces to benchmark serving configurations under realistic agentic load — the kind of workload that synthetic benchmarks systematically miss. | ||
|
|
||
| The tool at the center of that effort is [inference-perf](https://github.com/kubernetes-sigs/inference-perf), a Kubernetes SIG benchmarking tool that replays OTel traces as directed acyclic graphs. Rather than sending independent requests at a fixed rate, it reconstructs the dependency structure of a real agent run — the order and dependencies between calls, and how context accumulated across them — and drives that structure against the inference endpoint under test. Downstream calls receive the actual live-generated output from upstream calls, so KV-cache behavior and context growth stay realistic throughout the replay. |
There was a problem hiding this comment.
Too tool-first — two of four paras describe inference-perf. Compress the mechanics to 1–2 sentences; spend the space on why these traces make the benchmark real (real dependency structure, context growth, KV-cache reuse that synthetic generators can't reproduce).
Also: "live-generated output from upstream" is subtle and under-explained — clarify that structure+inputs come from the recorded run while outputs are regenerated live by the endpoint under test; that's what keeps cache/context honest vs. replaying frozen text.
|
|
||
| The tool at the center of that effort is [inference-perf](https://github.com/kubernetes-sigs/inference-perf), a Kubernetes SIG benchmarking tool that replays OTel traces as directed acyclic graphs. Rather than sending independent requests at a fixed rate, it reconstructs the dependency structure of a real agent run — the order and dependencies between calls, and how context accumulated across them — and drives that structure against the inference endpoint under test. Downstream calls receive the actual live-generated output from upstream calls, so KV-cache behavior and context growth stay realistic throughout the replay. | ||
|
|
||
| Because the Exgentic traces are already in OTel format, they plug into this workflow without any conversion. The result is end-to-end run latency measured against real agent traffic, not a synthetic proxy for it — exactly the signal needed to evaluate whether a serving configuration change is worth shipping. |
There was a problem hiding this comment.
"end-to-end run latency" undersells it — earlier we argue agentic eval is more than latency. Broaden to "latency, cache hit rates, and cost under real load," or write "for example, end-to-end latency" so it doesn't read as the ceiling.
|
|
||
| The runs are public and ready to load — one open format, 236 MB, [on Hugging Face](https://huggingface.co/datasets/Exgentic/agent-llm-traces-v2). Whatever you are building agents to do, or worried they might do, you can now start from what they actually did instead of a single line of results. Open a few runs and follow how they get there. | ||
|
|
||
| > The how was always there, inside every run. Now anyone can read it. |
There was a problem hiding this comment.
Consider adding a "contribute your own runs" section — run Exgentic on your tasks/models/agents of interest and get traces in the same format — which turns a static release into a growing shared record and reinforces our "value compounds as more are added" line.
There was a problem hiding this comment.
the thing is - the traces generate by Exgentic v2 are not in the same format that we released. That is, they are a kind of OTEL (this is a broad format) but are not similar to our OTELs. There are 2 main differences - some meta data that we added on the top layer (harness, model, max tokens) is not available in these traces. Also (and more importantly), inner structure of input/output messages is not similar to the structure we have in our OTEL (and is defined in the gen.ai spec). We realized this gap last week, and still considering what to do, so I'm not sure it will be accurate to write - run exgentic and get traces in the same format.
…, tighten inference section, fluency fixes
|
|
||
| Agents are moving into real work: writing code, handling customer support, doing research, automating tasks across hundreds of apps. Benchmarks are how we study that work before it reaches the people who depend on it — each one a controlled mirror of a real job, a stand-in for the messy thing we actually care about. So the question is never really how an agent scores on the mirror. It is how the agent will behave doing the real thing: what it costs, where it gives up, whether it recovers when something breaks. And almost everything we know about that comes from a single number per run, because that number is usually all anyone keeps. | ||
|
|
||
| That leaves a gap. We study *models* on mountains of public data. We study *agents* — the multi-step, tool-using, failing-and-recovering systems people actually deploy — on almost nothing public. The behavior that decides whether an agent is worth shipping lives in the full record of what it did, and that record rarely leaves the lab that produced it. |
There was a problem hiding this comment.
Reconsider the "study models on mountains of public data" / "study agents ... on almost nothing public" parallel — it is a bit off. People do study agents on public data (benchmarks, leaderboards); the gap is that this data is score-only, not behavior-rich. As written, it also slightly undercuts the post, since the fix being announced here is itself the first time behavior-level data becomes public. Suggested rewording:
That leaves a gap. We study models on mountains of public data. We study agents — the multi-step, tool-using, failing-and-recovering systems people actually deploy — mostly through a single score per run, because that's almost all that's public. The behavior that decides whether an agent is worth shipping lives in the full record of what it did, and that record rarely leaves the lab that produced it.
This keeps the models/agents contrast but relocates the gap from "public data exists vs. doesn't" to "public data is score-only vs. behavior-rich," which also lines up better with the score-vs-sequence framing used elsewhere in the post.
4 participants
Adds a new blog post announcing the Exgentic Agent LLM Traces dataset release.
What's in it
blogs/agent-llm-traces-v2/blog.md(slugagent-llm-traces-v2, dated 2026-07-01, ~8 min read).index.html).The post
An understated launch in the house research-blog voice. Spine: agents are moving into real work, and this is the first large, uniform, public record of what they actually do — 10K real runs across five leading models and six domains (software engineering, deep research, personal-task automation, customer service, technical support), every model call preserved in one open format (OpenTelemetry GenAI conventions).
Includes a composition table, an honest "what it doesn't cover" section, and links to the dataset, the full corpus, and the methodology.
Notes
blog/output is gitignored and produced by CI, so only the source and theindex.htmlcard are committed.