From 43e2b5e63e227557378c6671b3ecea7410826f5d Mon Sep 17 00:00:00 2001 From: Elron Bandel Date: Wed, 1 Jul 2026 19:32:11 +0300 Subject: [PATCH 1/6] Add blog post: What Frontier Agents Actually Do Announces the Exgentic Agent LLM Traces dataset release (10K real agent runs across six domains, one standard format). Adds the blog source and the blog card on the home page. --- blogs/agent-llm-traces-v2/blog.md | 74 +++++++++++++++++++++++++++++++ index.html | 10 +++++ 2 files changed, 84 insertions(+) create mode 100644 blogs/agent-llm-traces-v2/blog.md diff --git a/blogs/agent-llm-traces-v2/blog.md b/blogs/agent-llm-traces-v2/blog.md new file mode 100644 index 0000000..879ee30 --- /dev/null +++ b/blogs/agent-llm-traces-v2/blog.md @@ -0,0 +1,74 @@ +--- +title: "What Frontier Agents Actually Do" +subtitle: "10K real agent runs, in one standard format, so anyone can study how agents behave and not just what they scored." +author: Elron Bandel +date: 2026-07-01 +reading_time: 8 min +slug: agent-llm-traces-v2 +--- + +AI agents now book our travel, write our code, and answer our customers. Two can finish the same task and look identical on paper — one taking three clean steps, the other opening a file it shouldn't, crashing a service, and recovering by luck. We already know whether they succeed and what they cost. It's time we saw how they get there. + +Today we're releasing [Exgentic Agent LLM Traces](https://huggingface.co/datasets/Exgentic/agent-llm-traces-v2): 10K real agent runs, across five leading models and six kinds of task, with every model call preserved in one standard format. It's the part of the run that normally gets thrown away. + +> We've always known whether agents succeed or failed. Now we can see how. + +## Why this matters + +Agents are moving into real work: writing code, handling customer support, doing research, automating tasks across hundreds of apps. Benchmarks are how we study that work before it reaches the people who depend on it — each one a controlled mirror of a real job, a stand-in for the messy thing we actually care about. So the question is never really how an agent scores on the mirror. It is how the agent will behave doing the real thing: what it costs, where it gives up, whether it recovers when something breaks. And almost everything we know about that comes from a single number per run, because that number is usually all anyone keeps. + +That leaves a gap. We study *models* on mountains of public data. We study *agents* — the multi-step, tool-using, failing-and-recovering systems people actually deploy — on almost nothing public. The behavior that decides whether an agent is worth shipping lives in the full record of what it did, and that record rarely leaves the lab that produced it. + +The gap exists for a concrete reason: producing real agent runs at scale is slow, costly, and fragmented. It means running the strongest available models across many different environments, thousands of times over, and paying for every token they generate. And even once you have the runs, every tool records them in its own shape, so what data exists stays private and mutually incompatible. There has been no large, uniform, public record of what capable agents actually do. + +> Model behavior is studied everywhere. Agent behavior is barely studied at all, because the data isn't public. + +This release is a piece of that missing record. Real runs, at scale, from frontier models, across the domains agents are deployed in, in one format anyone can load. That combination didn't exist before today. + +## What you can do with it + +Preserving the full run, not just the verdict, is what makes the data useful. Each of these was previously gated behind running the evaluations yourself: + +- **Train and fine-tune agents on real runs.** Long, tool-using runs from strong models are the kind of data that is hard to obtain and expensive to generate. +- **Build tools that evaluate agents, using real failures.** Study where agents actually break instead of on made-up examples. +- **Replay a recorded moment against a new model.** Feed the exact situation an agent faced to a different model and compare, without re-running the whole evaluation. +- **Debug your own agent against a reference set.** Compare its behavior to how the strongest models handle the same kinds of tasks. + +## What's in it + +The dataset is 10K sessions and 242K model calls, filtered down from a full corpus of 10.5K sessions and 626K calls. Roughly two-thirds of the raw calls were removed on purpose: the scaffolding around the agent under test. + +Each session passes through a filter that strips out the supporting machinery — the simulated user, the automatic grader — collapses repeated identical retries, and keeps only the calls the tested model actually made. What remains is the model under test, and nothing the evaluation wrapped around it. The totals for steps, tokens, and cost are recomputed from the calls that survive, so they describe the agent's own footprint and not the machinery around it. + +Every call is written in one open, standard format for recording model calls (the [OpenTelemetry GenAI conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/), which a growing part of the ecosystem already emits). A call carries the models used, the full input and output messages, the tools that were available, token usage, why the call stopped, and an error type when it failed. Each row also carries run-level context — which benchmark, which agent design, the score, the cost, the number of steps — so you can slice the data however you need. Because every run speaks the same format, these traces accumulate into something comparable instead of fragmenting into one dialect per tool — the value compounds as more are added. + +The coverage spans the range of real agent work — software engineering, deep research, personal-task automation across hundreds of apps, and policy-bound customer service and technical support. The token averages alone show how differently these tasks load a model: + +| Domain | Benchmark | Runs | Avg tokens / run | +|--------|-----------|-----:|-----------------:| +| Software engineering | SWE-bench | 2K | 2.1M | +| Deep research | BrowseComp Plus | 1.9K | 1.6M | +| Personal-task automation | AppWorld | 1.5K | 1.7M | +| Technical support | τ²-bench Telecom | 1.8K | 332K | +| Customer service | τ²-bench Retail | 1.8K | 198K | +| Customer service | τ²-bench Airline | 1K | 255K | + +Five frontier models are represented in comparable volume — DeepSeek-V3.2 (2.3K runs), Kimi-K2.5 (2.3K), GPT-5.2 (2.1K), Claude Opus 4.5 (1.9K), and Gemini 3 Pro (1.4K) — each run through up to five different agent designs. Because the format is identical across all of them, comparing how two models handled the same task, or how one model's behavior shifts between agent designs, is a filter, not a data-cleaning project. + +## What it doesn't cover + +The dataset is deliberately narrow in a few ways worth stating plainly. It captures the model's own chat calls, not the non-model actions around them, so an agent's file edits or environment steps show up only through the calls that produced them. The models are specific snapshots evaluated at one point in time, not standing verdicts on any model family. And six benchmarks, however varied, are not the full range of work agents will eventually do. If you need the surrounding scaffolding — the user-simulator turns and grading calls we filtered out — the full corpus below keeps all of it. + +## See for yourself + +The runs are public and ready to load — one open format, 236 MB, [on Hugging Face](https://huggingface.co/datasets/Exgentic/agent-llm-traces-v2). Whatever you are building agents to do, or worried they might do, you can now start from what they actually did instead of a single line of results. Open a few runs and follow how they get there. + +> The how was always there, inside every run. Now anyone can read it. + +## Sources + +- Exgentic Agent LLM Traces: [huggingface.co/datasets/Exgentic/agent-llm-traces-v2](https://huggingface.co/datasets/Exgentic/agent-llm-traces-v2) +- Full corpus: [huggingface.co/datasets/Exgentic/traces-v2](https://huggingface.co/datasets/Exgentic/traces-v2) +- OpenTelemetry GenAI semantic conventions: [opentelemetry.io](https://opentelemetry.io/docs/specs/semconv/gen-ai/) +- Exgentic & Open Agent Leaderboard: [arXiv:2602.22953](https://arxiv.org/abs/2602.22953) · [exgentic.ai](https://www.exgentic.ai) + diff --git a/index.html b/index.html index da80813..87c4e78 100644 --- a/index.html +++ b/index.html @@ -198,6 +198,16 @@

Open Source. Open Data. Open Leaderboard.

Blog

Research updates, benchmarks, and findings from the Exgentic team.

+
+
+ + · + 8 min read +
+

What Frontier Agents Actually Do

+

10K real agent runs, in one standard format, so anyone can study how agents behave and not just what they scored.

+ Read more → +
From 6c0c2b1b74a169a6aed1bfd2da4d0e591e43a21e Mon Sep 17 00:00:00 2001 From: Elron Bandel Date: Wed, 1 Jul 2026 19:34:04 +0300 Subject: [PATCH 2/6] Make blog post title more informative --- blogs/agent-llm-traces-v2/blog.md | 2 +- index.html | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/blogs/agent-llm-traces-v2/blog.md b/blogs/agent-llm-traces-v2/blog.md index 879ee30..d1f6031 100644 --- a/blogs/agent-llm-traces-v2/blog.md +++ b/blogs/agent-llm-traces-v2/blog.md @@ -1,5 +1,5 @@ --- -title: "What Frontier Agents Actually Do" +title: "What Frontier Agents Actually Do: A Public Dataset of 10K Agent Runs" subtitle: "10K real agent runs, in one standard format, so anyone can study how agents behave and not just what they scored." author: Elron Bandel date: 2026-07-01 diff --git a/index.html b/index.html index 87c4e78..4eba70b 100644 --- a/index.html +++ b/index.html @@ -204,7 +204,7 @@

Blog

· 8 min read
-

What Frontier Agents Actually Do

+

What Frontier Agents Actually Do: A Public Dataset of 10K Agent Runs

10K real agent runs, in one standard format, so anyone can study how agents behave and not just what they scored.

Read more →
From 3e73a4d92275d1d228313835e203a72fbfdfcf94 Mon Sep 17 00:00:00 2001 From: Elron Bandel Date: Wed, 1 Jul 2026 19:34:58 +0300 Subject: [PATCH 3/6] Remove author from traces post; render meta cleanly when author is empty --- blogs/agent-llm-traces-v2/blog.md | 2 +- scripts/build_blog.py | 10 ++++++---- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/blogs/agent-llm-traces-v2/blog.md b/blogs/agent-llm-traces-v2/blog.md index d1f6031..ca0c154 100644 --- a/blogs/agent-llm-traces-v2/blog.md +++ b/blogs/agent-llm-traces-v2/blog.md @@ -1,7 +1,7 @@ --- title: "What Frontier Agents Actually Do: A Public Dataset of 10K Agent Runs" subtitle: "10K real agent runs, in one standard format, so anyone can study how agents behave and not just what they scored." -author: Elron Bandel +author: "" date: 2026-07-01 reading_time: 8 min slug: agent-llm-traces-v2 diff --git a/scripts/build_blog.py b/scripts/build_blog.py index a9f6e21..62e3e0b 100644 --- a/scripts/build_blog.py +++ b/scripts/build_blog.py @@ -117,9 +117,7 @@

{title}

{subtitle}

- {author} - · - + {author_block} · {reading_time} read
-

What Frontier Agents Actually Do: A Public Dataset of 10K Agent Runs

+

What Agents Do: A Large Scale Public Dataset of Agent Runs

10K real agent runs, in one standard format, so anyone can study how agents behave and not just what they scored.

Read more →
From 7e63c2b3104552e678938fb9590ae465604649a1 Mon Sep 17 00:00:00 2001 From: Lena Dankin Date: Thu, 16 Jul 2026 09:17:24 +0300 Subject: [PATCH 5/6] Add dataset stats, inference-perf section, and terminology fixes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add "A closer look" section with three tables: benchmark complexity, per-model token cost, and per-agent prefix retention - Add "Inference benchmarking: an ongoing example" section covering inference-perf (kubernetes-sigs/inference-perf) and its OTel trace replay - Add load-testing bullet to "What you can do with it" - Fix "sessions" → "runs" throughout - Add inference-perf source link Co-Authored-By: Claude Sonnet 4.6 --- blogs/agent-llm-traces-v2/blog.md | 59 +++++++++++++++++++++++++++++-- 1 file changed, 57 insertions(+), 2 deletions(-) diff --git a/blogs/agent-llm-traces-v2/blog.md b/blogs/agent-llm-traces-v2/blog.md index 0a46fa2..85f995d 100644 --- a/blogs/agent-llm-traces-v2/blog.md +++ b/blogs/agent-llm-traces-v2/blog.md @@ -33,12 +33,13 @@ Preserving the full run, not just the verdict, is what makes the data useful. Ea - **Build tools that evaluate agents, using real failures.** Study where agents actually break instead of on made-up examples. - **Replay a recorded moment against a new model.** Feed the exact situation an agent faced to a different model and compare, without re-running the whole evaluation. - **Debug your own agent against a reference set.** Compare its behavior to how the strongest models handle the same kinds of tasks. +- **Load-test inference infrastructure against realistic traffic.** Standard load tests send independent requests; real agent runs are dependency chains where the output of one call shapes the input of the next. Because these traces are already in OTel format, they work directly with tools that replay recorded workloads against live endpoints. ## What's in it -The dataset is 10K sessions and 242K model calls, filtered down from a full corpus of 10.5K sessions and 626K calls. Roughly two-thirds of the raw calls were removed on purpose: the scaffolding around the agent under test. +The dataset is 10K runs and 242K model calls, filtered down from a full corpus of 10.5K runs and 626K calls. Roughly two-thirds of the raw calls were removed on purpose: the scaffolding around the agent under test. -Each session passes through a filter that strips out the supporting machinery — the simulated user, the automatic grader — collapses repeated identical retries, and keeps only the calls the tested model actually made. What remains is the model under test, and nothing the evaluation wrapped around it. The totals for steps, tokens, and cost are recomputed from the calls that survive, so they describe the agent's own footprint and not the machinery around it. +Each run passes through a filter that strips out the supporting machinery — the simulated user, the automatic grader — collapses repeated identical retries, and keeps only the calls the tested model actually made. What remains is the model under test, and nothing the evaluation wrapped around it. The totals for steps, tokens, and cost are recomputed from the calls that survive, so they describe the agent's own footprint and not the machinery around it. Every call is written in one open, standard format for recording model calls (the [OpenTelemetry GenAI conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/), which a growing part of the ecosystem already emits). A call carries the models used, the full input and output messages, the tools that were available, token usage, why the call stopped, and an error type when it failed. Each row also carries run-level context — which benchmark, which agent design, the score, the cost, the number of steps — so you can slice the data however you need. Because every run speaks the same format, these traces accumulate into something comparable instead of fragmenting into one dialect per tool — the value compounds as more are added. @@ -55,6 +56,59 @@ The coverage spans the range of real agent work — software engineering, deep r Five frontier models are represented in comparable volume — DeepSeek-V3.2 (2.3K runs), Kimi-K2.5 (2.3K), GPT-5.2 (2.1K), Claude Opus 4.5 (1.9K), and Gemini 3 Pro (1.4K) — each run through up to five different agent designs. Because the format is identical across all of them, comparing how two models handled the same task, or how one model's behavior shifts between agent designs, is a filter, not a data-cleaning project. +## A closer look + +Three views of the data, each answering a different question. + +**What kind of work is this?** Task complexity differs sharply across benchmarks: + +| Benchmark | Turns (median / mean / max) | Avg tool calls / run | Responses with >1 tool call (%) | Avg duration (min) | +|-----------|----------------------------:|---------------------:|--------------------------------:|-------------------:| +| SWE-bench | 38 / 46.8 / 259 | 31.7 | 10% | 16 | +| AppWorld | 19 / 32.3 / 300 | 27.8 | 10% | 14 | +| BrowseComp Plus | 13 / 19.7 / 167 | 15.4 | 14% | 15 | +| τ²-bench Telecom | 13 / 16.3 / 198 | 5.9 | 3% | 8 | +| τ²-bench Airline | 10 / 11.4 / 151 | 6.2 | 9% | 5 | +| τ²-bench Retail | 11 / 11.9 / 151 | 5.5 | 6% | 6 | + +Turns are the number of model calls in a run. Tool calls are how many tool invocations the model issued in total across those calls. The third column counts the fraction of model calls where the model returned more than one tool call in a single response — batching multiple requests at once rather than one at a time. Duration is the wall-clock time from the first to the last model call in a run. + +Taken together, these four numbers show how structurally different the workloads are: SWE-bench and AppWorld runs are long, tool-heavy, and slow; τ²-bench runs are short, light on tools, and fast. The workloads are not interchangeable, and which ones you pull matters for training, evaluation, and infrastructure work. + +**How much does it cost?** Token consumption varies as much by model as by task: + +| Model | Avg tokens / model call | Avg tokens / run | +|-------|------------------------:|-----------------:| +| Claude Opus 4.5 | 66.8K | 2.4M | +| Gemini 3 Pro | 41.8K | 1.1M | +| DeepSeek-V3.2 | 29.0K | 880K | +| GPT-5.2 | 26.3K | 552K | +| Kimi-K2.5 | 23.1K | 655K | + +A Claude Opus 4.5 run consumes on average 4× the tokens of a GPT-5.2 run — a difference that compounds across hundreds of calls. Selecting traces by model matters if you're estimating replay costs, building token-budget-aware training pipelines, or studying how different models use their context window. + +**How does context evolve?** Because agent design determines how the prompt is constructed on each call, it is the primary driver of how context accumulates — not the task type. Prefix retention measures what fraction of messages carry over from one call to the next: 1.0 means a single growing conversation; lower values mean the agent reconstructs the context window more aggressively each time. + +| Agent design | Avg turns / run | Prefix retention | +|--------------|----------------:|-----------------:| +| claude_code | 33.0 | 0.90 | +| smolagents_code | 21.3 | 0.87 | +| tool_calling | 22.2 | 0.87 | +| openai_solo | 18.3 | 0.86 | +| tool_calling_with_shortlisting | 50.1 | 0.03 | + +Four of the five designs maintain high retention throughout a run — they accumulate a single long conversation. The exception is `tool_calling_with_shortlisting`, which drops to 0.03: it selects a fresh subset of tools for each call, rebuilding the prompt from scratch every time. Despite having the highest average turn count by far, it never builds up a long conversational context. This makes its traces structurally different from the others at the message level — worth knowing before using them for fine-tuning or prefix caching analysis. + +## Inference benchmarking: an ongoing example + +One use of this data is already underway. A team working on LLM inference infrastructure is using these traces to benchmark serving configurations under realistic agentic load — the kind of workload that synthetic benchmarks systematically miss. + +The tool at the center of that effort is [inference-perf](https://github.com/kubernetes-sigs/inference-perf), a Kubernetes SIG benchmarking tool that replays OTel traces as directed acyclic graphs. Rather than sending independent requests at a fixed rate, it reconstructs the dependency structure of a real agent run — the order and dependencies between calls, and how context accumulated across them — and drives that structure against the inference endpoint under test. Downstream calls receive the actual live-generated output from upstream calls, so KV-cache behavior and context growth stay realistic throughout the replay. + +Because the Exgentic traces are already in OTel format, they plug into this workflow without any conversion. The result is end-to-end run latency measured against real agent traffic, not a synthetic proxy for it — exactly the signal needed to evaluate whether a serving configuration change is worth shipping. + +This is one example of what becomes possible when traces are public and in a standard format. The work is ongoing; we'll share results as they come in. + ## What it doesn't cover The dataset is deliberately narrow in a few ways worth stating plainly. It captures the model's own chat calls, not the non-model actions around them, so an agent's file edits or environment steps show up only through the calls that produced them. The models are specific snapshots evaluated at one point in time, not standing verdicts on any model family. And six benchmarks, however varied, are not the full range of work agents will eventually do. If you need the surrounding scaffolding — the user-simulator turns and grading calls we filtered out — the full corpus below keeps all of it. @@ -70,5 +124,6 @@ The runs are public and ready to load — one open format, 236 MB, [on Hugging F - Exgentic Agent LLM Traces: [huggingface.co/datasets/Exgentic/agent-llm-traces-v2](https://huggingface.co/datasets/Exgentic/agent-llm-traces-v2) - Full corpus: [huggingface.co/datasets/Exgentic/traces-v2](https://huggingface.co/datasets/Exgentic/traces-v2) - OpenTelemetry GenAI semantic conventions: [opentelemetry.io](https://opentelemetry.io/docs/specs/semconv/gen-ai/) +- inference-perf: [github.com/kubernetes-sigs/inference-perf](https://github.com/kubernetes-sigs/inference-perf) · [Benchmarking LLM Inference with Production Agent Traces](https://medium.com/inference-perf/benchmarking-llm-inference-with-production-agent-traces-f47f7f994aff) - Exgentic & Open Agent Leaderboard: [arXiv:2602.22953](https://arxiv.org/abs/2602.22953) · [exgentic.ai](https://www.exgentic.ai) From 2188bfa0c35622fd83638680177d5750c20594db Mon Sep 17 00:00:00 2001 From: Lena Dankin Date: Mon, 3 Aug 2026 16:12:36 +0300 Subject: [PATCH 6/6] Address PR review comments: surface agent-design axis, clarify traces, tighten inference section, fluency fixes --- blogs/agent-llm-traces-v2/blog.md | 34 +++++++++++++++++-------------- 1 file changed, 19 insertions(+), 15 deletions(-) diff --git a/blogs/agent-llm-traces-v2/blog.md b/blogs/agent-llm-traces-v2/blog.md index 85f995d..a540d6d 100644 --- a/blogs/agent-llm-traces-v2/blog.md +++ b/blogs/agent-llm-traces-v2/blog.md @@ -9,9 +9,9 @@ slug: agent-llm-traces-v2 AI agents now book our travel, write our code, and answer our customers. Two can finish the same task and look identical on paper — one taking three clean steps, the other opening a file it shouldn't, crashing a service, and recovering by luck. We already know whether they succeed and what they cost. It's time we saw how they get there. -Today we're releasing [Exgentic Agent LLM Traces](https://huggingface.co/datasets/Exgentic/agent-llm-traces-v2): 10K real agent runs, across five leading models and six kinds of task, with every model call preserved in one standard format. It's the part of the run that normally gets thrown away. +Today we're releasing [Exgentic Agent LLM Traces](https://huggingface.co/datasets/Exgentic/agent-llm-traces-v2): 10K real agent runs — five leading models, five agent designs, six kinds of task — with every model call preserved in one standard format. It's the part of the run that normally gets thrown away. -> We've always known whether agents succeed or failed. Now we can see how. +> We've always known whether agents succeed. Now the record of how is public. ## Why this matters @@ -19,7 +19,9 @@ Agents are moving into real work: writing code, handling customer support, doing That leaves a gap. We study *models* on mountains of public data. We study *agents* — the multi-step, tool-using, failing-and-recovering systems people actually deploy — on almost nothing public. The behavior that decides whether an agent is worth shipping lives in the full record of what it did, and that record rarely leaves the lab that produced it. -The gap exists for a concrete reason: producing real agent runs at scale is slow, costly, and fragmented. It means running the strongest available models across many different environments, thousands of times over, and paying for every token they generate. And even once you have the runs, every tool records them in its own shape, so what data exists stays private and mutually incompatible. There has been no large, uniform, public record of what capable agents actually do. +A single trace is the full sequence of what an agent did: every model call, in order — the input it received, the tools it had, what it output, and whether it errored. A score tells you whether an agent succeeded; the sequence tells you how it got there — which tools it chose, how its context grew, where it hesitated or recovered. That's the part of the run that normally gets discarded, and the part that's hardest to study without a large public record of it. + +The gap exists for a concrete reason: producing real agent runs at scale is slow, costly, and fragmented. It means running frontier models across many different environments, thousands of times over, and paying for every token they generate. And even once you have the runs, every tool records them in its own shape, so what data exists stays private and mutually incompatible. There has been no large, uniform, public record of what capable agents actually do. > Model behavior is studied everywhere. Agent behavior is barely studied at all, because the data isn't public. @@ -27,13 +29,13 @@ This release is a piece of that missing record. Real runs, at scale, from fronti ## What you can do with it -Preserving the full run, not just the verdict, is what makes the data useful. Each of these was previously gated behind running the evaluations yourself: +Preserving the full run, not just the verdict, is what makes the data useful. Each of these previously required running the evaluations yourself: -- **Train and fine-tune agents on real runs.** Long, tool-using runs from strong models are the kind of data that is hard to obtain and expensive to generate. +- **Debug your own agent against a reference set.** Compare its behavior to how frontier models handle the same kinds of tasks. - **Build tools that evaluate agents, using real failures.** Study where agents actually break instead of on made-up examples. -- **Replay a recorded moment against a new model.** Feed the exact situation an agent faced to a different model and compare, without re-running the whole evaluation. -- **Debug your own agent against a reference set.** Compare its behavior to how the strongest models handle the same kinds of tasks. -- **Load-test inference infrastructure against realistic traffic.** Standard load tests send independent requests; real agent runs are dependency chains where the output of one call shapes the input of the next. Because these traces are already in OTel format, they work directly with tools that replay recorded workloads against live endpoints. +- **Replay a recorded step against a new model.** Each step in a trace captures the full context the model saw — the conversation history, system instructions, tool calls, and tool results — along with what it output. Feed that exact context to a different model to see how it would have acted, without the overhead of deploying and running the agent yourself. +- **Evaluate inference infrastructure under realistic load.** Real agent runs are dependency chains — one call's output shapes the next — not the independent requests standard load tests send. Replay these traces against a live endpoint to measure how a serving configuration behaves under true agentic traffic. +- **Train and fine-tune agents on real runs.** Long, tool-using runs from frontier models are the kind of data that is hard to obtain and expensive to generate. ## What's in it @@ -54,11 +56,13 @@ The coverage spans the range of real agent work — software engineering, deep r | Customer service | τ²-bench Retail | 1.8K | 198K | | Customer service | τ²-bench Airline | 1K | 255K | -Five frontier models are represented in comparable volume — DeepSeek-V3.2 (2.3K runs), Kimi-K2.5 (2.3K), GPT-5.2 (2.1K), Claude Opus 4.5 (1.9K), and Gemini 3 Pro (1.4K) — each run through up to five different agent designs. Because the format is identical across all of them, comparing how two models handled the same task, or how one model's behavior shifts between agent designs, is a filter, not a data-cleaning project. +*(The token counts above group by benchmark; the model table below groups by model — averages differ because each model ran a different mix of benchmarks.)* + +Five frontier models are represented in comparable volume — DeepSeek-V3.2 (2.3K runs), Kimi-K2.5 (2.3K), GPT-5.2 (2.1K), Claude Opus 4.5 (1.9K), and Gemini 3 Pro (1.4K) — each run through up to five different agent designs. Because the format is identical across all of them, the runs line up directly — the same task across models, or one model across agent designs, without any per-tool reconciliation. ## A closer look -Three views of the data, each answering a different question. +These aren't our findings — they're examples of what the traces let you ask. **What kind of work is this?** Task complexity differs sharply across benchmarks: @@ -73,7 +77,7 @@ Three views of the data, each answering a different question. Turns are the number of model calls in a run. Tool calls are how many tool invocations the model issued in total across those calls. The third column counts the fraction of model calls where the model returned more than one tool call in a single response — batching multiple requests at once rather than one at a time. Duration is the wall-clock time from the first to the last model call in a run. -Taken together, these four numbers show how structurally different the workloads are: SWE-bench and AppWorld runs are long, tool-heavy, and slow; τ²-bench runs are short, light on tools, and fast. The workloads are not interchangeable, and which ones you pull matters for training, evaluation, and infrastructure work. +These four numbers let you see how structurally different the workloads are: SWE-bench and AppWorld runs are long, tool-heavy, and slow; τ²-bench runs are short, light on tools, and fast. The workloads are not interchangeable, and which ones you pull matters for training, evaluation, and infrastructure work. **How much does it cost?** Token consumption varies as much by model as by task: @@ -85,7 +89,7 @@ Taken together, these four numbers show how structurally different the workloads | GPT-5.2 | 26.3K | 552K | | Kimi-K2.5 | 23.1K | 655K | -A Claude Opus 4.5 run consumes on average 4× the tokens of a GPT-5.2 run — a difference that compounds across hundreds of calls. Selecting traces by model matters if you're estimating replay costs, building token-budget-aware training pipelines, or studying how different models use their context window. +Token consumption varying as much by model as by task is the kind of signal that matters when estimating replay costs, building token-budget-aware training pipelines, or studying how different models use their context window. **How does context evolve?** Because agent design determines how the prompt is constructed on each call, it is the primary driver of how context accumulates — not the task type. Prefix retention measures what fraction of messages carry over from one call to the next: 1.0 means a single growing conversation; lower values mean the agent reconstructs the context window more aggressively each time. @@ -97,15 +101,15 @@ A Claude Opus 4.5 run consumes on average 4× the tokens of a GPT-5.2 run — a | openai_solo | 18.3 | 0.86 | | tool_calling_with_shortlisting | 50.1 | 0.03 | -Four of the five designs maintain high retention throughout a run — they accumulate a single long conversation. The exception is `tool_calling_with_shortlisting`, which drops to 0.03: it selects a fresh subset of tools for each call, rebuilding the prompt from scratch every time. Despite having the highest average turn count by far, it never builds up a long conversational context. This makes its traces structurally different from the others at the message level — worth knowing before using them for fine-tuning or prefix caching analysis. +Four of the five designs maintain high retention throughout a run — they accumulate a single long conversation. The exception is `tool_calling_with_shortlisting`, which drops to 0.03: it selects a fresh subset of tools for each call, rebuilding the prompt from scratch every time. Despite having the highest average turn count by far, it never builds up a long conversational context. This makes its traces structurally different from the others at the message level — something to check before using them for fine-tuning or prefix caching analysis. ## Inference benchmarking: an ongoing example One use of this data is already underway. A team working on LLM inference infrastructure is using these traces to benchmark serving configurations under realistic agentic load — the kind of workload that synthetic benchmarks systematically miss. -The tool at the center of that effort is [inference-perf](https://github.com/kubernetes-sigs/inference-perf), a Kubernetes SIG benchmarking tool that replays OTel traces as directed acyclic graphs. Rather than sending independent requests at a fixed rate, it reconstructs the dependency structure of a real agent run — the order and dependencies between calls, and how context accumulated across them — and drives that structure against the inference endpoint under test. Downstream calls receive the actual live-generated output from upstream calls, so KV-cache behavior and context growth stay realistic throughout the replay. +What makes these runs different from synthetic benchmarks is their structure: real agent runs are dependency chains, where each call's output becomes the next call's input, context accumulates, and KV-cache pressure builds exactly as it does in production. A synthetic load generator can't reproduce that. The replay keeps structure and inputs from the recorded run while regenerating outputs live against the endpoint under test, so downstream calls receive fresh output from upstream — context grows as it would in a real run, and cache behavior reflects what production would produce. -Because the Exgentic traces are already in OTel format, they plug into this workflow without any conversion. The result is end-to-end run latency measured against real agent traffic, not a synthetic proxy for it — exactly the signal needed to evaluate whether a serving configuration change is worth shipping. +The tool driving this is [inference-perf](https://github.com/kubernetes-sigs/inference-perf), a Kubernetes SIG benchmarking tool that replays OTel traces against a live endpoint. Because the Exgentic traces are already in OTel format, they plug in without conversion. The result is latency, cache hit rates, and cost measured under real agentic load — exactly the signal needed to evaluate whether a serving configuration change is worth shipping. This is one example of what becomes possible when traces are public and in a standard format. The work is ongoing; we'll share results as they come in.