Skip to content

Add RubyLLM::Agents.track for multi-agent request grouping#16

Merged
adham90 merged 8 commits intomainfrom
feat/track_block
Mar 8, 2026
Merged

Add RubyLLM::Agents.track for multi-agent request grouping#16
adham90 merged 8 commits intomainfrom
feat/track_block

Conversation

@adham90
Copy link
Owner

@adham90 adham90 commented Feb 28, 2026

Summary

  • Adds RubyLLM::Agents.track block wrapper that collects every Result produced by agent calls inside it, aggregates cost/token/timing stats, and returns a TrackReport
  • Injects shared request_id and tags into execution records automatically, enabling grouped observability
  • Adds a Requests dashboard page to browse and inspect tracked multi-agent workflows
  • Supports nesting, error capture, shared defaults (tenant, tags), and all 12 result types via a Trackable mixin

What's included

Core (lib/ruby_llm/agents/)

  • Tracker — internal collector that accumulates Results during a tracked block using Thread.current
  • TrackReport — public aggregation class with total_cost, total_tokens, duration_ms, cost_breakdown, models_used, errors, all_successful?, and more
  • Results::Trackable mixin — included in all 12 result classes (text, embedding, speech, transcription, 8 image types) for automatic tracker registration
  • BaseAgent integration — merges tracker defaults (tenant, tags) into agent options
  • Instrumentation middleware — injects request_id and tags from the active tracker into execution records

Dashboard (app/)

  • RequestsController with index (grouped by request_id, sortable, paginated) and show (execution timeline, summary stats)
  • Views with consistent design matching the existing dashboard aesthetic
  • Navigation links added to both desktop and mobile layouts

Example app & seeds

  • SummarizeAgent + ClassifyAgent tracking examples with comprehensive usage docs
  • 8 diverse seed request scenarios (customer support, content pipeline, error handling, batch processing, streaming conversation, multi-model, in-progress, single-agent)

Tests (TDD — 116 new specs)

  • tracker_spec.rb — 10 specs for Tracker internals
  • track_report_spec.rb — 30 specs for TrackReport aggregation
  • result_tracking_spec.rb — 7 specs for Result hook across all types
  • track_integration_spec.rb — 19 specs for .track method (nesting, errors, defaults)
  • track_instrumentation_spec.rb — 5 specs for request_id/tags injection
  • requests_controller_spec.rb — 8 specs for dashboard controller

Usage

report = RubyLLM::Agents.track(request_id: "support_123", tags: { workflow: "support" }) do
  summary = SummarizeAgent.call(text: ticket.body)
  category = ClassifyAgent.call(message: summary.content)
end

report.total_cost      # => 0.0023
report.call_count      # => 2
report.all_successful? # => true
report.models_used     # => ["gpt-4o-mini", "gpt-4o"]

Test plan

  • All 116 new specs pass
  • Full test suite runs with no new failures (4,242 examples)
  • StandardRB lint passes
  • Seeds run successfully in example app
  • Dashboard pages render with seeded data

🤖 Generated with Claude Code

adham90 and others added 8 commits February 28, 2026 01:55
Introduces the core data structures for the track block feature:
- Tracker: internal collector that accumulates Results during a block
- TrackReport: public aggregation class with cost/token/timing summaries

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Every Result (base, embedding, speech, transcription, image) now:
- Stores agent_class_name for cost_breakdown reporting
- Auto-registers with active Thread.current tracker via Trackable mixin
- Includes agent_class_name in to_h output

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements the core .track API:
- Wraps block, collects all Results, returns TrackReport
- Supports nesting with result bubbling to parent tracker
- Captures errors without re-raising (report.error)
- Merges shared defaults (tenant, tags) into agent options
- Cleans up thread-local in ensure block

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The instrumentation middleware now reads @_track_request_id and
@_track_tags from the agent instance and injects them into execution
records, enabling dashboard grouping by request_id and tag filtering.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New dashboard page that groups executions by request_id:
- Index view with sortable columns, pagination, aggregate stats
- Show view with execution timeline, summary stats, timing details
- Navigation link in header (desktop + mobile)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds SummarizeAgent + ClassifyAgent as helper agents for demonstrating
RubyLLM::Agents.track, with comprehensive usage examples in comments
and a new "Multi-Agent Tracking" section in the example README.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Seeds 8 multi-agent tracked requests with diverse scenarios:
customer support, content pipeline, error handling, batch
processing, streaming conversation, multi-model, in-progress,
and single-agent cached requests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add llm_executions method to TestOrganization so stubs work when
  verify_partial_doubles is enabled by earlier spec files
- Remove invalid allow_any_instance_of(ActiveRecord::Relation) stub
  for avg_time_to_first_token since the actual method works correctly

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@adham90 adham90 merged commit 7c51a86 into main Mar 8, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant