Skip to content

Add LangChain auto-instrumentation and cross-framework benchmarks - #11

Closed
manemsai wants to merge 13 commits into
mainfrom
feature/notebook
Closed

Add LangChain auto-instrumentation and cross-framework benchmarks#11
manemsai wants to merge 13 commits into
mainfrom
feature/notebook

Conversation

@manemsai

@manemsai manemsai commented Jul 6, 2026 •
edited by pramodbn27
Loading

Copy link
Copy Markdown
Collaborator

Summary

This PR grows AgenticLens in three main directions:

  • adds optional LangChain / LangGraph auto-instrumentation via agenticlens.adapters.langchain.AgenticLensCallbackHandler
  • adds a cross-framework benchmark harness plus saved benchmark artifacts for support-refund and live travel-briefing workflows
  • expands docs and examples so the new integrations and benchmark workflows are easier to run and review

What Changed

  • Added agenticlens[langchain] with a LangChain callback adapter that records LLM, tool, and retriever activity into the active AgenticLens workflow.
  • Added adapter tests covering LLM usage extraction, tool/retriever step capture, duplicate-tool-call detection, and behavior outside an active profile() context.
  • Added benchmark runners for AutoGen, CrewAI, LangGraph, LlamaIndex, Semantic Kernel, and native Python.
  • Added shared benchmark datasets, benchmark aggregation/reporting scripts, and checked-in benchmark result artifacts.
  • Added practical examples including support_copilot.py, multiagent_edge_cases_demo.py, and live_multiagent_travel_briefing.py.
  • Updated README, changelog, MkDocs nav, and dedicated LangChain integration docs.
  • Updated CI dependency install so the langchain extra is available during checks.
  • Included a follow-up fix for CrewAI live benchmark Ruff naming/style issues.

Why

This makes AgenticLens easier to adopt in existing LangChain-style stacks while also giving the repo a concrete, repeatable way to compare token, cost, latency, retrieval, and tool-call behavior across multiple agent frameworks.

Validation

  • uv run ruff check benchmarks/frameworks/crewai/run_crewai_live.py
  • ruff check .
  • ruff format --check .
  • mypy
  • pytest

Reviewer Notes

  • The checked-in benchmark reports and charts are intentional; they document the current comparison baseline.
  • langchain-core is optional and exposed through the new langchain extra.
  • This branch was also updated with the latest main so the PR is now mergeable again.

manemsai and others added 11 commits July 5, 2026 23:34
@manemsai
updated readme file
95f3b51
@manemsai
updated readme file
ab608e9
@manemsai @claude
Add cross-framework benchmark harness and practical support examples
3cbef69 
Adds a benchmark suite that runs the same refund-ticket workload through
AutoGen, CrewAI, LangGraph, LlamaIndex, Semantic Kernel, and native Python,
profiling each with AgenticLens to normalize tokens/cost/latency for an
apples-to-apples comparison. Also adds two more realistic examples
(support_copilot.py, multiagent_edge_cases_demo.py) exercising the full
step lifecycle plus recommender edge cases.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@manemsai @claude
Link framework benchmark comparison from README
6b82ebf 
Surfaces the new benchmarks/ harness so the apples-to-apples
token/cost/latency comparison across agent frameworks is discoverable
from the main README instead of buried in a results folder.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@manemsai @claude
Fix lint and formatting in benchmark harness and example scripts
420eac7 
Applies ruff format, wraps long string literals under the 100-char
line limit, drops an unused import, and adds noqa for the intentional
sys.path manipulation in the native-Python benchmark runner. These
files were added without a lint pass in the prior commit.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@manemsai @claude
Add LangChain/LangGraph auto-instrumentation adapter
f7e2a7f 
Adds agenticlens.adapters.langchain.AgenticLensCallbackHandler, an
optional (pip install "agenticlens[langchain]") callback handler that
turns LangChain's own on_llm_*/on_tool_*/on_retriever_* events into
AgenticLens steps automatically, so LLM calls, tool calls, and
retrieval don't need manual step() wrapping. Since it produces the
same Step shape as the manual API, every existing recommender,
exporter, and CLI command works against LangChain-sourced workflows
unmodified.

This was the top item on the roadmap's near-term integrations list
(README, ROADMAP). Adds langchain-core as the optional `langchain`
extra, wires it into CI so the adapter is actually tested, and adds a
mypy override to stop following into langchain-core's internals
(its transitive numpy stub can be newer than our target Python
version). Documented in docs/langchain-integration.md and linked from
the README and mkdocs nav.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@manemsai @claude
Fix benchmark runner to propagate PYTHONPATH to subprocesses
61271a9 
benchmark_runner.py invokes each framework script via `agenticlens
profile <script>` in a subprocess. Every script except native_python
imports from the `benchmarks` package, which only resolved if the
project root happened to already be on PYTHONPATH in the caller's
shell -- otherwise every framework but native_python failed with
ModuleNotFoundError. Pass PYTHONPATH explicitly so the harness works
out of the box for any contributor.

Also refreshes the committed report/result fixtures from a clean run
through the fixed harness, confirming all six frameworks (including
AutoGen, previously unverified in this environment) now execute
end-to-end and reproduce the same token/cost figures as before.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@manemsai @claude
Add live cross-framework benchmark: real APIs, real latency variance
8ee16c2 
Extends the benchmark suite with a second workload that hits real, live,
free APIs for every tool/retriever step (Open-Meteo geocoding + forecast,
Frankfurter exchange rates, Wikipedia REST summary) instead of scripted
fixtures. Adds a *_live.py implementation per framework (native Python,
LangGraph, CrewAI, AutoGen, LlamaIndex, Semantic Kernel), a shared
benchmarks/shared/live_travel_tasks.py task module, and
live_benchmark_runner.py to orchestrate and compare them.

Unlike the deterministic support-refund benchmark -- where every framework
reports identical numbers because everything is a FakeResponse -- token
counts stay stable here (LLM steps still fall back without an
OPENAI_API_KEY) but latency is now genuinely live: it varies run to run
and framework to framework based on real network conditions, with
geocoding consistently the slowest step (~600-670ms) across all six.

Also adds examples/live_multiagent_travel_briefing.py, the single-run
version of the same workflow this benchmark is derived from.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@manemsai
Merge branch 'feature/notebook' of https://github.com/DeepAgentLabs/a…
93b4cab 
…genticlens into feature/notebook
@pramodbn27
Merge main into feature/notebook
a95fddf
@pramodbn27
Merge remote-tracking branch 'origin/feature/notebook' into codex/pr1…
988d6e7 
…1-merge-fix
@pramodbn27
pramodbn27 requested a review from a team as a code owner August 9, 2026 06:02
@pramodbn27 pramodbn27 changed the title updated readme file Add LangChain auto-instrumentation and cross-framework benchmarks Aug 9, 2026
@pramodbn27 pramodbn27 assigned pramodbn27 and manemsai and unassigned pramodbn27 Aug 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pramodbn-cloud pramodbn-cloud pramodbn-cloud approved these changes
@pramodbn27 pramodbn27 pramodbn27 approved these changes

Assignees

@manemsai manemsai

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@manemsai @pramodbn-cloud @pramodbn27 @jothsnapraveena