Add LangChain auto-instrumentation and cross-framework benchmarks

.claude/settings.json

@@ -0,0 +1,19 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(python -c ' *)",
+      "Bash(python -m mypy --no-site-packages src/agenticlens/models/step.py)",
+      "Bash(pip uninstall *)",
+      "Bash(git check-ignore *)",
+      "Bash(python -m pytest tests/test_adapters_langchain.py -q)",
+      "Bash(python -m pytest -q tests/)",
+      "Bash(python -m ruff check benchmarks/shared/benchmark_runner.py)",
+      "Bash(git commit -m ' *)",
+      "Bash(mkdir -p \"C:\\\\Users\\\\manem\\\\AppData\\\\Local\\\\Temp\\\\claude\\\\e--agenticlens\\\\9e618684-551e-4264-8655-deac3e2e58bd\\\\scratchpad\")",
+      "Skill(artifact-design)",
+      "Skill(artifact-design:*)",
+      "Bash(python -c \"import os; print\\('OPENAI_API_KEY set:', bool\\(os.getenv\\('OPENAI_API_KEY'\\)\\)\\)\")",
+      "Bash(python examples/live_multiagent_travel_briefing.py)"
+    ]
+  }
+}

.github/workflows/ci.yml

@@ -23,7 +23,7 @@ jobs:
         run: uv python install ${{ matrix.python-version }}
 
       - name: Install dependencies
-        run: uv sync --extra dev --python ${{ matrix.python-version }}
+        run: uv sync --extra dev --extra langchain --python ${{ matrix.python-version }}
 
       - name: Lint (ruff check)
         run: uv run ruff check .

.gitignore

@@ -14,6 +14,11 @@ site/
 .env
 test/
 
+# Generated workflow reports from running examples/benchmarks locally
+/*_report.json
+/workflow.json
+/report.json
+
 # Local patent/provisional drafting artifacts
 docs/patents/
 tools/build_agenticlens_provisional.py

CHANGELOG.md

@@ -4,6 +4,23 @@ All notable changes to this project will be documented here.
 
 This project follows [Semantic Versioning](https://semver.org/).
 
+## Unreleased
+
+### Added
+
+- `agenticlens.adapters.langchain.AgenticLensCallbackHandler`, an optional
+  (`pip install "agenticlens[langchain]"`) LangChain/LangGraph callback
+  handler that auto-instruments LLM, tool, and retriever calls as AgenticLens
+  steps without manual `step()` blocks. Documented in
+  `docs/langchain-integration.md`.
+- `benchmarks/`, a cross-framework benchmark harness that profiles the same
+  refund-ticket workload through AutoGen, CrewAI, LangGraph, LlamaIndex,
+  Semantic Kernel, and native Python for an apples-to-apples
+  token/cost/latency comparison. Linked from the README.
+- `examples/support_copilot.py`, `examples/multiagent_edge_cases_demo.py`, and
+  `examples/live_multiagent_travel_briefing.py`, additional practical,
+  edge-case, and live multi-agent profiling examples.
+
 ## 0.4.0 - 2026-08-08
 
 ### Added

README.md

@@ -56,6 +56,9 @@ The product idea is simple:
 - [Features](#features)
 - [Cost Calculation](#cost-calculation)
 - [Configuration Reference](#configuration-reference)
+- [Framework Benchmarks](#framework-benchmarks)
+- [Notebooks](#notebooks)
+- [LangChain Integration](#langchain-integration)
 - [CLI Reference](#cli-reference)
 - [Current Limitations](#current-limitations)
 - [Development](#development)
@@ -961,6 +964,9 @@ Other examples:
 - `examples/rag_customer_support_demo.py`
 - `examples/multiagent_support_demo.py`
 - `examples/multiagent_token_optimization_demo.py`
+- `examples/support_copilot.py` — practical support workflow profiling example
+- `examples/multiagent_edge_cases_demo.py` — edge-case instrumentation example
+- `examples/live_multiagent_travel_briefing.py` — live provider multi-agent travel briefing demo
 - `examples/reference_workflows/langgraph_supervisor.py` — offline LangGraph supervisor
 - `examples/export_demo.py` — export to Markdown and Jira
 - `examples/live_evaluation_demo.py` — trusted live Python target for `evaluate-live`
@@ -979,6 +985,28 @@ official framework repositories. See
 [docs/multi-agent-reference-workflows.md](docs/multi-agent-reference-workflows.md)
 for setup, source links, dependency isolation, and instrumentation boundaries.
 
+## Framework Benchmarks
+
+`benchmarks/` runs the same practical refund-ticket workload through AutoGen,
+CrewAI, LangGraph, LlamaIndex, Semantic Kernel, and native Python, profiling
+each with AgenticLens to normalize tokens, cost, latency, tool calls, and
+retrieved chunks across implementations. See
+[benchmarks/results/benchmark_summary.md](benchmarks/results/benchmark_summary.md)
+for the current comparison table.
+
+## Notebooks
+
+Beginner-friendly notebooks are available in:
+
+```text
+notebooks/agenticlens_workflow_demo_beginner.ipynb
+notebooks/agenticlens_multiagent_demo_beginner.ipynb
+```
+
+The notebooks walk through step-by-step workflow and multi-agent profiling,
+token usage tables, latency and cost charts, saved AgenticLens artifacts, and
+optimization analysis.
+
 ## Exporting Reports
 
 ### Markdown
@@ -1026,6 +1054,29 @@ Set credentials via environment variables for safety — see
 
 For sample output previews of all formats, see [docs/export-formats.md](docs/export-formats.md).
 
+## LangChain Integration
+
+Auto-instrument a LangChain (or LangGraph) run via its callback system instead
+of wrapping every call in `step()`:
+
+```bash
+pip install "agenticlens[langchain]"
+```
+
+```python
+from agenticlens import profile
+from agenticlens.adapters.langchain import AgenticLensCallbackHandler
+
+with profile("My LangChain App") as workflow:
+    chain.invoke(inputs, config={"callbacks": [AgenticLensCallbackHandler()]})
+```
+
+LLM calls, tool calls, and retriever calls are tracked automatically as
+`llm_call`, `tool_call`, and `retriever` steps, with token usage extracted the
+same way `s.record(...)` does. See
+[docs/langchain-integration.md](docs/langchain-integration.md) for details on
+what is and is not tracked.
+
 ## CLI Reference
 
 Profile a Python script:

benchmarks/compare_results.py

@@ -0,0 +1,254 @@
+import json
+from pathlib import Path
+
+import matplotlib.pyplot as plt
+import pandas as pd
+
+REPORTS = {
+    "Native Python": "benchmarks/reports/native_python/support_refund_report.json",
+    "LangGraph": "benchmarks/reports/langgraph/support_refund_report.json",
+    "CrewAI": "benchmarks/reports/crewai/support_refund_report.json",
+    "AutoGen": "benchmarks/reports/autogen/support_refund_report.json",
+    "LlamaIndex": "benchmarks/reports/llamaindex/support_refund_report.json",
+    "Semantic Kernel": "benchmarks/reports/semantic_kernel/support_refund_report.json",
+}
+
+RESULTS_DIR = Path("benchmarks/results")
+RESULTS_DIR.mkdir(parents=True, exist_ok=True)
+
+
+def load_report(path: str | Path) -> dict:
+    path = Path(path)
+    if not path.exists():
+        raise FileNotFoundError(f"Report not found: {path}")
+    return json.loads(path.read_text(encoding="utf-8"))
+
+
+def summarize_report(framework: str, report: dict) -> dict:
+    steps = report.get("steps", [])
+
+    total_tokens = 0
+    prompt_tokens = 0
+    completion_tokens = 0
+    total_cost = 0.0
+    total_latency = 0.0
+    tool_calls = 0
+    retrieved_chunks = 0
+
+    highest_token_step = None
+    highest_step_tokens = -1
+
+    highest_cost_step = None
+    highest_step_cost = -1.0
+
+    for step in steps:
+        metrics = step.get("metrics") or {}
+        metadata = step.get("metadata") or {}
+
+        step_tokens = metrics.get("total_tokens") or 0
+        step_prompt_tokens = metrics.get("prompt_tokens") or 0
+        step_completion_tokens = metrics.get("completion_tokens") or 0
+        step_cost = metrics.get("cost") or 0.0
+        step_latency = metrics.get("latency") or 0.0
+
+        total_tokens += step_tokens
+        prompt_tokens += step_prompt_tokens
+        completion_tokens += step_completion_tokens
+        total_cost += step_cost
+        total_latency += step_latency
+
+        if step.get("type") == "tool_call":
+            tool_calls += 1
+
+        if step.get("type") == "retriever":
+            retrieved_chunks += metadata.get("chunk_count") or 0
+
+        if step_tokens > highest_step_tokens:
+            highest_step_tokens = step_tokens
+            highest_token_step = step.get("name")
+
+        if step_cost > highest_step_cost:
+            highest_step_cost = step_cost
+            highest_cost_step = step.get("name")
+
+    return {
+        "framework": framework,
+        "workflow_name": report.get("name"),
+        "step_count": len(steps),
+        "total_tokens": total_tokens,
+        "prompt_tokens": prompt_tokens,
+        "completion_tokens": completion_tokens,
+        "total_cost_usd": round(total_cost, 8),
+        "total_latency_sec": round(total_latency, 8),
+        "tool_calls": tool_calls,
+        "retrieved_chunks": retrieved_chunks,
+        "highest_token_step": highest_token_step,
+        "highest_step_tokens": highest_step_tokens,
+        "highest_cost_step": highest_cost_step,
+        "highest_step_cost_usd": round(highest_step_cost, 8),
+    }
+
+
+def extract_step_rows(framework: str, report: dict) -> list[dict]:
+    rows = []
+
+    for step in report.get("steps", []):
+        metrics = step.get("metrics") or {}
+        metadata = step.get("metadata") or {}
+
+        rows.append(
+            {
+                "framework": framework,
+                "workflow_name": report.get("name"),
+                "step_name": step.get("name"),
+                "step_type": step.get("type"),
+                "provider": step.get("provider"),
+                "model": step.get("model"),
+                "prompt_tokens": metrics.get("prompt_tokens") or 0,
+                "completion_tokens": metrics.get("completion_tokens") or 0,
+                "total_tokens": metrics.get("total_tokens") or 0,
+                "cost_usd": metrics.get("cost") or 0.0,
+                "latency_sec": metrics.get("latency") or 0.0,
+                "chunk_count": metadata.get("chunk_count"),
+                "tool_name": metadata.get("tool_name"),
+            }
+        )
+
+    return rows
+
+
+def create_markdown_summary(summary_df: pd.DataFrame, output_path: Path) -> None:
+    lines = [
+        "# AgenticLens Framework Benchmark Comparison",
+        "",
+        "Use case: Practical customer support refund workflow.",
+        "",
+        "The workflow includes:",
+        "",
+        "- ticket intent classification",
+        "- query rewriting",
+        "- refund policy retrieval",
+        "- order lookup",
+        "- refund eligibility check",
+        "- customer reply generation",
+        "",
+        "## Summary Results",
+        "",
+        "| Framework | Total Tokens | Prompt Tokens | Completion Tokens | Cost USD | "
+        "Latency Sec | Steps | Tool Calls | Retrieved Chunks | Highest Token Step |",
+        "|---|---:|---:|---:|---:|---:|---:|---:|---:|---|",
+    ]
+
+    for _, row in summary_df.iterrows():
+        lines.append(
+            f"| {row['framework']} | "
+            f"{row['total_tokens']} | "
+            f"{row['prompt_tokens']} | "
+            f"{row['completion_tokens']} | "
+            f"${row['total_cost_usd']:.8f} | "
+            f"{row['total_latency_sec']:.8f} | "
+            f"{row['step_count']} | "
+            f"{row['tool_calls']} | "
+            f"{row['retrieved_chunks']} | "
+            f"{row['highest_token_step']} |"
+        )
+
+    lines.extend(
+        [
+            "",
+            "## Key Finding",
+            "",
+            "The final customer reply step is the highest token-consuming step across "
+            "the benchmark runs.",
+            "",
+            "## Important Note",
+            "",
+            "These results are workload-specific. They should not be treated as a "
+            "universal ranking of frameworks.",
+            "The purpose is to show how AgenticLens can normalize and compare token, "
+            "cost, latency, retrieval, and tool-call metrics across framework "
+            "implementations.",
+        ]
+    )
+
+    output_path.write_text("\n".join(lines), encoding="utf-8")
+
+
+def plot_total_tokens(summary_df: pd.DataFrame) -> None:
+    plt.figure(figsize=(10, 5))
+    plt.bar(summary_df["framework"], summary_df["total_tokens"])
+    plt.title("AgenticLens Benchmark: Total Tokens by Framework")
+    plt.xlabel("Framework")
+    plt.ylabel("Total Tokens")
+    plt.xticks(rotation=30, ha="right")
+    plt.tight_layout()
+
+    output = RESULTS_DIR / "benchmark_tokens_chart.png"
+    plt.savefig(output)
+    plt.close()
+
+    print(f"Saved token chart: {output}")
+
+
+def plot_total_cost(summary_df: pd.DataFrame) -> None:
+    plt.figure(figsize=(10, 5))
+    plt.bar(summary_df["framework"], summary_df["total_cost_usd"])
+    plt.title("AgenticLens Benchmark: Estimated Cost by Framework")
+    plt.xlabel("Framework")
+    plt.ylabel("Estimated Cost USD")
+    plt.xticks(rotation=30, ha="right")
+    plt.tight_layout()
+
+    output = RESULTS_DIR / "benchmark_cost_chart.png"
+    plt.savefig(output)
+    plt.close()
+
+    print(f"Saved cost chart: {output}")
+
+
+def main() -> None:
+    summary_rows = []
+    step_rows = []
+
+    for framework, report_path in REPORTS.items():
+        path = Path(report_path)
+
+        if not path.exists():
+            print(f"Skipping {framework}: report not found at {report_path}")
+            continue
+
+        report = load_report(path)
+
+        summary_rows.append(summarize_report(framework, report))
+        step_rows.extend(extract_step_rows(framework, report))
+
+    if not summary_rows:
+        raise RuntimeError("No reports found. Run AgenticLens profile commands first.")
+
+    summary_df = pd.DataFrame(summary_rows)
+    step_df = pd.DataFrame(step_rows)
+
+    summary_df = summary_df.sort_values(by=["total_tokens", "framework"])
+
+    summary_csv = RESULTS_DIR / "benchmark_results.csv"
+    step_csv = RESULTS_DIR / "benchmark_step_breakdown.csv"
+    summary_md = RESULTS_DIR / "benchmark_summary.md"
+
+    summary_df.to_csv(summary_csv, index=False)
+    step_df.to_csv(step_csv, index=False)
+    create_markdown_summary(summary_df, summary_md)
+
+    plot_total_tokens(summary_df)
+    plot_total_cost(summary_df)
+
+    print("\nBenchmark comparison complete.")
+    print(f"Summary CSV: {summary_csv}")
+    print(f"Step breakdown CSV: {step_csv}")
+    print(f"Markdown summary: {summary_md}")
+
+    print("\nSummary:")
+    print(summary_df.to_string(index=False))
+
+
+if __name__ == "__main__":
+    main()