A single humanitarian data research pipeline — implemented three times, once per framework — to compare how LangGraph, CrewAI, and AutoGen actually handle multi-agent orchestration, tool use, and conditional retry logic in practice.
Given a natural-language humanitarian question (e.g. "What is the flood risk in Assam?"), the system autonomously:
- Scouts HDX (Humanitarian Data Exchange) for a relevant dataset, using LLM-driven keyword search — not a hardcoded dataset ID
- Validates whether the found dataset is actually relevant and sufficient to answer the question, and if not, sends control back to Scout with specific feedback on what's missing (a genuine retry loop, capped at 3 attempts)
- Analyzes the confirmed dataset(s) using a real data-querying tool — filtering and aggregating actual numbers, not guessing from a sample
- Writes a structured markdown report (Executive Summary, Key Findings, Risk Assessment, Recommendations)
The same four-agent design — Scout, Validator, Analyst, Writer — is implemented once per framework, using the exact same underlying tool functions (shared_tools/), so the comparison isolates framework differences, not logic differences.
Why this design, specifically: an earlier single-agent version of this pipeline (one persona doing sequential steps) was a legitimate but different thing — not genuinely multi-agent. This version was deliberately redesigned so each agent owns a narrow responsibility and its own tools, and the graph/conversation has real conditional branching: Validator can loop control back to Scout with targeted feedback, not just pass data forward in a fixed sequence.
| LangGraph | CrewAI (Flows) | AutoGen (AG2) | |
|---|---|---|---|
| Mental model | Explicit graph — nodes and edges you wire yourself | Agents + Tasks + Flow steps, orchestration partly handled for you | Agents converse in a shared chat loop; a custom function decides who speaks next |
| Control level | Highest — you write the tool-calling loop by hand, see every message | Medium — Crew handles the internal tool loop; you control step sequencing via Flow | Medium-low — conversation history is shared automatically; explicit control requires injecting messages yourself |
| Retry/loop support | Native (add_conditional_edges) — this is what LangGraph is built for |
Not supported by plain Crew/Process.sequential; requires the separate Flows API (@start/@listen/@router) to genuinely loop |
Native, via a custom speaker_selection_method function — arguably the most direct one-function expression of the same idea |
| Structured output | Forced via tool_choice — clean, reliable, no parsing needed |
Forced via output_pydantic on a Task — clean, reliable |
No first-class equivalent found; relied on a "fake tool" whose arguments are the structured fields, then parsed the result back out of conversation history with regex — noticeably rougher |
| Code verbosity | Highest — every tool-calling loop, every state transition is explicit | Medium — Agent/Task/Crew boilerplate per step, but the internal ReAct loop is hidden | Lowest per-agent, but the custom routing function ends up carrying logic that the other two frameworks give you for free |
| Best-use-case | Production systems needing full visibility, debuggability, and precise control over agent handoffs | Teams wanting quick-to-write agent personas with less orchestration code, and multi-step Crews without deep custom control needs | Conversational/collaborative agent setups where "agents talking to each other" is a natural fit for the problem — less natural for a fixed pipeline like this one |
Easiest to build: CrewAI's basic Agent/Task/Crew model, for a single linear pass — but that ease disappears the moment real looping is needed. Plain Crew has no genuine retry mechanism; ConditionalTask only supports a single conditional skip/run, not a bounded multi-round loop. Getting real looping required CrewAI Flows — a separate, newer part of the framework with its own decorator-based syntax (@start/@listen/@router) and its own state model. Once in Flows, the amount of code converges toward what LangGraph already required, just organized differently.
Most control: LangGraph, without question. Every message, every tool call, every state transition is visible and explicit. This is also its cost — nothing is hidden, so nothing is free; the ReAct tool-calling loop had to be written by hand for both Scout and Analyst.
Most surprising gap: AutoGen's structured-output story. LangGraph's tool_choice and CrewAI's output_pydantic both give a clean, reliable way to force an agent's final answer into a specific shape. AutoGen has no first-class equivalent for this — the workaround here was defining a "fake tool" (submit_validation_result) whose arguments are the structured fields, then extracting them back out of the conversation's function-call result using regex on a formatted string. It works, but it's the roughest edge of the three frameworks for this specific need.
Genuinely interesting finding: AutoGen's custom_speaker_selection_func is arguably the most direct expression of conditional routing among the three — one plain Python function, no decorators, no separate Flow layer required. The tradeoff is that AutoGen's conversation-sharing model means context (like Validator's judgment, or which columns a dataset has) isn't automatically well-structured for a later agent to consume — it had to be explicitly injected into the shared message history at the right moments, rather than being handed forward as clean state (as LangGraph and CrewAI both do).
What I'd choose for production, and why: LangGraph. The explicitness that makes it more verbose to write is exactly what makes a production agentic system debuggable, observable (LangSmith tracing, as used in a separate live project), and safe to reason about when something goes wrong at 2am. CrewAI Flows is a reasonable second choice if the team values faster initial development and is comfortable with a somewhat newer, less battle-tested part of the framework. AutoGen's conversational model is compelling for genuinely collaborative, open-ended agent interactions, but for a fixed pipeline with a real retry requirement, its lack of native structured output made it the framework that needed the most workaround code, not the least.
A design decision considered and rejected: an "upfront query-planning" agent that would decide, before any searching, how many datasets a question needs. Rejected because Validator's reactive sufficiency check — grounded in what data actually exists on HDX — already solves the same problem without guessing ahead of time, at lower cost.
This is a learning and comparison project, not a production deployment. It is intentionally small: no persistent database, no authentication, no deployment infrastructure. The goal is to demonstrate genuine hands-on understanding of three different multi-agent orchestration paradigms solving the identical problem — not to ship a production humanitarian data service.
Not implemented (explicitly out of scope):
- ReliefWeb as a second data source (HDX only, by design — kept the project buildable)
- CI/CD, containerization, or cloud deployment for this specific project
- A shared UI across the three versions
multi-agent-framework-comparison/
├── shared_tools/ # framework-agnostic HDX fetch + query functions
├── langgraph_version/ # explicit StateGraph, manual ReAct loops
├── crewai_version/ # CrewAI Flows — Agent/Task/Crew per step, @start/@listen/@router
├── autogen_version/ # AG2 GroupChat with a custom speaker-selection function
└── README.md
Python 3.11 · Claude Haiku (claude-haiku-4-5-20251001) via the Anthropic API, used consistently across all three frameworks · LangGraph · CrewAI (with Flows) · AG2 (AutoGen) · pandas · HDX CKAN API