Add AutoGen integration (auto-instrumentation + Penelope Target) - #2008
Add AutoGen integration (auto-instrumentation + Penelope Target)#2008Ayush7614 wants to merge 2 commits into
Conversation
Implement AutoGenIntegration with generate_reply patching for automatic telemetry, add AutoGenTarget for multi-turn Penelope testing, and add unit tests for issue rhesis-ai#1866.
![peqy[bot]](https://avatars.githubusercontent.com/in/1930020?s=60&v=4){kind=small}
There was a problem hiding this comment.
AutoGen integration/target looks solid overall and tests cover the basics.
Main things to address:
- Penelope AutoGenTarget currently drops
filessilently (either support or explicitly error). - AutoGenTarget leaves history mutated on exceptions.
- Token attributes helper drops explicit 0s.
penelope[all]doesn’t include the newautogenextra.
Found 4 issues (0 critical, 4 improvements).
| self, | ||
| message: str, | ||
| conversation_id: Optional[str] = None, | ||
| files: Optional[List] = None, |
There was a problem hiding this comment.
Improvement files is part of the Target interface, but it’s currently ignored.
Fix: either explicitly reject with a clear error (so callers don’t think attachments were used), or map/forward
filesinto AutoGen’s expected input format if supported.
There was a problem hiding this comment.
✅ Addressed in 0dee4a7: send_message() now explicitly rejects files with a clear error and does not call the agent.
| "session_messages_count": len(history), | ||
| }, | ||
| ) | ||
| except Exception as exc: |
There was a problem hiding this comment.
Improvement On exception, the user message has already been appended to history, so a transient failure will leave the conversation state “ahead” by one turn.
Fix: consider rolling back the last user entry on failure (or explicitly record an error turn), and include
conversation_id=session_keyin the errorTargetResponsefor consistency.
There was a problem hiding this comment.
✅ Addressed in 0dee4a7: on exception the last user turn is rolled back and the error TargetResponse includes conversation_id=session_key.
| return | ||
| try: | ||
| prompt, completion, total = extract_token_usage(usage) | ||
| if prompt: |
There was a problem hiding this comment.
Improvement set_token_attributes() only sets attributes when counts are truthy, so explicit 0 token values are dropped.
Other integrations in this repo tend to set 0s when usage is present.
Fix: use
is not Nonechecks (or set all 3 when any usage is present) so0is recorded consistently.
There was a problem hiding this comment.
✅ Addressed in 0dee4a7: set_token_attributes() now records explicit 0 token counts when usage is provided (test added).
| autogen = [ | ||
| "pyautogen>=0.2.0", | ||
| ] | ||
| all = [ |
There was a problem hiding this comment.
Improvement autogen optional extra isn’t included in the all extra.
Fix: add
pyautogen>=0.2.0toallifallis meant to truly enable every target/integration.
There was a problem hiding this comment.
✅ Addressed in 0dee4a7: pyautogen>=0.2.0 is now included in the all extra.
Reject unsupported file attachments explicitly, roll back session history on generate_reply failures, record zero token counts, and include pyautogen in penelope[all].
|
Reviewed latest commit 0dee4a7. The previously flagged issues (files handling, rollback on exception + conversation_id, token 0s, No new issues spotted in the updated diff. Ship it. |
|
cc: @harry-rhesis |
1 participant
Purpose
Complete AutoGen SDK auto-instrumentation and add a Penelope target so users can observe and test AutoGen agents automatically.
Closes #1866
What Changed
AutoGenIntegrationwithgenerate_reply/a_generate_replypatching and OTel spans (model, tokens, latency, I/O)tracing_helpers.pyfor framework integration span utilitiesAutoGenTargetwrappingConversableAgentwith per-conversation_idhistoryAutoGenTargetfrom Penelope targets and addedautogenoptional dependencyTesting
cd sdk && uv run pytest ../tests/sdk/telemetry/integrations/test_autogen_integration.py ../tests/sdk/telemetry/integrations/test_langgraph_integration.py -qcd penelope && uv run pytest ../tests/penelope/targets/test_autogen.py -q