fix(synthesis): Strategy 0 in _parse_response returns prematurely on single JSON objects (#94)#175
Merged
himanshu231204 merged 3 commits intoJul 17, 2026
Conversation
Both QuestionGenerator and AdversarialTestCaseGenerator had a bug in their _parse_response methods where Strategy 0 (single JSON object parsing) would return immediately after finding one question, instead of continuing to other strategies to find more questions. Changes: - Removed premature eturn test_cases in Strategy 0 for both files - Added deduplication via seen_questions set and _add_test_case helper - All parsing strategies now use the helper to avoid duplicate questions Fixes issue #94 where requesting N questions but receiving a single JSON object would only return 1 question instead of attempting other parsing strategies.
|
🎉 Congratulations @himanshu231204! Your pull request has been successfully merged into main. 🚀 Thank you for contributing to OpenAgentHQ and helping improve the project. We truly appreciate your contribution and hope to see you back with more amazing PRs! Happy Open Sourcing! ❤️ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes a bug in both
QuestionGeneratorandAdversarialTestCaseGeneratorwhere Strategy 0 in_parse_responsewould parse a single JSON object (non-array LLM response), create ONE test case, and return immediately — instead of continuing to other strategies to find more questions.Root cause: When the LLM returns a single JSON object like
{"question": "...", "answer": "..."}instead of a JSON array, Strategy 0 would successfully parse it but thenreturn test_casesprematurely, skipping Strategies 1-4 that could extract additional questions from the same response.Fix applied to both files:
returnin Strategy 0 — now it adds the test case and continues to other strategiesseen_questionsset and_add_test_casehelper to prevent duplicates when multiple strategies find the same questionType of Change
Related Issues
Closes #94
How Has This Been Tested?
uv run pytest tests/unit/test_synthesis/ -v)uv run ruff check .)uv run mypy openagent_eval/)Checklist
Additional Notes
All 69 synthesis tests pass. The fix is minimal and focused — only the
_parse_responsemethods inquestion_gen.pyandadversarial.pywere modified.