test: add unit tests for the synth CLI command (fixes #102)#174
Merged
Conversation
Cover argument parsing and wiring for `oaeval synth` using typer's CliRunner, mocking the SyntheticDataGenerator boundary so no LLM or network call is made: - corpus/text routing to generate / generate_from_text - --count (incl. default of 10), --adversarial + --types parsing - --chunk-size / --chunk-overlap / --max-concurrent constructor wiring - --llm-provider / --llm-model provider selection - usage errors (missing/both corpus+text, invalid adversarial type) -> exit 2 - SynthesisError -> exit 1 - --output writes the dataset to disk Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
🎉 Congratulations @Nitjsefnie! Your pull request has been successfully merged into main. 🚀 Thank you for contributing to OpenAgentHQ and helping improve the project. We truly appreciate your contribution and hope to see you back with more amazing PRs! Happy Open Sourcing! ❤️ |
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #102.
Premise check against current code (this issue batch is dated 2026-07-14)
openagent_eval/cli/commands/synth.pyexists and is registered asapp.command(name="synth")— the issue's premise holds.--count/-n, not--num-questions. Tests follow the code.tests/unit/test_cli/test_synth.pyalready existed with a single_create_providerhelper test (not CLI-level). This PR extends that file (existing test preserved) rather than colliding with a new one.What's covered (12 new tests, CliRunner +
strip_ansi, matching the existing CLI-test conventions)generate/generate_from_text(mockedSyntheticDataGeneratorboundary — no LLM or network is ever touched)--countwiring incl. the default of 10;--adversarial+--typesparsing; invalid type → exit 2--chunk-size/--chunk-overlap/--max-concurrentconstructor wiring--llm-provider/--llm-modelprovider selection (offline via the mock provider)SynthesisError→ exit 1--outputwrites the dataset file (real file intmp_path, contents asserted)Verification
synth.py(flag rename,Exit(2)→Exit(0),count=1, default 10→5, droppedadversarial_types,chunk_size=999, wrong model,_save_output→pass, …) — each killed by exactly the test that claims to pin it. Two of the twelve were independently re-run before opening this PR (default-count and output-write): both fail on the mutant, pass restored.uv run pytest tests/unit -q: baseline on clean main 1 failed, 839 passed, 4 skipped → with this PR 1 failed, 851 passed, 4 skipped. The single failure is identical and pre-existing (test_metrics/test_generation.py::TestBERTScore::test_identical), unrelated to synth.git diff openagent_eval/is empty). Env note:uv sync --all-extraswas used (plainuv syncleaves optional deps unresolved), and its unrelateduv.lockmarker churn was reverted out of the commit.🤖 This PR was written with AI assistance (Claude), directed and verified by a human-supervised workflow.