refactor(evaluator)!: drop the plugin's local execution path - #1262
Draft
SandyChapman wants to merge 1 commit into
Draft
Conversation
`SKILL.md` already told callers not to build on `client.evaluator.run()` because it was being retired. This removes it, along with the executor methods that existed only to serve it: `run_local`, `evaluate_remote`, `evaluate`, and `evaluate_benchmark` on both the sync and async executors. Nothing in production called it. The plugin's remote path is `submit`, and callers wanting local execution use the standalone `nemo_evaluator_sdk.Evaluator` directly, which is what the skill already recommends. Removing it orphans a supporting cast, so that goes too: `sdk/fs_utils.py` entirely, since `local_artifact_path` was only reachable from `local_result_path` and `EvaluatorLocalRunResult` only from tests of the removed methods; and `filter_evaluation_result`/`filter_benchmark_result` from `sdk/utils.py`, which keeps `filter_aggregate_scores` for `job_resources`. Two `create` tests patched `_executor.asyncio.to_thread` to prove creation never bridges through a thread. With local execution gone the module has no asyncio import at all, so they now assert its absence — the property holds by construction rather than by one observed call. The CLI `run` verb is unaffected: it comes from the job framework's `NemoJobScheduler.run_local`, which is a different path from the executor method of the same name. BREAKING CHANGE: `client.evaluator.run()` is removed from both the sync and async plugin resources. Use `submit` for platform evaluation, or `nemo_evaluator_sdk.Evaluator` for local execution. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Sandy Chapman <schapman@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1202 — review that first; this PR's diff is against its branch and should be retargeted to
mainonce it lands.SKILL.mdalready told callers not to build onclient.evaluator.run()because it was being retired. This removes it, plus the executor methods that existed only to serve it —run_local,evaluate_remote,evaluate,evaluate_benchmarkon both executors.Nothing in production called it. The remote path is
submit; local execution isnemo_evaluator_sdk.Evaluator, which is what the skill already points at.What goes
_executor.pyresources.pyrun(4 overloads × sync/async)fs_utils.pyutils.pyfilter_evaluation_result,filter_benchmark_resulttest_sdk.pyNet: 870 deletions, 6 insertions.
Notes for review
runverb is unaffected — it comes from the job framework'sNemoJobScheduler.run_local, a different path that happens to share a name.createtests patched_executor.asyncio.to_threadto prove creation never bridges through a thread. With local execution gone the module no longer imports asyncio, so they now assert its absence — the property holds by construction rather than by one observed call.test_*_resource_does_not_expose_backend_methodswere kept: they are negative assertions that only get more true.tyreports the same 4 pre-existing diagnostics as the base, at shifted lines. No new ones.Follow-up
SKILL.mdstill says the path "is being retired… even though--helpstill lists it", which now misdescribes the SDK. Left untouched deliberately, since skill updates were pulled out of the previous PR — worth a separate change.🤖 Generated with Claude Code