docs(evaluator): update the evaluator skill for #1071 and #1173 - #1237
Draft
SandyChapman wants to merge 8 commits into
Draft
docs(evaluator): update the evaluator skill for #1071 and #1173#1237SandyChapman wants to merge 8 commits into
SandyChapman wants to merge 8 commits into
Conversation
A task is an evaluation unit; how it runs is a property of the task, not a
different kind of record. The target side already models this — `AgentRunnerTarget`
is a `kind`-discriminated union of codex/fabric/harbor — so the stored side now
matches, and a user manages every evaluation unit in one place regardless of
which runner executes it.
Task content moves under a discriminated `spec`:
- `EvaluatorTaskDefinition` (kind="evaluator") — intent, inputs, reference,
metrics, views
- `HarborTaskDefinition` (kind="harbor") — a reference to the task's packaged
directory in the Files service, plus Harbor's own config
Nested rather than flattened with nullable per-kind fields, so each variant's
required fields stay required and the revision digest covers the spec as a unit;
two kinds with coincidentally similar metadata cannot collide on content.
`kind` is a `Literal`, matching how the runner targets discriminate. The two
definitions live in their own modules under `api/task_definitions/`; the shared
field types they need moved to `api/fields.py`, since the definitions are
imported *by* `schemas` and cannot import back from it.
A single model per kind, rather than a stored/input pair: only `metrics` widens
on the way in, and the service narrows it to references when storing. That keeps
the API surface small at the cost of making the narrowing a service invariant
rather than a type-level one.
`EvaluatorTaskDefinition` gains the grader-only `reference` — held-out ground
truth, surfaced to metrics but never seeded into the agent's workspace. It has
existed on the inline `AgentEvalTaskInput` since #566, where persisting it was
deferred because the stored schemas then lived in the root OpenAPI/SDK; they are
plugin-owned now, so that reason has lapsed. Until this, a taskset-driven run
expanded to an empty reference, so any task needing ground truth the agent cannot
edit had to give up stored tasks and tasksets entirely.
It is covered by the revision digest. The rule: the digest covers anything that
affects a task's execution output or how it is graded, and `reference` decides
what a metric grades against — two revisions that score differently must not
share a digest, or publish-time dedup would collapse them and a pin would stop
fixing the grading. Held out from the *agent*, not from the API: anyone who can
read the task can read it.
Harbor's `config` is the one exclusion, and it does not contradict that rule. It
is a projection of `task.toml`; Harbor reads the real file out of the
materialized archive at run time, and `archive_digest` is authoritative over
every file in that directory. A config change that genuinely alters execution or
grading therefore already moves the digest, while hashing the projection would
make revision history sensitive to Harbor's serialization. That makes
`archive_digest` load-bearing: a Harbor field ever read from the stored record
rather than from the archive would have to be digested.
Harbor specifics:
- One fileset per task, so a task shared by several tasksets is stored once.
- `archive_ref` is shape-validated, so a malformed reference is rejected at
publish rather than surfacing as a download failure mid-run.
- `config` is stored but excluded from the revision digest, as above.
- Which agent runs a task is not stored: that comes from the run's target, so
the same stored task can be evaluated against different agents.
Taskset expansion rejects a `harbor` member rather than projecting it onto an
agent-eval DTO: that content is a directory of files, not fields, so a pure
projection would silently produce a task with no intent and no metrics — an
evaluation that runs and scores nothing. Mixed tasksets stay storable; the
mismatch surfaces at submit as a 422.
The rejection is unconditional, not target-dependent. Storage landed ahead of
the execution bridge, so no target can run a stored `harbor` task yet and the
message says so plainly instead of suggesting the reader find a compatible one.
Bridging the two — and encoding runner/task-kind compatibility declaratively
rather than as an isinstance check here — is AALGO-481.
Note for anyone with existing task rows: this is a breaking schema change with
no migration. Rows stored in the previous flat shape fail validation on read,
which surfaces as a 500 when listing tasks. Clear them before upgrading.
Signed-off-by: Sandy Chapman <schapman@nvidia.com>
`TaskDefinition` is discriminated on `kind`, but both variants defaulted it, so the generated schema left `kind` out of `required` while the validator demanded it. A raw create or replace body without `kind` fails with `union_tag_not_found` — meaning a client generated from that spec would omit the field and 422 on every write. Make `kind` a required field on both definitions, matching how the metric payload DTOs in the same package already declare their discriminator, and regenerate the plugin spec. Tests cover both halves of the mismatch: raw POST and PUT bodies without `kind` are rejected, and the published schema keeps `kind` in `required`. Signed-off-by: Sandy Chapman <schapman@nvidia.com>
…he task docs Review follow-ups on #1071. Reference parsing was duplicated. `nmp.common.entities.utils` already re-exports `nemo_platform_plugin.refs.parse_entity_ref`, which ~10 services and three other plugins use; the evaluator was the last place carrying its own copy under the same name. Delete it and delegate: `parse_subentity_ref` now adds only the `#fragment` that a revisioned entity needs on top of the shared split, and `ENTITY_REF_PATTERN` / `FILESET_REF_PATTERN` move next to the parser and the `FilesetRef` type they describe. `_SUBENTITY_REF_PATTERN` is spliced from the shared constant, so widening what counts as a `workspace/name` widens both shapes at once instead of leaving one behind. One behavior detail this makes explicit: taskset duplicate-detection relied on the old parser silently stripping `#fragment`, so `task-a` and `task-a#<digest>` deduped as one member. The platform parser does not strip, so that path now discards the fragment deliberately. Restore the `CloudpickleMetricPayload` / `InlineMetricPayload` / `MetricPayload` re-exports from `api.schemas`, which `fields.py` promises in its module docstring and lost when they moved. The `manage-tasks-tasksets` revision snippets still passed the pre-`spec` flat shape. `make docs-check-python-snippets` did not catch it because the snippet linter passes ty a rule name that was renamed upstream, so ty answered with `warning[unknown-rule]` and the check failed for every doc regardless of its content. Fix the rule name, fix the snippets, and name both task kinds before the sentence that refers to "both kinds". Type-checking a snippet would not have caught one that type-checks and then fails at run time, nor a documented output gone stale — which is the shape of what review found here. So add an integration test that walks the doc top to bottom against a real platform and asserts the results it claims. No OpenAPI change: the spec regenerates byte-identical. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Sandy Chapman <schapman@nvidia.com>
The evaluator skill's `resources.md` and `plugin_sdk_examples.py` were updated here for the new `spec` shape, which put a `skills/` file in the diff and so put the PR behind the NVSkills gate. That gate cannot currently pass: tier 3 is invoked with `--env-mode local`, whose bubblewrap sandbox fails its smoke test on the runners, so no evaluation runs and the gate blocks on empty coverage. It is an infrastructure problem with the nvcarps pipeline, already reported, and nothing in this repo can resolve it. With no `skills/` file touched, the gate no longer applies to this PR and the storage change can land on its own merits. The skill updates are not lost — they move to a stacked follow-up PR, which can sit behind the gate for as long as it takes without holding this one. Reverting them costs nothing in tests: no test invokes `store_resources`, and the one assertion in `test_skill_examples.py` that pinned the new wording is reverted alongside the content it describes. Known cost while the two are apart: the skill documents the pre-`spec` task shape, which no longer validates. Anyone following the skill in that window writes a task the API rejects. That is the price of unblocking, and it ends when the follow-up lands. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Sandy Chapman <schapman@nvidia.com>
Moving a stored task's content under a discriminated `spec` invalidates the skill example's `TaskInput(intent=..., inputs=..., metrics=...)`, and ty checks `skills/**/*.py`, so the example fails the type gate. The fix for the example is written and sits in #1237. It cannot ride along here: editing any file under `skills/` puts the PR behind the NVSkills gate, and that gate currently cannot pass — tier 3 runs with `--env-mode local`, whose bubblewrap sandbox fails its smoke test on the nvcarps runners, so nothing is evaluated and it blocks on empty coverage. Keeping the example correct and keeping this PR out of the gate are mutually exclusive until that is fixed. Chosen as an override rather than a `[tool.ty.src].exclude` entry, which that list's own header asks contributors not to grow: an override keeps every other rule live on the file and names the two the stale call actually produces, so it cannot quietly widen into cover for unrelated drift. #1237 removes it in the same commit that corrects the example. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Sandy Chapman <schapman@nvidia.com>
Split out of #1071 so that PR is not held behind the NVSkills gate. The gate cannot currently pass for this repo: tier 3 is invoked with `--env-mode local`, whose bubblewrap sandbox fails its smoke test on the nvcarps runners, so nothing is evaluated and the gate blocks on empty coverage. This PR carries the whole cost of that, and can wait for the infrastructure fix without blocking the storage change. `store_resources` moves to the discriminated `spec`, and `resources.md` teaches held-out ground truth on a *stored* task rather than an inline one — the skill steered users to `AgentEvalTaskInput` only because the stored schema had no `reference` field, and #1071 gives it one. Routing them back to inline would cost them tasksets and revision pinning for no reason. Until this lands, the published skill documents the pre-`spec` shape, which no longer validates against #1071. Merge promptly once the gate is healthy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Sandy Chapman <schapman@nvidia.com>
The override added alongside #1071 exists only because the skill example was left stale there. This PR corrects the example, so the exemption expires with it — ty passes on this branch with no override at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Sandy Chapman <schapman@nvidia.com>
SandyChapman
force-pushed
the
evaluator-skill-spec-shape/schapman
branch
from
August 11, 2026 19:13
142e783 to
b12a97a
Compare
SandyChapman
added a commit
that referenced
this pull request
Aug 11, 2026
Retiring `run_sync`/`submit` in favour of `run_dataset_sync`/`evaluate_dataset` meant updating the evaluator skill to match, which put six `skills/` files in the diff and so put this PR behind the NVSkills gate. That gate cannot currently pass: tier 3 is invoked with `--env-mode local`, whose bubblewrap sandbox fails its smoke test on the nvcarps runners, so nothing is evaluated and it blocks on empty coverage. It is an infrastructure problem, already reported, and nothing in this repo can resolve it. With no `skills/` file touched, the gate no longer applies and the backend contract change can land on its own merits. The skill updates move to #1237, which can sit behind the gate for as long as it takes. Unlike the equivalent split on #1071, this one has a cost worth naming. The skill's `evaluate_standalone` example is *executed* by `test_skill_standalone_example_scores_pass_and_failure`, and the reverted example calls the retired `Evaluator.run_sync`, so the test now fails for a real reason: the shipped example is genuinely broken against this refactor. It is skipped rather than deleted, with the reason and the restoring PR named in the marker, so the gap is visible and expires. The other 29 tests in that file still run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Sandy Chapman <schapman@nvidia.com>
…change #1173 retires `run_sync`/`submit` for `run_dataset_sync`/`evaluate_dataset`, which needed the same six skill files updated and so put that PR behind the NVSkills gate too. Rather than open a second skills PR that waits on the same broken gate, its skill updates join this one. `plugin_sdk_examples.py` is the one file both PRs touch, and they touch different functions — #1071 moves `store_resources` to the discriminated `spec`, #1173 moves `evaluate_standalone` and `submit_and_collect` to the dataset API — so the two are merged here rather than one overwriting the other. `test_skill_examples.py` likewise carries both sets of assertions. `test_skill_standalone_example_scores_pass_and_failure` fails on this branch and is expected to: it calls `run_dataset_sync`, which exists on #1173's branch, not on this PR's #1071 base. It passes once #1173 lands. It is left failing rather than skipped, so that it is re-verified for real instead of quietly staying off. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Sandy Chapman <schapman@nvidia.com>
SandyChapman
changed the title
spec shape
ryana
pushed a commit
to ryana/nemo-platform
that referenced
this pull request
Aug 12, 2026
…IA-NeMo#1071) * feat(evaluator): make a stored task runner-polymorphic via kind A task is an evaluation unit; how it runs is a property of the task, not a different kind of record. The target side already models this — `AgentRunnerTarget` is a `kind`-discriminated union of codex/fabric/harbor — so the stored side now matches, and a user manages every evaluation unit in one place regardless of which runner executes it. Task content moves under a discriminated `spec`: - `EvaluatorTaskDefinition` (kind="evaluator") — intent, inputs, reference, metrics, views - `HarborTaskDefinition` (kind="harbor") — a reference to the task's packaged directory in the Files service, plus Harbor's own config Nested rather than flattened with nullable per-kind fields, so each variant's required fields stay required and the revision digest covers the spec as a unit; two kinds with coincidentally similar metadata cannot collide on content. `kind` is a `Literal`, matching how the runner targets discriminate. The two definitions live in their own modules under `api/task_definitions/`; the shared field types they need moved to `api/fields.py`, since the definitions are imported *by* `schemas` and cannot import back from it. A single model per kind, rather than a stored/input pair: only `metrics` widens on the way in, and the service narrows it to references when storing. That keeps the API surface small at the cost of making the narrowing a service invariant rather than a type-level one. `EvaluatorTaskDefinition` gains the grader-only `reference` — held-out ground truth, surfaced to metrics but never seeded into the agent's workspace. It has existed on the inline `AgentEvalTaskInput` since NVIDIA-NeMo#566, where persisting it was deferred because the stored schemas then lived in the root OpenAPI/SDK; they are plugin-owned now, so that reason has lapsed. Until this, a taskset-driven run expanded to an empty reference, so any task needing ground truth the agent cannot edit had to give up stored tasks and tasksets entirely. It is covered by the revision digest. The rule: the digest covers anything that affects a task's execution output or how it is graded, and `reference` decides what a metric grades against — two revisions that score differently must not share a digest, or publish-time dedup would collapse them and a pin would stop fixing the grading. Held out from the *agent*, not from the API: anyone who can read the task can read it. Harbor's `config` is the one exclusion, and it does not contradict that rule. It is a projection of `task.toml`; Harbor reads the real file out of the materialized archive at run time, and `archive_digest` is authoritative over every file in that directory. A config change that genuinely alters execution or grading therefore already moves the digest, while hashing the projection would make revision history sensitive to Harbor's serialization. That makes `archive_digest` load-bearing: a Harbor field ever read from the stored record rather than from the archive would have to be digested. Harbor specifics: - One fileset per task, so a task shared by several tasksets is stored once. - `archive_ref` is shape-validated, so a malformed reference is rejected at publish rather than surfacing as a download failure mid-run. - `config` is stored but excluded from the revision digest, as above. - Which agent runs a task is not stored: that comes from the run's target, so the same stored task can be evaluated against different agents. Taskset expansion rejects a `harbor` member rather than projecting it onto an agent-eval DTO: that content is a directory of files, not fields, so a pure projection would silently produce a task with no intent and no metrics — an evaluation that runs and scores nothing. Mixed tasksets stay storable; the mismatch surfaces at submit as a 422. The rejection is unconditional, not target-dependent. Storage landed ahead of the execution bridge, so no target can run a stored `harbor` task yet and the message says so plainly instead of suggesting the reader find a compatible one. Bridging the two — and encoding runner/task-kind compatibility declaratively rather than as an isinstance check here — is AALGO-481. Note for anyone with existing task rows: this is a breaking schema change with no migration. Rows stored in the previous flat shape fail validation on read, which surfaces as a 500 when listing tasks. Clear them before upgrading. Signed-off-by: Sandy Chapman <schapman@nvidia.com> * fix(evaluator): require the kind discriminator on a task definition `TaskDefinition` is discriminated on `kind`, but both variants defaulted it, so the generated schema left `kind` out of `required` while the validator demanded it. A raw create or replace body without `kind` fails with `union_tag_not_found` — meaning a client generated from that spec would omit the field and 422 on every write. Make `kind` a required field on both definitions, matching how the metric payload DTOs in the same package already declare their discriminator, and regenerate the plugin spec. Tests cover both halves of the mismatch: raw POST and PUT bodies without `kind` are rejected, and the published schema keeps `kind` in `required`. Signed-off-by: Sandy Chapman <schapman@nvidia.com> * refactor(evaluator): adopt the platform's entity-ref parser and fix the task docs Review follow-ups on NVIDIA-NeMo#1071. Reference parsing was duplicated. `nmp.common.entities.utils` already re-exports `nemo_platform_plugin.refs.parse_entity_ref`, which ~10 services and three other plugins use; the evaluator was the last place carrying its own copy under the same name. Delete it and delegate: `parse_subentity_ref` now adds only the `#fragment` that a revisioned entity needs on top of the shared split, and `ENTITY_REF_PATTERN` / `FILESET_REF_PATTERN` move next to the parser and the `FilesetRef` type they describe. `_SUBENTITY_REF_PATTERN` is spliced from the shared constant, so widening what counts as a `workspace/name` widens both shapes at once instead of leaving one behind. One behavior detail this makes explicit: taskset duplicate-detection relied on the old parser silently stripping `#fragment`, so `task-a` and `task-a#<digest>` deduped as one member. The platform parser does not strip, so that path now discards the fragment deliberately. Restore the `CloudpickleMetricPayload` / `InlineMetricPayload` / `MetricPayload` re-exports from `api.schemas`, which `fields.py` promises in its module docstring and lost when they moved. The `manage-tasks-tasksets` revision snippets still passed the pre-`spec` flat shape. `make docs-check-python-snippets` did not catch it because the snippet linter passes ty a rule name that was renamed upstream, so ty answered with `warning[unknown-rule]` and the check failed for every doc regardless of its content. Fix the rule name, fix the snippets, and name both task kinds before the sentence that refers to "both kinds". Type-checking a snippet would not have caught one that type-checks and then fails at run time, nor a documented output gone stale — which is the shape of what review found here. So add an integration test that walks the doc top to bottom against a real platform and asserts the results it claims. No OpenAPI change: the spec regenerates byte-identical. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Sandy Chapman <schapman@nvidia.com> * chore(evaluator): move the skill updates out of this PR The evaluator skill's `resources.md` and `plugin_sdk_examples.py` were updated here for the new `spec` shape, which put a `skills/` file in the diff and so put the PR behind the NVSkills gate. That gate cannot currently pass: tier 3 is invoked with `--env-mode local`, whose bubblewrap sandbox fails its smoke test on the runners, so no evaluation runs and the gate blocks on empty coverage. It is an infrastructure problem with the nvcarps pipeline, already reported, and nothing in this repo can resolve it. With no `skills/` file touched, the gate no longer applies to this PR and the storage change can land on its own merits. The skill updates are not lost — they move to a stacked follow-up PR, which can sit behind the gate for as long as it takes without holding this one. Reverting them costs nothing in tests: no test invokes `store_resources`, and the one assertion in `test_skill_examples.py` that pinned the new wording is reverted alongside the content it describes. Known cost while the two are apart: the skill documents the pre-`spec` task shape, which no longer validates. Anyone following the skill in that window writes a task the API rejects. That is the price of unblocking, and it ends when the follow-up lands. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Sandy Chapman <schapman@nvidia.com> * chore(lint): scope a ty override to the stale evaluator skill example Moving a stored task's content under a discriminated `spec` invalidates the skill example's `TaskInput(intent=..., inputs=..., metrics=...)`, and ty checks `skills/**/*.py`, so the example fails the type gate. The fix for the example is written and sits in NVIDIA-NeMo#1237. It cannot ride along here: editing any file under `skills/` puts the PR behind the NVSkills gate, and that gate currently cannot pass — tier 3 runs with `--env-mode local`, whose bubblewrap sandbox fails its smoke test on the nvcarps runners, so nothing is evaluated and it blocks on empty coverage. Keeping the example correct and keeping this PR out of the gate are mutually exclusive until that is fixed. Chosen as an override rather than a `[tool.ty.src].exclude` entry, which that list's own header asks contributors not to grow: an override keeps every other rule live on the file and names the two the stale call actually produces, so it cannot quietly widen into cover for unrelated drift. NVIDIA-NeMo#1237 removes it in the same commit that corrects the example. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Sandy Chapman <schapman@nvidia.com> --------- Signed-off-by: Sandy Chapman <schapman@nvidia.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
SandyChapman
added a commit
that referenced
this pull request
Aug 12, 2026
Retiring `run_sync`/`submit` in favour of `run_dataset_sync`/`evaluate_dataset` meant updating the evaluator skill to match, which put six `skills/` files in the diff and so put this PR behind the NVSkills gate. That gate cannot currently pass: tier 3 is invoked with `--env-mode local`, whose bubblewrap sandbox fails its smoke test on the nvcarps runners, so nothing is evaluated and it blocks on empty coverage. It is an infrastructure problem, already reported, and nothing in this repo can resolve it. With no `skills/` file touched, the gate no longer applies and the backend contract change can land on its own merits. The skill updates move to #1237, which can sit behind the gate for as long as it takes. Unlike the equivalent split on #1071, this one has a cost worth naming. The skill's `evaluate_standalone` example is *executed* by `test_skill_standalone_example_scores_pass_and_failure`, and the reverted example calls the retired `Evaluator.run_sync`, so the test now fails for a real reason: the shipped example is genuinely broken against this refactor. It is skipped rather than deleted, with the reason and the restoring PR named in the marker, so the gap is visible and expires. The other 29 tests in that file still run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Sandy Chapman <schapman@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Carries the evaluator-skill updates from both #1071 and #1173, so neither of those is held behind the NVSkills gate.
Why this exists
Both PRs change the API the skill documents, so both had to update
skills/— which put each of them behindrequire-nvskills. That gate cannot currently pass for this repo: tier 3 is invoked with--env-mode local, whose bubblewrap sandbox fails its smoke test on the nvcarps runners, so no agent evaluation runs andgate:content:v1blocks on empty coverage. The message reads as a content rejection (canonical Tier 3 coverage is invalid) but nothing was evaluated —execution_status: skipped,agents_run: []. Reproduced across runner classes, reported in the nvcarps support channel, and not fixable from this repo.One skills PR absorbs the wait for both.
What's here
store_resourcesmoves tospec=EvaluatorTaskDefinition(kind="evaluator", ...)resources.mdteaches held-outreferenceon a stored task, not an inline oneevaluate_standalone/submit_and_collectmove torun_dataset_sync/evaluate_datasetSKILL.md,api-auth.md,execution.md,metric-selection.md,troubleshooting.mdfollow the same renametyoverride #1071 added for the then-stale exampleplugin_sdk_examples.pyis the one file both PRs touch. They touch different functions, so both sets of edits are merged here rather than one overwriting the other;test_skill_examples.pycarries both sets of assertions.Expected red until #1173 lands
test_skill_standalone_example_scores_pass_and_failurefails on this branch because it callsrun_dataset_sync, which exists on #1173's branch and not on this PR's #1071 base. It passes once #1173 merges.It is deliberately left failing rather than skipped, so it gets re-verified for real instead of quietly staying off. (#1173 itself skips this same test, with a marker pointing here — un-skipping it is part of landing this PR.)
Known cost while unmerged
The published skill documents the pre-
spectask shape and the retiredrun_sync/submitmethods. Anyone following it in that window writes code the API rejects. Merge promptly once the gate is healthy.Sequencing
harbor-task-kind/schapman(feat(evaluator): make a stored task runner-polymorphic via kind #1071), because the examples needEvaluatorTaskDefinition. Retarget tomainonce feat(evaluator): make a stored task runner-polymorphic via kind #1071 merges.pytest plugins/nemo-evaluator/tests/test_skill_examples.pyand confirm 30 passed before undrafting.test_skill_standalone_example_scores_pass_and_failureif refactor(evaluator)!: unify the backend contract on evaluate/evaluate_dataset #1173's skip marker survives the merge./nvskills-cionly after all edits are final — the signing bot pushes a signature commit on top of the branch, and any later push re-breaks the gate.🤖 Generated with Claude Code