feat(evaluator): model evaluate wire publish_to_intake() to API - #1152
Merged
Conversation
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe evaluator now supports optional Intake publication for dataset-row results. It adds strict publication schemas, stable row-result adaptation, target identity resolution, post-persistence publication, local job payload serialization, and unit and integration coverage. ChangesEvaluator Intake publication
Sequence Diagram(s)sequenceDiagram
participant EvaluationJob
participant publish_row_eval_result
participant row_result_to_agent_eval_result
participant publish_agent_eval_result
participant Intake
EvaluationJob->>publish_row_eval_result: completed row results and publication settings
publish_row_eval_result->>row_result_to_agent_eval_result: row results and identity field
row_result_to_agent_eval_result-->>publish_row_eval_result: AgentEvalResult
publish_row_eval_result->>publish_agent_eval_result: adapted result and target identity
publish_agent_eval_result->>Intake: publish evaluation result
Intake-->>EvaluationJob: publication outcome
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 2📝 Generate docstrings 💡
🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (5)
plugins/nemo-evaluator/tests/intake/test_publish.py (1)
117-128: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd a test for the missing
started_atpath.
publish_to_intakenow raisesPublishErrorwhenresult.metadata.started_atis unset. No test covers that branch. Add one that builds the result withRunMetadata()and asserts the raise.💚 Proposed test
async def test_publish_requires_started_at() -> None: result = AgentEvalResult( run_id="run-1", tasks=[], trials=[_trial("t-1")], scores=[], summary=AgentEvalSummary(), metadata=RunMetadata(), ) with pytest.raises(PublishError, match="started_at is unset"): await publish_to_intake(result, platform=_platform(), experiment_id="exp", workspace="ws")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-evaluator/tests/intake/test_publish.py` around lines 117 - 128, Add a test for the missing started_at branch in publish_to_intake, constructing an AgentEvalResult with RunMetadata() and representative trial data, then assert that awaiting publish_to_intake raises PublishError with the expected “started_at is unset” message.plugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.py (1)
21-21: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider moving
target_agent_identityout ofagent_spec.This PR moved the publication spec classes into a neutral module because both eval specs share them.
target_agent_identityhas the same property: it resolves bareModel/AgentBasetargets that only the row spec uses, andevaluate.pypluspublication.pynow importagent_specsolely for it. Placing it next topublication_spec.pyapplies the same reasoning and removes the remaining cross-spec dependency.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.py` at line 21, Move target_agent_identity from agent_spec into the neutral module alongside publication_spec.py, then update evaluate.py and publication.py to import it from the new location. Remove the old definition and preserve its existing behavior and callers.plugins/nemo-evaluator/src/nemo_evaluator/jobs/publication.py (2)
206-229: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winExtract the duplicated failure branching; the missing-
run_idpath logs nothing.Both failure blocks repeat the
requiredraise-or-return logic. The first block (206-215) omits thelogger.warningthat theRowIdentityErrorblock andpublish_agent_eval_result.failboth emit, so an optional publication failure caused by a missing job id leaves no log trace.♻️ Proposed refactor
+ def fail(error: str, cause: Exception | None = None) -> PublicationOutcome: + outcome = _failed(spec.evaluation_id, error, None) + if spec.required: + raise PublicationFailedError(outcome) from cause + logger.warning("Publication to Intake failed for evaluation %r: %s", spec.evaluation_id, error) + return outcome + if run_id is None: - outcome = _failed( - spec.evaluation_id, + return fail( "No job id to publish under (platformless local run); a dataset-driven evaluation takes " - "its run identity from the job.", - None, + "its run identity from the job." ) - if spec.required: - raise PublicationFailedError(outcome) - return outcome try: adapted = row_result_to_agent_eval_result( result, run_id=run_id, started_at=started_at, test_case_id_field=spec.test_case_id_field, ) except RowIdentityError as error: - outcome = _failed(spec.evaluation_id, str(error), None) - if spec.required: - raise PublicationFailedError(outcome) from error - logger.warning("Publication to Intake failed for evaluation %r: %s", spec.evaluation_id, error) - return outcome + return fail(str(error), error)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-evaluator/src/nemo_evaluator/jobs/publication.py` around lines 206 - 229, Refactor the failure handling in the publication flow around the missing run_id and RowIdentityError branches into a shared raise-or-return helper or equivalent, preserving PublicationFailedError behavior for required evaluations. Ensure the missing-run_id path also emits the same warning log as other publication failures before returning for optional evaluations, while retaining each branch’s existing outcome details.
31-31: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winImport the exceptions from
nemo_platform.NeMoPlatformErrorandNotFoundErrorare public exports.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-evaluator/src/nemo_evaluator/jobs/publication.py` at line 31, Update the import in the publication module to import NeMoPlatformError and NotFoundError directly from the public nemo_platform package, instead of the private nemo_platform._exceptions module.plugins/nemo-evaluator/tests/jobs/test_publication.py (1)
535-564: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd a row-path test for required publication failure.
Both row failure tests use
required=False. The row path implements its own raise-or-return logic inpublish_row_eval_result, separate frompublish_agent_eval_result.fail, so a regression that drops therequiredraise for rows would pass this suite. Add a case that assertsPublicationFailedErrorfor a missing job id or a badtest_case_id_fieldwithrequired=True.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-evaluator/tests/jobs/test_publication.py` around lines 535 - 564, The row publication tests only cover optional failures and do not verify required failures raise. Add a test for the row path through EvaluateJob.run using either missing job_id or an invalid test_case_id_field with required=True, and assert PublicationFailedError is raised while preserving the existing no-publication behavior setup.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-evaluator/src/nemo_evaluator/intake/row_adapter.py`:
- Around line 38-51: Update the intake flow using _test_case_id to track every
generated test-case id and raise RowIdentityError when an id is produced more
than once, including collisions from repeated test_case_id_field values and
mixed positional fallbacks. Perform the check before inserting or replacing
trial mappings so duplicate rows fail loudly instead of overwriting existing
trajectories.
In `@plugins/nemo-evaluator/src/nemo_evaluator/jobs/publication_spec.py`:
- Around line 34-43: Constrain the agent_name and agent_version fields in the
publication specification to non-empty strings by applying the same
minimum-length validation used by evaluation_id. Preserve agent_name’s
optionality and agent_version’s existing default while ensuring explicitly
supplied empty values fail validation before
_require_resolvable_publication_identity runs.
---
Nitpick comments:
In `@plugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.py`:
- Line 21: Move target_agent_identity from agent_spec into the neutral module
alongside publication_spec.py, then update evaluate.py and publication.py to
import it from the new location. Remove the old definition and preserve its
existing behavior and callers.
In `@plugins/nemo-evaluator/src/nemo_evaluator/jobs/publication.py`:
- Around line 206-229: Refactor the failure handling in the publication flow
around the missing run_id and RowIdentityError branches into a shared
raise-or-return helper or equivalent, preserving PublicationFailedError behavior
for required evaluations. Ensure the missing-run_id path also emits the same
warning log as other publication failures before returning for optional
evaluations, while retaining each branch’s existing outcome details.
- Line 31: Update the import in the publication module to import
NeMoPlatformError and NotFoundError directly from the public nemo_platform
package, instead of the private nemo_platform._exceptions module.
In `@plugins/nemo-evaluator/tests/intake/test_publish.py`:
- Around line 117-128: Add a test for the missing started_at branch in
publish_to_intake, constructing an AgentEvalResult with RunMetadata() and
representative trial data, then assert that awaiting publish_to_intake raises
PublishError with the expected “started_at is unset” message.
In `@plugins/nemo-evaluator/tests/jobs/test_publication.py`:
- Around line 535-564: The row publication tests only cover optional failures
and do not verify required failures raise. Add a test for the row path through
EvaluateJob.run using either missing job_id or an invalid test_case_id_field
with required=True, and assert PublicationFailedError is raised while preserving
the existing no-publication behavior setup.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 62533bc1-998e-4801-836c-dbcbcd0adab2
📒 Files selected for processing (17)
plugins/nemo-evaluator/openapi/openapi.yamlplugins/nemo-evaluator/src/nemo_evaluator/intake/mapping.pyplugins/nemo-evaluator/src/nemo_evaluator/intake/publish.pyplugins/nemo-evaluator/src/nemo_evaluator/intake/row_adapter.pyplugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_evaluate.pyplugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_spec.pyplugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.pyplugins/nemo-evaluator/src/nemo_evaluator/jobs/publication.pyplugins/nemo-evaluator/src/nemo_evaluator/jobs/publication_spec.pyplugins/nemo-evaluator/src/nemo_evaluator/sdk/_executor.pyplugins/nemo-evaluator/src/nemo_evaluator/sdk/http_utils.pyplugins/nemo-evaluator/tests/intake/test_mapping.pyplugins/nemo-evaluator/tests/intake/test_publish.pyplugins/nemo-evaluator/tests/intake/test_row_adapter.pyplugins/nemo-evaluator/tests/integration/test_publish_to_intake.pyplugins/nemo-evaluator/tests/jobs/test_publication.pyplugins/nemo-evaluator/tests/test_sdk.py
💤 Files with no reviewable changes (1)
- plugins/nemo-evaluator/src/nemo_evaluator/sdk/http_utils.py
nv-odrulea
changed the title
nv-odrulea
changed the base branch from
main
to
od/agent-eval-api-publishes-to-intake
August 6, 2026 23:16
Contributor
|
nv-odrulea
force-pushed
the
od/agent-eval-api-publishes-to-intake
branch
from
August 7, 2026 05:50
77edb93 to
d90c4d4
Compare
SandyChapman
approved these changes
Aug 7, 2026
nv-odrulea
force-pushed
the
od/agent-eval-api-publishes-to-intake
branch
from
August 7, 2026 20:46
677ef46 to
9180e45
Compare
shanaiabuggy
approved these changes
Aug 7, 2026
nv-odrulea
force-pushed
the
od/model-eval-api-publish-to-intake
branch
from
August 8, 2026 04:07
1fbd77f to
8705a94
Compare
Contributor
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-evaluator/src/nemo_evaluator/intake/row_adapter.py`:
- Around line 78-80: Update the completion check in the row adapter around
AgentOutput so it tests response is None rather than response truthiness.
Preserve valid falsy JSON responses such as 0, false, [], and {} while still
returning None when both output_text and response are absent.
In `@plugins/nemo-evaluator/tests/jobs/test_publication.py`:
- Around line 517-529: Update _FakeRowEvaluator.run_sync to invoke the imported
run_sync helper around an async result-producing method, matching
_FakeEvaluator, instead of returning EvaluationResult directly. Ensure the
evaluator event loop is created and closed during the call so the publication
test exercises loop binding.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 61f1cb6d-09ed-47f0-9dcb-27e1a1c79a62
📒 Files selected for processing (12)
plugins/nemo-evaluator/openapi/openapi.yamlplugins/nemo-evaluator/src/nemo_evaluator/intake/row_adapter.pyplugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_spec.pyplugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.pyplugins/nemo-evaluator/src/nemo_evaluator/jobs/publication.pyplugins/nemo-evaluator/src/nemo_evaluator/jobs/publication_spec.pyplugins/nemo-evaluator/src/nemo_evaluator/sdk/_executor.pyplugins/nemo-evaluator/src/nemo_evaluator/sdk/http_utils.pyplugins/nemo-evaluator/tests/intake/test_row_adapter.pyplugins/nemo-evaluator/tests/integration/test_publish_to_intake.pyplugins/nemo-evaluator/tests/jobs/test_publication.pyplugins/nemo-evaluator/tests/test_sdk.py
💤 Files with no reviewable changes (1)
- plugins/nemo-evaluator/src/nemo_evaluator/sdk/http_utils.py
🚧 Files skipped from review as they are similar to previous changes (8)
- plugins/nemo-evaluator/tests/test_sdk.py
- plugins/nemo-evaluator/src/nemo_evaluator/sdk/_executor.py
- plugins/nemo-evaluator/src/nemo_evaluator/jobs/publication_spec.py
- plugins/nemo-evaluator/tests/intake/test_row_adapter.py
- plugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_spec.py
- plugins/nemo-evaluator/tests/integration/test_publish_to_intake.py
- plugins/nemo-evaluator/src/nemo_evaluator/jobs/publication.py
- plugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.py
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
plugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.py (1)
388-398: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winDo not publish after queryable persistence fails.
persist_evaluate_resultcatches all exceptions and continues, so this publication call still runs when no queryable result record exists. This can leave Intake data without its local result record and contradicts the stated “publish after persistence succeeds” contract. Gate publication on successful persistence, or define and test the partial-success behavior.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.py` around lines 388 - 398, Update the evaluate flow around persist_evaluate_result and publish_row_eval_result so publication occurs only when queryable-result persistence succeeds. Propagate or record the persistence failure instead of allowing the broad exception handling to continue into publication, while preserving the existing required-publication behavior for successful persistence.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.py`:
- Around line 238-252: The EvaluateJob republish flow must reload the persisted
run metadata instead of relying only on the in-memory result and full_result
artifact. Update the local result loading/publication path around
EvaluationResultFiles and local_result_path to read run-metadata.json and reuse
its run_id and started_at for retries; alternatively include those fields in the
primary artifact consumed by output["artifact"] and bundle_ref, while preserving
the existing Intake identity.
---
Outside diff comments:
In `@plugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.py`:
- Around line 388-398: Update the evaluate flow around persist_evaluate_result
and publish_row_eval_result so publication occurs only when queryable-result
persistence succeeds. Propagate or record the persistence failure instead of
allowing the broad exception handling to continue into publication, while
preserving the existing required-publication behavior for successful
persistence.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 8c91374f-ad3c-4e80-a484-724cd1095ccd
📒 Files selected for processing (3)
plugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.pyplugins/nemo-evaluator/src/nemo_evaluator/jobs/publication.pyplugins/nemo-evaluator/tests/jobs/test_publication.py
🚧 Files skipped from review as they are similar to previous changes (1)
- plugins/nemo-evaluator/tests/jobs/test_publication.py
nv-odrulea
enabled auto-merge
August 8, 2026 04:31
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Aug 8, 2026
nv-odrulea
force-pushed
the
od/model-eval-api-publish-to-intake
branch
from
August 12, 2026 20:21
a0d152e to
aaa67b0
Compare
Contributor
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
nv-odrulea
enabled auto-merge
August 12, 2026 20:23
Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
nv-odrulea
force-pushed
the
od/model-eval-api-publish-to-intake
branch
from
August 12, 2026 20:23
aaa67b0 to
b50ad05
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.py`:
- Around line 394-408: Gate the publication block around publish_row_eval_result
on successful result persistence, using the existing persistence-success state
from the exception handling near lines 375-379. When persistence fails, skip
publication (or terminate before reaching it); retain the current publication
behavior only after the queryable evaluation record is durable.
In `@plugins/nemo-evaluator/tests/jobs/test_publication.py`:
- Around line 686-695: Update test_row_agent_target_derives_its_name to execute
the publication or resolution path that derives the agent identity, rather than
only checking spec.publication is present. Assert that the resolved or published
agent name is “my-agent”, preserving the existing validation setup.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 345535d0-c546-4e1d-b1d4-fc846a3910da
📒 Files selected for processing (12)
plugins/nemo-evaluator/openapi/openapi.yamlplugins/nemo-evaluator/src/nemo_evaluator/intake/row_adapter.pyplugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_spec.pyplugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.pyplugins/nemo-evaluator/src/nemo_evaluator/jobs/publication.pyplugins/nemo-evaluator/src/nemo_evaluator/jobs/publication_spec.pyplugins/nemo-evaluator/src/nemo_evaluator/sdk/_executor.pyplugins/nemo-evaluator/src/nemo_evaluator/sdk/http_utils.pyplugins/nemo-evaluator/tests/intake/test_row_adapter.pyplugins/nemo-evaluator/tests/integration/test_publish_to_intake.pyplugins/nemo-evaluator/tests/jobs/test_publication.pyplugins/nemo-evaluator/tests/test_sdk.py
💤 Files with no reviewable changes (1)
- plugins/nemo-evaluator/src/nemo_evaluator/sdk/http_utils.py
🚧 Files skipped from review as they are similar to previous changes (8)
- plugins/nemo-evaluator/tests/intake/test_row_adapter.py
- plugins/nemo-evaluator/src/nemo_evaluator/jobs/publication_spec.py
- plugins/nemo-evaluator/src/nemo_evaluator/jobs/publication.py
- plugins/nemo-evaluator/tests/integration/test_publish_to_intake.py
- plugins/nemo-evaluator/openapi/openapi.yaml
- plugins/nemo-evaluator/src/nemo_evaluator/sdk/_executor.py
- plugins/nemo-evaluator/tests/test_sdk.py
- plugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_spec.py
Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Aug 12, 2026
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
POST /evaluate/jobs(dataset-driven evaluation) can now publish its results to Intake, so thoseruns reach Experiments the same way agent evaluations do after #. Studio submits to both
endpoints, so users saw one "evaluation" concept where only half of it could be published.
{ "publication": { "intake": { "evaluation_id": "email-security-baseline", "agent_name": "email-security-analyst", "test_case_id_field": "question_id", "required": true } } }Absent → no publish, zero Intake calls.
Related Issue
https://linear.app/nvidia/issue/ASTD-384/agent-experiments-evaluator-publish-to-intake-from-api
Approach: adapt, don't duplicate
Row results are adapted into an
AgentEvalResultand go through the existingpublish_to_intake.intake/publish.pyandintake/mapping.pyare untouched, so the dataset path inherits theidempotency guarantee rather than reimplementing it.
This isn't a semantic squeeze: agent-eval's published trajectory is already a stub single step
carrying output text, and
RowScore.sampleis literally{"output_text": ..., "response": ...}—the field names of
AgentOutput. One row → one trial, one (row, metric key) → one score.Notable decisions
test_case_iddefaults to row position, overridable by column.row_indexis weaker than itlooks:
discover_filesusesrglob/globwith nosorted(), so multi-file/glob datasetsconcatenate in filesystem order and positions shift between runs. Naming a column via
test_case_id_fieldgives stable identity; a configured column missing from the row raisesrather than silently falling back, since a silent fallback would reinstate exactly the instability
the field exists to remove.
run_idis the job id. Unlike agent-eval, whose result carries a generatedrun_id, a rowresult has none — so a platformless local run has nothing stable to key sessions on and reports a
publication failure instead of publishing.
started_atis stamped by the job. The row evaluator records no timing anywhere andEvaluationResulthas nomodel_config, so there is nowhere to put it. Publication needs a starttime that's a function of the run, not of when it was published.
Modeltarget names a model, not an agent. Without theported validator every trajectory would publish under
agent_name="". Now a 422 at submit.Refactors this required
evaluate.pycouldn't importjobs/publication.py:publication → intake.publish → sdk.http_utils → jobs.evaluate. Fixed structurally rather thanwith a
TYPE_CHECKINGimport (AGENTS.md forbids those): movedcreate_job_payload— two lines,one consumer that already imported
EvaluateInputSpec— out ofhttp_utilsintosdk/_executor.py.The edge is gone for every future caller.
PublicationSpec/IntakePublicationSpecmoved fromjobs/agent_spec.pyto a neutraljobs/publication_spec.py, which also holds the row variants. Subclassed rather than addingtest_case_id_fieldto the shared class, so each endpoint's generated contract stays honest.tyerrors in files this change had to stage (evaluate.py's too-narrowto_specasync_sdkannotation, 3 intest_sdk.py). Plugin diagnostics: 73 → 69.Testing
pre-commitgreen including thestaged-files
tyhook.test_row_result_publishes_and_is_idempotent— re-publishing yields one trace, withtest_case_idtaken from the configured column.
Caveats
{metric_type}.{output}, the multi-metric convention. Asingle-metric run will show
exact_match.scorein Intake where its own aggregate saysscore.Deliberate — consistency with Intake and agent-eval wins.
ctx.job_id, stable across a retry of the samejob but not across a re-run. A re-run is a new run and correctly gets new sessions.
BenchmarkEvaluationResultrepeats every row underper_metric; the adapter reads top-levelrow_scoresonly (commented and pinned by a test) or each row would publish once per metric.Summary by CodeRabbit
New Features
Bug Fixes