Skip to content

test: add regression test for eval sample logging when sample is a list#1

Open
devin-ai-integration[bot] wants to merge 1 commit into
mainfrom
devin/1780032870-regression-test-eval-sample-logging
Open

test: add regression test for eval sample logging when sample is a list#1
devin-ai-integration[bot] wants to merge 1 commit into
mainfrom
devin/1780032870-regression-test-eval-sample-logging

Conversation

@devin-ai-integration

Copy link
Copy Markdown

Summary

Adds a regression test for the bug fixed in 8efb116 ([fix] eval sample logging when sample is a list (#1836)).

The bug: eval_rollout_single_dataset accessed .prompt / .response / .reward directly on the sample variable, which could be a list[Sample] (when n_samples_per_prompt > 1), causing an AttributeError.

The fix extracted logged_sample = sample[0] if isinstance(sample, list) else sample before logging.

This PR adds three focused unit tests in tests/utils/test_eval_sample_logging.py:

  1. test_eval_sample_logging_with_list_sample — verifies that when sample is a list, the first element's prompt, response, and reward are correctly logged
  2. test_eval_sample_logging_with_single_sample — verifies the non-list (single Sample) path still works
  3. test_eval_sample_logging_list_raises_without_fix — proves the original bug existed by showing list.prompt raises AttributeError

Review & Testing Checklist for Human

  • Verify the helper _log_eval_sample mirrors the fixed logging logic in eval_rollout_single_dataset (line 576-581 of slime/rollout/sglang_rollout.py)
  • Run pytest tests/utils/test_eval_sample_logging.py -v to confirm all 3 tests pass

Notes

The tests are lightweight unit tests that only depend on slime.utils.types.Sample and logging — no GPU or external services required.

Link to Devin session: https://app.devin.ai/sessions/f4670acf0c004d6691b0aa0b83a07b74
Requested by: @ShaneGZhu

Reproduces the bug fixed in 8efb116 where eval_rollout_single_dataset
accessed .prompt/.response/.reward on a raw list instead of extracting
the first element.

Three test cases:
- list sample: verifies first element's prompt/response/reward are logged
- single sample: verifies direct Sample logging still works
- attribute error: proves the original bug (list has no .prompt attr)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant