test: add regression test for eval sample logging when sample is a list#1
Open
devin-ai-integration[bot] wants to merge 1 commit into
Open
Conversation
Reproduces the bug fixed in 8efb116 where eval_rollout_single_dataset accessed .prompt/.response/.reward on a raw list instead of extracting the first element. Three test cases: - list sample: verifies first element's prompt/response/reward are logged - single sample: verifies direct Sample logging still works - attribute error: proves the original bug (list has no .prompt attr) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Author
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a regression test for the bug fixed in 8efb116 (
[fix] eval sample logging when sample is a list (#1836)).The bug:
eval_rollout_single_datasetaccessed.prompt/.response/.rewarddirectly on thesamplevariable, which could be alist[Sample](whenn_samples_per_prompt > 1), causing anAttributeError.The fix extracted
logged_sample = sample[0] if isinstance(sample, list) else samplebefore logging.This PR adds three focused unit tests in
tests/utils/test_eval_sample_logging.py:test_eval_sample_logging_with_list_sample— verifies that whensampleis a list, the first element's prompt, response, and reward are correctly loggedtest_eval_sample_logging_with_single_sample— verifies the non-list (singleSample) path still workstest_eval_sample_logging_list_raises_without_fix— proves the original bug existed by showinglist.promptraisesAttributeErrorReview & Testing Checklist for Human
_log_eval_samplemirrors the fixed logging logic ineval_rollout_single_dataset(line 576-581 ofslime/rollout/sglang_rollout.py)pytest tests/utils/test_eval_sample_logging.py -vto confirm all 3 tests passNotes
The tests are lightweight unit tests that only depend on
slime.utils.types.Sampleandlogging— no GPU or external services required.Link to Devin session: https://app.devin.ai/sessions/f4670acf0c004d6691b0aa0b83a07b74
Requested by: @ShaneGZhu