Skip to content

feat(rewards): support batched reward-hook execution (#225) - #435

Open
candy972023 wants to merge 1 commit into
inclusionAI:mainfrom
candy972023:feature/batched-reward-hook-225
Open

feat(rewards): support batched reward-hook execution (#225)#435
candy972023 wants to merge 1 commit into
inclusionAI:mainfrom
candy972023:feature/batched-reward-hook-225

Conversation

@candy972023

Copy link
Copy Markdown
  • ✨ New feature
  • ⚡ Performance improvement

Summary

Closes #225

Add an optional batch reward interface alongside the existing per-example adapter. Users can define reward_fn_batch(records) -> list[float] in the same module as reward_fn; when present, AReno calls the batch interface once per batch instead of looping per record. This benefits reward functions with heavy per-call overhead (e.g. external model scoring, batch inference).

Files

File Type Description
areno/api/rewards.py Modified Add load_reward_fns, validate_batch_rewards, compute_rewards with cardinality validation and execution timing
areno/api/trainers/policy_only.py Modified Use compute_rewards in both agentic and rollout paths
areno/api/trainers/ppo.py Modified Forward reward_fn_batch to PolicyOnlyTrainer
areno/api/trainer_factory.py Modified Accept and forward reward_fn_batch
areno/cli/train.py Modified Load both functions via load_reward_fns
tests/test_batched_reward_cpu.py New 15 CPU tests

Design

  • Dual interface: reward_fn(record) -> float (existing) and reward_fn_batch(records) -> list[float] (optional). Users opt in by defining the batch function; no config change needed.
  • Cardinality validation: validate_batch_rewards checks output length matches input length. On mismatch, raises ValueError with the batch index so the caller can identify which prompt-group batch failed.
  • Execution timing: Both paths record elapsed time via time.perf_counter() and log it through the trainer's logger.
  • Backward compatibility: load_reward_fn preserved as public API, delegates to load_reward_fns(path)[0]. When no batch function is defined, behavior is identical to before.
  • Architecture: Reuses existing RewardRecord contract, no new dependencies.

How was it tested?

CPU tests (no GPU required), 15/15 passed:

  • Batch and per-example paths produce identical results on a deterministic hook
  • Cardinality validation: short output, long output, batch index in error message
  • Empty batch returns empty list
  • Float coercion of integer rewards
  • load_reward_fns loads both functions, detects absent batch fn, rejects non-callable
  • Backward compatibility: dispatch without batch fn matches manual per-example loop
  • Execution timing logged for both paths

Checklist

  • Default behavior remains backward compatible
  • No external database or mandatory sandbox
  • Uses existing AReno contracts (RewardRecord, Callable)
  • No new dependencies
  • Focused CPU tests cover success, invalid input, and boundary cases

Add an optional batch reward interface alongside the existing per-example
adapter. Users can define `reward_fn_batch(records) -> list[float]` in the
same module as `reward_fn`; when present, AReno calls the batch interface
once per batch instead of looping per record.

Changes:
- rewards.py: add load_reward_fns, validate_batch_rewards, compute_rewards
  with execution timing for both paths; load_reward_fn delegates to
  load_reward_fns to avoid duplication
- policy_only.py: use compute_rewards in both agentic and rollout paths
- ppo.py: pass reward_fn_batch through to PolicyOnlyTrainer
- trainer_factory.py: accept and forward reward_fn_batch
- cli/train.py: load both functions via load_reward_fns
- tests/test_batched_reward_cpu.py: CPU tests covering agreement,
  cardinality validation, empty batch, backward compat, and module loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support batched reward-hook execution

1 participant