Feature/207 multiple weighted reward

[Shik2024]

What does this PR do?

AReno's RL training previously supported only single-dimension, simple scoring: areno train --reward-fn-path accepted a single reward file, each completion produced one scalar reward, and the dashboard exposed only the aggregate rollout/rewards_mean. This is fine for single objectives like GSM8K ("answer right/wrong"), but falls short for real, multi-dimensional scenarios — code generation needs both logical correctness and code style, while question answering / chat needs factual correctness and humanistic care and format compliance, etc.

This PR makes --reward-fn-path repeatable, with the form path[:weight] (weight defaults to 1.0), so users can register multiple named reward components in one run and combine them into a single final reward via weight-normalized summation. Each component's mean (reward/<name>_mean) and failure count (reward/<name>_invalid_count) are emitted as independent metrics through the existing train_stats → record_training_stats channel, surfacing on TensorBoard (train/reward/<name>_*), the CLI log, and the dashboard — so every dimension of the reward is individually observable.

Design is byte-for-byte backward compatible: a single --reward-fn-path without :weight behaves exactly as before. CompositeReward.__call__ returns score().total, remaining a drop-in for the reward_fn(record)→float contract the trainer already uses; the branch happens inside _score_reward via isinstance, so no Trainer/Backend public signature or TrainerConfig.reward_fn_path field is touched.

Main changes: areno/api/rewards.py (new CompositeScore/CompositeReward), areno/cli/train.py (repeatable --reward-fn-path + --reward-on-error + parsing/validation/assembly), areno/api/trainers/policy_only.py and ppo.py (per-component accumulation & metric injection; PPO reuses via inheritance), areno/api/metrics.py (reuses the existing write loop), examples/math/accuracy_reward.py + format_reward.py (multi-dimension demo), and tests/ (new coverage).

Related issue

Fixes #(issue)

Type of change

• 🐛 Bug fix
• ✨ New feature
• 💥 Breaking change (public API / CLI behavior changes in a non-backward-compatible way)
• 📝 Documentation update
• ♻️ Refactoring
• ⚡ Performance improvement
• ✅ Test coverage improvement

How was it tested?

New tests (CPU, no GPU required):

• tests/test_composite_reward_cpu.py: CompositeReward math / validation / error modes / component access / example semantics / CLI parsing, plus a new TrainerComponentStatsTest (3 cases: multi-component emits reward/<name>_mean, mark_invalid increments the invalid count, and a plain single reward does not leak these keys).
• tests/test_metrics_cpu.py: new RewardComponentMetricsTest using a FakeWriter to assert record_training_stats writes train/reward/<name>_mean and train/reward/<name>_invalid_count.

Commands to run (pending CI / Linux + Python 3.10+):

pytest tests/test_composite_reward_cpu.py tests/test_metrics_cpu.py -q
pytest tests/ -k cpu          # regression: single-reward path takes the else branch
ruff check . && pyright

End-to-end smoke (requires GPU):

areno train --ckpt Qwen/Qwen3-0.6B --dataset-path gsm8k:main \
  --dataset-loader-fn examples/math/dataset_loader.py \
  --reward-fn-path examples/math/accuracy_reward.py:0.7 \
  --reward-fn-path examples/math/format_reward.py:0.3 \
  --algo gspo --tp-size 1 --world-size 1 --batch-size 1
# expected: train_stats contains reward/accuracy_reward_mean and reward/format_reward_mean

Hardware / environment limitations (disclosed honestly): the local checkout is on macOS with only the system Python 3.9; pytest/pydantic/ruff are unavailable and the repo relies on @dataclass(slots=True) (needs 3.10+), so the test suite cannot be executed locally. Only static checks were run locally: all 5 changed files pass python3 -m py_compile, and grep -rc _reward_components_legacy areno/ tests/ returns 0 (dead code removed). Runtime pytest/ruff/pyright and the GPU smoke run are deferred to CI/Linux; no "tests pass" claim is made until CI confirms it.

Checklist

• The PR title summarizes the contribution.
• Linked the related issue in the description (if any).
• Existing tests pass (pytest tests/ -k cpu).
• New behavior is covered by tests.
• Described the test commands run and any hardware limitations.
• Public API / CLI changes are additive and backward-compatible (see CONTRIBUTING.md).

Breaking change details

None. This PR is purely additive and backward compatible: the single-reward usage is byte-for-byte unchanged; --reward-fn-path only goes from single-value to repeatable (option name unchanged) with a new additive --reward-on-error switch; CompositeReward/CompositeScore are new SDK exports that do not alter any existing Trainer/Backend/TrainerConfig public surface.