refactor(algorithms): separate rollout and reward requirements

[yuetongz417]

Summary

This PR separates an algorithm’s rollout requirement from its reward requirement.

Previously, the training CLI assumed that every algorithm with requires_rollout=True must also provide either --reward-fn-path or --reward-ckpt. This assumption is valid for the existing GSPO, GRPO, and PPO workflows, but it prevents rollout-based algorithms that use other forms of supervision, such as On-Policy Distillation.

This PR adds an explicit requires_reward capability to AlgorithmSpec.

Motivation

Rollout and reward are related but distinct algorithm requirements.

For example:

Algorithm | Requires rollout | Requires reward -- | -- | -- SFT | No | No DPO | No | No GSPO | Yes | Yes GRPO | Yes | Yes PPO | Yes | Yes OPD | Yes | No

In OPD, the student generates on-policy responses, but supervision comes from token-level teacher scores rather than a scalar reward function or reward model.

Separating these capabilities allows AReno to support this workflow without weakening validation for existing reinforcement-learning algorithms.

Changes

• Add requires_reward to AlgorithmSpec.

• Validate that an algorithm cannot require reward without also requiring rollout.

• Preserve requires_reward when algorithm names are normalized.

• Mark GSPO, GRPO, and PPO as requiring reward.

• Update CLI reward-source validation to use requires_reward instead of requires_rollout.

• Display requires_reward in the resolved training configuration summary.

• Include the new capability in the algorithm inspection script.

• Update contributor and architecture documentation.

• Add tests for:

  • built-in algorithm reward metadata;

  • invalid reward-without-rollout registrations;

  • rollout algorithms that do not require reward;

  • training configuration summary output;

  • preservation of existing GSPO reward validation.

Behavior after this change

Existing reward-based algorithms continue to require:

--reward-fn-path

or:

--reward-ckpt

However, a future algorithm may now declare:

AlgorithmSpec(
    name="opd",
    trainer_cls=OPDTrainer,
    default_loss_fn=opd_loss_fn,
    requires_rollout=True,
    requires_reward=False,
)

and use rollout configuration without being forced to provide an unrelated reward source.

Testing

python -m pytest \
  tests/test_algorithms_cpu.py \
  tests/test_train_cli_config_cpu.py \
  -q

Result:

92 passed

The algorithm inspection script was also checked with:

PYTHONPATH=. python \
  .agents/skills/areno-add-algorithm/scripts/inspect_algorithms.py

Scope

This PR does not add the OPD algorithm itself. It introduces the registry and CLI capability separation needed for OPD and other rollout-based algorithms that use non-reward supervision.