Harden and document the validated GRPO pipeline

[EternalmBlue]

Summary

• enable the default external GRPO reward judge with strict grpo_judge_v2 validation and opt-in built-in rewards
• isolate direct Judge API keys in a Git-ignored local YAML and recursively redact generated snapshots/reports
• validate GRPO base adapters before DPO -> Fact-SFT -> CPT fallback
• use hardware-aware precision defaults with non-finite-gradient fail-fast behavior
• update README, configuration docs, security guidance, data contracts, and the operator Wiki from the completed 0.8B full-chain run
• strengthen CI with unit tests, prompt-only GRPO validation, and plaintext API-key rejection

Validation

• py -m unittest discover -s tests -v: 29 passed
• py -m compileall -q pipeline scripts tests serve_inference.py: passed
• CI-equivalent JSONL, config-doc, README-reference, forbidden-string, Markdown link/fence checks: passed
• staged/tracked/history/output exact-key scans: 0 matches
• full-chain artifacts: finite grad_norm in CPT/SFT/DPO/GRPO; DPO -> GRPO changed 192/192 LoRA tensors

Quality boundary

The pipeline completed successfully, but the current 0.8B run is not presented as production-ready: the heuristic safety evaluation passed only 2/5 cases and GRPO completion clipping remained high. Documentation now requires independent safety review, finite-gradient checks, real adapter deltas, reward variance, and clipping review before release.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fdcb1448dd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Skip stale DPO adapters when DPO is disabled

When dpo.enabled is false but an old outputs/dpo_adapter from a previous experiment still exists, these candidates are still checked before the current Fact-SFT/CPT outputs, so GRPO can silently train from a stale DPO adapter even though the run intentionally disabled DPO. This can corrupt experiment results; only fall back to the DPO output when DPO is active or when the user explicitly chooses that adapter.

Useful? React with 👍 / 👎.