OLMo3 1B baseline pipeline

[RmZeta2718]

Architecture Proposal

#1

Implementation Details

This PR implements the OLMo 3 root baseline associated with Issue #1. It provides a reproducible training recipe and does not propose, modify, or evaluate a new model architecture.

Implemented components:

• Shared OLMo 3 1B model, optimizer, trainer, checkpoint, and evaluation configuration.
• Separate entry points for pretraining, midtraining, and long-context extension.
• A three-stage launcher supporting:
  • single-node and multi-node torchrun;
  • checkpoint handoff between stages;
  • same-stage resume with model, optimizer, trainer, data-loader, and RNG state;
  • completion markers for safely skipping finished stages;
  • online or offline Weights & Biases logging.
• Hugging Face checkpoint conversion and OLMES evaluation instructions.
• English and Chinese documentation covering installation, data layout, training, resume behavior, and evaluation.

Deviations from the official OLMo 3 recipes:

• OLMo 3 does not publish an officially tuned 1B three-stage recipe. These configurations adapt the official 7B recipes to the existing OLMo 3 1B model definition.
• Stage 1 uses the 150B sampled data mix rather than the complete multi-trillion-token training corpus.
• The current implementation uses Muon by default while retaining an AdamW-compatible path.
• The third stage covers long-context extension rather than instruction tuning.
• Consequently, this PR should be treated as an experimental 1B baseline recipe, not an exact reproduction of an officially released OLMo 3 checkpoint.

Archive

• WandB Report (include training and evaluation logs): TODO

• HuggingFace Collection (include model checkpoints): TODO

---

Reviewer Assessment (for repo reviewers)

Merge Checklist:

• The implementation PR is linked to an in-progress Architecture Proposal issue (#).
• The implementation correctness of the proposed architecture is verified.
• The effectiveness of the proposed architecture is validated at 1B, 3B, and 8B model scales.
• The archive information is complete, including WandB reports and HuggingFace collections of model checkpoints.