[codex] Document auto-nanogpt launch plan

[morganmcg1]

Summary

Adds a small launch bundle for the auto-nanoGPT Senpai experiment:

• launches/launch-auto-nanogpt.sh launches five independent 1-GPU/student tags, each with one advisor and eight students.
• launches/auto-nanogpt-extra-instructions.md documents the task framing, fairness controls, source restrictions, telemetry expectations, and step-budget guidance passed through --extra_instructions.

This PR is intentionally documentation plus launch plumbing only. It does not change the default Senpai config or runtime prompts.

Target Snapshot

The launch is pinned to the target package snapshot:

• target repo: https://github.com/morganmcg1/modded-nanogpt-senpai.git
• target branch: senpai-launch-20260515
• target commit at branch creation: 0ba525c

That target branch contains the forked modded-nanogpt benchmark plus Senpai target files, W&B telemetry, shared FineWeb PVC cache support, and explicit source/fairness instructions in program.md and instructions/prompt-advisor.md.

Experiment Plan

The launch script creates:

• tags/branches: auto-nanogpt-1gpu-r1 through auto-nanogpt-1gpu-r5
• students per advisor: 8
• GPUs per student: 1
• timeout: 30240 minutes, three weeks
• max epochs: 100000, treated as a hard ceiling rather than a target
• W&B entity/project: wandb-applied-ai-team/modded-nanogpt-senpai
• history scope: fresh
• corrected total GPU footprint: 40 student GPUs

Fairness Controls

The extra instructions keep the run close to the public modded-nanogpt track 3 benchmark:

• optimize steps to FineWeb validation loss below 3.28
• keep dataset, batch size, model architecture, and one forward-backward pass per optimizer step fixed
• focus on optimizer algorithms, schedules, initialization, and optimizer hyperparameters
• balance exploitation with exploration, avoiding a run that is mostly scalar LR/WD search
• use predeclared step counts and non-cherry-picked seed batches for final claims
• focus each advisor only on its own branch/tag/PR/W&B stream
• use the checked-in target snapshot; do not refresh post-launch upstream PRs, records, branches, or issues

The instructions explicitly ban agents from reading or using Prime Intellect's released autonomous-run materials, including:

• https://www.primeintellect.ai/auto-nanogpt
• https://github.com/PrimeIntellect-ai/experiments-autonomous-speedrunning
• raw GitHub URLs, files, branches, issues, PRs, or archives under that repo

Those are named only as banned sources/comparison artifacts for humans after the run.

Data And Telemetry

FineWeb has been pre-cached on pai-2 at:

• /mnt/new-pvc/datasets/fineweb10B
• verified file count: 21
• verified size: 4.0G

The target training script falls back to normal download behavior if needed, but Senpai pods should use the shared PVC cache automatically through PVC_MOUNT_PATH.

The target logs W&B metrics including validation loss, steps-to-target, train loss slopes over trailing 10% windows, grad norms, grad-to-weight norm, weight stats, per-type/per-param stats, and sampled histograms.

Validation

Validated locally from /Users/mmcguire/ML/senpai:

• bash -n launches/launch-auto-nanogpt.sh
• SENPAI_PYTHON=.venv/bin/python launches/launch-auto-nanogpt.sh --dry_run
• SENPAI_PYTHON=.venv/bin/python launches/launch-auto-nanogpt.sh --preflight_only

Preflight passed for GitHub push access, target branch existence, Anthropic API key, and Exa API key for all five planned launches.

Launched

The initial launch from commit 44b359c21c03a9540c5d11f336abb9adc5fcc8c2 used --gpus_per_student 8 by mistake, creating an oversized auto-nanogpt-r1 through auto-nanogpt-r5 stream. That fleet was stopped, all old k8s deployments/configmaps/secrets were deleted, and the old target PR stream against auto-nanogpt-r* branches was closed as an aborted launch artifact.

The corrected five-replication launch was applied from branch codex/auto-nanogpt-launch at commit a76f45a48fe005d54726ff80013994b162c40b48.

Corrected verification showed:

• corrected tags/branches: auto-nanogpt-1gpu-r1 through auto-nanogpt-1gpu-r5
• advisors ready: 5/5
• students ready: 40/40
• total deployments ready: 45/45
• GPUs requested by corrected auto-nanoGPT fleet: 40
• per replicate: 8 one-GPU student deployments plus one CPU-only advisor

W&B corrected launch report: https://wandb.ai/wandb-applied-ai-team/modded-nanogpt-senpai/reports/Auto-nanoGPT-Senpai-corrected-1-GPU-launch--VmlldzoxNjg5MjI0Mg==

[github-actions]

Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.

---

I have read the CLA Document and I hereby sign the CLA

---

_{You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.}