Add standalone checkpoint evaluation for Puzzletron

examples/puzzletron/README.md

@@ -10,6 +10,7 @@ distill the selected model.
 - [Start here](#start-here)
 - [Setup wizard](#setup-wizard)
 - [Installation](#installation)
+- [Evaluate a checkpoint](#evaluate-a-checkpoint)
 - [Run with an agent](#run-with-an-agent)
 - [Configuration](#configuration)
 - [Run a campaign](#run-a-campaign)
@@ -22,6 +23,8 @@ distill the selected model.
   smoke and production bundles.
 - **Generated campaign:** complete the [installation](#installation), then
   [run the campaign](#run-a-campaign) with its generated bundle.
+- **Checkpoint evaluation:** use [Evaluate a checkpoint](#evaluate-a-checkpoint)
+  for a local model without creating or running a pruning campaign.
 - **Agent-assisted campaign:** follow [Run with an agent](#run-with-an-agent)
   with your model, data, compute environment, and deployment goals.
 - **Existing results:** see [Reports](#reports) to regenerate a campaign report
@@ -318,6 +321,24 @@ python -m pip check
 Record the three source revisions and verification output with the campaign.
 Re-run verification after pulling either fork or rebuilding a CUDA extension.
 
+## Evaluate a checkpoint
+
+Basic evaluation is independent of MIP and the campaign DAG. In the Puzzletron
+worker environment, run any compatible local Hugging Face checkpoint directly:
+
+```bash
+python examples/puzzletron/evaluate_lmms_checkpoint.py \
+  --checkpoint /path/to/checkpoint \
+  --output-dir /path/to/results/checkpoint-smoke
+```
+
+The default one-GPU smoke evaluates eight samples each from IFEval and GSM8K.
+Qwen 3.5 checkpoints are configured automatically. See
+[checkpoint evaluation](docs/checkpoint_evaluation.md) to choose tasks, run a
+full evaluation, find results, or override model detection. For options not
+covered by the convenience command, append `--lmms-eval-args` followed by the
+native lmms-eval options.
+
 ## Run with an agent
 
 The canonical agent workflow is
@@ -467,11 +488,9 @@ the same command for the full campaign. Add `--dry-run` to inspect either plan
 without submitting work, or select one stage while iterating, for example
 `--stage mip --dry-run`.
 
-The setup wizard can also add downstream `lmms-eval` nodes that evaluate
-materialized candidates through vLLM. They run in the standard Puzzletron
-worker environment, whose example requirements pin a compatible `lmms-eval`
-snapshot. See [post-MIP pipelines](docs/post_mip_pipeline.md) for configuration
-details and for adding downstream evaluation to an existing campaign.
+The setup wizard can also add downstream evaluation for materialized campaign
+candidates. See [post-MIP pipelines](docs/post_mip_pipeline.md) to configure it
+or add it to an existing campaign.
 
 ### Legacy checked-in Nano campaign
 

examples/puzzletron/docs/checkpoint_evaluation.md

@@ -0,0 +1,60 @@
+# Checkpoint evaluation
+
+Use the standalone command to evaluate a compatible local Hugging Face
+checkpoint without creating or running a Puzzletron campaign.
+
+## Quick start
+
+Install the Puzzletron worker requirements:
+
+```bash
+python -m pip install -r examples/puzzletron/requirements.txt
+```
+
+Then run the default smoke:
+
+```bash
+python examples/puzzletron/evaluate_lmms_checkpoint.py \
+  --checkpoint /path/to/checkpoint \
+  --output-dir /path/to/results/checkpoint-smoke
+```
+
+This evaluates eight samples each from IFEval and GSM8K on one GPU. Results and
+logs are written under the output directory.
+
+## Customize the evaluation
+
+Choose tasks and common runtime settings with command-line options:
+
+```bash
+python examples/puzzletron/evaluate_lmms_checkpoint.py \
+  --checkpoint /path/to/checkpoint \
+  --output-dir /path/to/results/custom-smoke \
+  --tasks ifeval,gsm8k \
+  --limit 32 \
+  --tensor-parallel-size 2 \
+  --dtype bfloat16 \
+  --max-model-len 8192
+```
+
+Qwen 3.5 checkpoints are detected from their local `config.json` and configured
+automatically. Use `--reasoning-parser` to override the detected parser or
+`--model-profile none` to disable model detection.
+
+Use `--trust-remote-code` only after reviewing the checkpoint-provided Python
+code. After the smoke succeeds, use `--full` with a separate output directory
+to evaluate the complete task datasets. Use `--timeout-seconds` if the full run
+needs a different limit.
+
+Pass additional native options after `--lmms-eval-args`, which must be the last
+wrapper option. See `python -m lmms_eval --help` for the available options.
+
+## Results and troubleshooting
+
+Each run creates a new `attempt_<id>/` directory. Start with `summary.json` for
+metrics. If a run fails, inspect `stderr.txt`; the command and raw evaluator
+output are retained in the same directory. Rerunning creates another attempt
+without overwriting the earlier one.
+
+To evaluate candidates as part of a pruning campaign, use
+[downstream evaluation](post_mip_pipeline.md#downstream-evaluation) instead.

examples/puzzletron/docs/post_mip_pipeline.md

@@ -141,21 +141,12 @@ from the original candidate.
 
 ## Downstream evaluation
 
-`downstream_evaluation` runs `python -m lmms_eval` as a subprocess from the
-Puzzletron worker environment. The runner passes an argument list directly and
-does not invoke a shell. Values in `command_prefix` and `extra_args` are arguments;
-shell syntax is not interpreted. The standard example requirements pin a snapshot
-compatible with the newer `wandb` required by the pinned AutoModel build:
-
-```bash
-python -m pip install -r examples/puzzletron/requirements.txt
-python -c 'import importlib.metadata as m; assert m.version("lmms-eval") == "0.7.0"'
-```
-
-The runner derives the realized checkpoint path, vLLM topology arguments, task
-list, and output path from the campaign config. Use `model_args` only for
-non-derived model options such as dtype or maximum model length, and `extra_args`
-only for non-reserved `lmms-eval` flags.
+`downstream_evaluation` adapts the generic
+[checkpoint evaluator](checkpoint_evaluation.md) to materialized campaign
+candidates and publishes their task metrics. Add it after a `materialize` node;
+the linked example config shows the complete flow. Use the standalone
+[checkpoint evaluation](checkpoint_evaluation.md) command when campaign
+lineage, filtering, and reports are not needed.
 
 ## Filters