Skip to content

feat(metrics): add areno metrics CLI backed by a shared TensorBoard reader (#254) - #424

Open
galaxyMoonStar wants to merge 8 commits into
inclusionAI:mainfrom
galaxyMoonStar:feat/issue-254-metrics-cli
Open

feat(metrics): add areno metrics CLI backed by a shared TensorBoard reader (#254)#424
galaxyMoonStar wants to merge 8 commits into
inclusionAI:mainfrom
galaxyMoonStar:feat/issue-254-metrics-cli

Conversation

@galaxyMoonStar

Copy link
Copy Markdown

Closes #254

Summary

  • Adds a read-only areno metrics subcommand that reads a single metric's history from local events.out.tfevents.* and prints last/min/max/mean/steps/recent/trend as a table (with a hand-written sparkline) or JSON.
  • The read side is extracted from areno/dashboard/server.py into a new shared module areno/api/metric_reader.py, so the dashboard and the CLI now share one reader — a single source of truth — instead of two parallel implementations.
  • No new dependencies, no GPU import, no change to train/serve/dashboard behavior.

Usage

areno metrics --metrics-dir DIR [--pid N] --name <tag> [--limit N] [--json]
areno metrics --metrics-dir /tmp/areno/tfevent --name rollout/rewards_mean         # default table
areno metrics --metrics-dir /tmp/areno/tfevent --pid 12345 --name ... --limit 50   # pick one run by pid in a mixed dir
areno metrics --metrics-dir /tmp/areno/tfevent --name ... --json                   # pipe to jq
areno metrics --metrics-dir /tmp/areno/tfevent                                      # no --name → list available tags

Table output:

metric   rollout/rewards_mean
count    5
steps    0 -> 4
last     1 (step 4)
min      0.5
max      1
mean     0.75
trend    ▁▃▅▇█
recent   step 0: 0.5, step 1: 0.625, step 2: 0.75, step 3: 0.875, step 4: 1

JSON output (--json):

{"name":"rollout/rewards_mean","count":5,"last":1.0,"last_step":4,"min":0.5,"max":1.0,
 "mean":0.75,"min_step":0,"max_step":4,"recent":[0.5,0.625,0.75,0.875,1.0],
 "recent_steps":[0,1,2,3,4],"trend":[0.0,0.25,0.5,0.75,1.0]}

Defaults: --metrics-dir=/tmp/areno/tfevent, --limit=20, table output.


What Changed

File Type Notes
areno/api/metric_reader.py Added Shared read-side pure functions: locate event files, read scalars, summarize, render
areno/cli/metrics.py Added areno metrics subcommand entry
areno/cli/main.py Modified Register the new command (1 line in _COMMANDS)
areno/dashboard/server.py Modified Read side now calls metric_reader; dead code removed; constant consolidated (+20 / -47)
areno/api/defaults.py Unchanged DEFAULT_METRICS_LOG_DIR single source, imported by both sides
docs/cli/metrics.rst, docs/reference/cli.rst Added/Modified New CLI guide page + toctree entry
tests/test_metric_reader_cpu.py Added Pure-function tests
tests/test_metric_reader_equivalence_cpu.py Added "before == after" extraction guard
tests/test_cli_metrics_cpu.py Added CLI tests

Total: +1182 / -47.


Design Decisions

  1. Shared reader (single source of truth) — Extract the dashboard's read side into areno/api/metric_reader.py and have both dashboard and CLI call it. This directly honors the issue's 4× emphasis on "reuse existing contracts, do not introduce a parallel subsystem." Two readers would drift; one shared reader evolves in one place.
  2. Behavior-preserving refactor — "lock then move" — Snapshot the dashboard reader's output before extraction, then assert byte-equal output after switching to metric_reader. The move is strict "move, don't rewrite": the size_guidance=10000 / [-500:] / NaN-skip / (name,step,value) dedup semantics are preserved verbatim. Any incidental cleanup is deferred.
  3. No new dependencies — The sparkline is hand-written UTF-8 (no sparklines/rich); tensorboard was already declared in pyproject.toml. The CLI stays a light, pure query path.
  4. Rejected: CLI imports dashboard.server directly — That would drag the ~1600-line http.server into a pure CLI and invert the dependency direction (cli → dashboard instead of both → api). Rejected on layering grounds.

Tests

27 CPU tests across 3 files (no GPU, no network, ARENO_BUILD_EXT=0):

File Coverage
test_metric_reader_equivalence_cpu.py "before == after" snapshot using a real SummaryWriter fixture (job.metrics + job.timeperf)
test_metric_reader_cpu.py normal / NaN / single-point / empty / flat / truncate-500 / trend-bounded / render
test_cli_metrics_cpu.py success / unknown name lists tags / empty dir / no event / single point / all-NaN / memory-bounded / missing tensorboard / --json

Run:

pytest tests/test_metric_reader_cpu.py tests/test_metric_reader_equivalence_cpu.py tests/test_cli_metrics_cpu.py -v

Kaggle end-to-end run on a real SFT training (real events.out.tfevents.*) is tracked in the local execution note.


Notes for Reviewer

  • Riskiest part is the dashboard/server.py refactor. It is behavior-preserving and guarded by the equivalence test — if the test passes, dashboard output is unchanged.
  • New read-only subcommand. train / serve / dashboard behavior is unchanged; defaults preserved.
  • Memory-bounded for long runs. size_guidance=10000 caps how many scalars TensorBoard loads per tag, [-500:] keeps only the tail window, and the summary is computed over that bounded window (extra memory independent of total training length).

jiangfangqin.jfq and others added 8 commits July 29, 2026 15:10
The metrics summary now carries training progress alongside values, so the
log reads at a glance:
- new summary fields: last_step, min_step, max_step, mean, recent_steps
- table: a 'steps A -> B' line, a mean line, 'last V (step N)', and recent
  paired with absolute steps; the hand-written sparkline trend is unchanged
- JSON: the five new keys, additive only (recent/trend shape preserved)

Aggregation stays a single streaming pass (O(1) memory); min_step/max_step
span the retained [-500:] window, not full training history. The dashboard
read-side and its equivalence guard are untouched -- the new fields live in
the CLI-only summarize/render layer.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Query metric history from the CLI

1 participant