feat(metrics): add areno metrics CLI backed by a shared TensorBoard reader (#254) - #424
Open
galaxyMoonStar wants to merge 8 commits into
Open
feat(metrics): add areno metrics CLI backed by a shared TensorBoard reader (#254)#424galaxyMoonStar wants to merge 8 commits into
galaxyMoonStar wants to merge 8 commits into
Conversation
The metrics summary now carries training progress alongside values, so the log reads at a glance: - new summary fields: last_step, min_step, max_step, mean, recent_steps - table: a 'steps A -> B' line, a mean line, 'last V (step N)', and recent paired with absolute steps; the hand-written sparkline trend is unchanged - JSON: the five new keys, additive only (recent/trend shape preserved) Aggregation stays a single streaming pass (O(1) memory); min_step/max_step span the retained [-500:] window, not full training history. The dashboard read-side and its equivalence guard are untouched -- the new fields live in the CLI-only summarize/render layer. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #254
Summary
areno metricssubcommand that reads a single metric's history from localevents.out.tfevents.*and printslast/min/max/mean/steps/recent/trendas a table (with a hand-written sparkline) or JSON.areno/dashboard/server.pyinto a new shared moduleareno/api/metric_reader.py, so the dashboard and the CLI now share one reader — a single source of truth — instead of two parallel implementations.Usage
Table output:
JSON output (
--json):{"name":"rollout/rewards_mean","count":5,"last":1.0,"last_step":4,"min":0.5,"max":1.0, "mean":0.75,"min_step":0,"max_step":4,"recent":[0.5,0.625,0.75,0.875,1.0], "recent_steps":[0,1,2,3,4],"trend":[0.0,0.25,0.5,0.75,1.0]}Defaults:
--metrics-dir=/tmp/areno/tfevent,--limit=20, table output.What Changed
areno/api/metric_reader.pyareno/cli/metrics.pyareno metricssubcommand entryareno/cli/main.py_COMMANDS)areno/dashboard/server.pymetric_reader; dead code removed; constant consolidated (+20 / -47)areno/api/defaults.pyDEFAULT_METRICS_LOG_DIRsingle source, imported by both sidesdocs/cli/metrics.rst,docs/reference/cli.rsttests/test_metric_reader_cpu.pytests/test_metric_reader_equivalence_cpu.pytests/test_cli_metrics_cpu.pyTotal: +1182 / -47.
Design Decisions
areno/api/metric_reader.pyand have both dashboard and CLI call it. This directly honors the issue's 4× emphasis on "reuse existing contracts, do not introduce a parallel subsystem." Two readers would drift; one shared reader evolves in one place.metric_reader. The move is strict "move, don't rewrite": thesize_guidance=10000/[-500:]/ NaN-skip /(name,step,value)dedup semantics are preserved verbatim. Any incidental cleanup is deferred.sparklines/rich);tensorboardwas already declared inpyproject.toml. The CLI stays a light, pure query path.dashboard.serverdirectly — That would drag the ~1600-linehttp.serverinto a pure CLI and invert the dependency direction (cli → dashboardinstead of both →api). Rejected on layering grounds.Tests
27 CPU tests across 3 files (no GPU, no network,
ARENO_BUILD_EXT=0):test_metric_reader_equivalence_cpu.pySummaryWriterfixture (job.metrics+job.timeperf)test_metric_reader_cpu.pytest_cli_metrics_cpu.py--jsonRun:
Kaggle end-to-end run on a real SFT training (real
events.out.tfevents.*) is tracked in the local execution note.Notes for Reviewer
dashboard/server.pyrefactor. It is behavior-preserving and guarded by the equivalence test — if the test passes, dashboard output is unchanged.size_guidance=10000caps how many scalars TensorBoard loads per tag,[-500:]keeps only the tail window, and the summary is computed over that bounded window (extra memory independent of total training length).