feat: overlay structured training events on metric charts (#271) - #432
Open
wenzeyu8888-rgb wants to merge 1 commit into
Open
feat: overlay structured training events on metric charts (#271)#432wenzeyu8888-rgb wants to merge 1 commit into
wenzeyu8888-rgb wants to merge 1 commit into
Conversation
…I#271) - Add detect_training_events() read-only derivation in server.py for four event kinds: non_finite, constant_reward, invalid_batch, oom - Fix NaN silent discard in _load_tensorboard_scalars (L338): NaN now produces non_finite events instead of being silently dropped - Add GET /api/jobs/<id>/events route (before catch-all) with ?types= and ?limit= query params; events route reads job._events without triggering get_job to avoid lock contention - Add fetch_events agent tool schema and handler - Add EventOverlay/EventFilters/EventContextPopover React components with click-through: OOM -> keyword log window (+/-20 lines), non-OOM -> metric context (step+/-3) - Add eventUtils.js (ESM pure functions) testable via node --input-type=module - Invalid batch streak: >=3 warn, 1-2 info (aligns with issue 'streaks') - OOM: severity=error, no recovery tracking (process crash makes it impossible within same job) - 29 CPU tests: success/malformed/boundary/overlap/legacy/route-order - Node smoke tests for eventUtils.js - Docs: API contract, limitations (500-step truncation, OOM semantics, NaN behaviour change), copyable Python fixture example
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements #271: overlay clickable structured training events on TensorBoard metric charts.
Changes
Backend (
areno/dashboard/server.py)detect_training_events(): read-only derivation of 4 event kinds from existing metrics + logs (no new artifact files, no trainer/metrics changes)non_finite: NaN/Inf in TensorBoard scalars (fixes silent discard at L338)constant_reward:rollout/rewards_std <= 1e-9withrewards_max == rewards_mindouble-confirminvalid_batch:rollout/advantages_std <= 1e-6consecutive streaks (>=3 warn, 1-2 info)oom: OOM pattern in captured subprocess logs (severity=error, no recovery tracking)get_job_events(): lightweight read of cachedjob._eventswithout triggeringget_job/_save_stateGET /api/jobs/<id>/eventsroute inserted before catch-all, with?types=and?limit=paramsfetch_eventsagent tool schema + handler_load_metric_filesto avoid lock contentionFrontend (
dashboard/src/)eventUtils.js(new): ESM pure functions (step-to-x mapping, event filter, log window, metric context) — testable vianode --input-type=moduleEventOverlay: SVG markers (cross/diamond/square/triangle by kind) on chartEventFilters: 4 checkboxes independent of metric curveEventContextPopover: click-through withlog_hint.kind3-way split — OOM shows +-20 log lines, non-OOM shows metric context (step+-3), none shows metadata onlyTests
tests/test_dashboard_events_cpu.py: 29 CPU tests covering success/malformed/boundary/overlap/legacy/route-order/injection-positioneventUtils.jsDocs
docs/reference/dashboard-training-events.rst: API contract, output fields, limitations (500-step truncation, OOM semantics, NaN behaviour change), copyable Python fixture exampleArchitecture
All 4 event kinds are read-only derived from existing TensorBoard scalars + captured logs at
_load_metric_filestime. No new artifact files, no new producers, no trainer/metrics changes — strictly per issue constraint: "reuse existing data ... rather than introducing a parallel subsystem".Test results
Acceptance criteria
Closes #271