Skip to content

feat: overlay structured training events on metric charts (#271) - #432

Open
wenzeyu8888-rgb wants to merge 1 commit into
inclusionAI:mainfrom
wenzeyu8888-rgb:feat/dashboard-training-events
Open

feat: overlay structured training events on metric charts (#271)#432
wenzeyu8888-rgb wants to merge 1 commit into
inclusionAI:mainfrom
wenzeyu8888-rgb:feat/dashboard-training-events

Conversation

@wenzeyu8888-rgb

Copy link
Copy Markdown

Summary

Implements #271: overlay clickable structured training events on TensorBoard metric charts.

Changes

Backend (areno/dashboard/server.py)

  • detect_training_events(): read-only derivation of 4 event kinds from existing metrics + logs (no new artifact files, no trainer/metrics changes)
    • non_finite: NaN/Inf in TensorBoard scalars (fixes silent discard at L338)
    • constant_reward: rollout/rewards_std <= 1e-9 with rewards_max == rewards_min double-confirm
    • invalid_batch: rollout/advantages_std <= 1e-6 consecutive streaks (>=3 warn, 1-2 info)
    • oom: OOM pattern in captured subprocess logs (severity=error, no recovery tracking)
  • get_job_events(): lightweight read of cached job._events without triggering get_job/_save_state
  • GET /api/jobs/<id>/events route inserted before catch-all, with ?types= and ?limit= params
  • fetch_events agent tool schema + handler
  • Detection injected at end of _load_metric_files to avoid lock contention

Frontend (dashboard/src/)

  • eventUtils.js (new): ESM pure functions (step-to-x mapping, event filter, log window, metric context) — testable via node --input-type=module
  • EventOverlay: SVG markers (cross/diamond/square/triangle by kind) on chart
  • EventFilters: 4 checkboxes independent of metric curve
  • EventContextPopover: click-through with log_hint.kind 3-way split — OOM shows +-20 log lines, non-OOM shows metric context (step+-3), none shows metadata only

Tests

  • tests/test_dashboard_events_cpu.py: 29 CPU tests covering success/malformed/boundary/overlap/legacy/route-order/injection-position
  • Node smoke tests for eventUtils.js

Docs

  • docs/reference/dashboard-training-events.rst: API contract, output fields, limitations (500-step truncation, OOM semantics, NaN behaviour change), copyable Python fixture example

Architecture

All 4 event kinds are read-only derived from existing TensorBoard scalars + captured logs at _load_metric_files time. No new artifact files, no new producers, no trainer/metrics changes — strictly per issue constraint: "reuse existing data ... rather than introducing a parallel subsystem".

Test results

  • Python: 29/29 PASSED (macOS Python 3.9 + Kaggle Python 3.12)
  • Node: 20/20 PASSED
  • Frontend build: vite build OK (1831 modules)
  • Kaggle GPU: verified end-to-end with fake fixture metrics — events detected, API returns correct results, dashboard UI shows markers + click-through popover

Acceptance criteria

  • Test overlapping events, legacy runs without events, click-through to bounded log context, active updates, and no mutation of metric data
  • The implementation uses existing AReno contracts and introduces no external database or mandatory sandbox
  • Default behavior remains backward compatible
  • Focused automated tests cover success, invalid input, and one boundary/failure path
  • User documentation includes a minimal runnable example and explains observable output

Closes #271

…I#271)

- Add detect_training_events() read-only derivation in server.py for four
  event kinds: non_finite, constant_reward, invalid_batch, oom
- Fix NaN silent discard in _load_tensorboard_scalars (L338): NaN now
  produces non_finite events instead of being silently dropped
- Add GET /api/jobs/<id>/events route (before catch-all) with ?types= and
  ?limit= query params; events route reads job._events without triggering
  get_job to avoid lock contention
- Add fetch_events agent tool schema and handler
- Add EventOverlay/EventFilters/EventContextPopover React components with
  click-through: OOM -> keyword log window (+/-20 lines), non-OOM ->
  metric context (step+/-3)
- Add eventUtils.js (ESM pure functions) testable via node --input-type=module
- Invalid batch streak: >=3 warn, 1-2 info (aligns with issue 'streaks')
- OOM: severity=error, no recovery tracking (process crash makes it
  impossible within same job)
- 29 CPU tests: success/malformed/boundary/overlap/legacy/route-order
- Node smoke tests for eventUtils.js
- Docs: API contract, limitations (500-step truncation, OOM semantics,
  NaN behaviour change), copyable Python fixture example
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Overlay structured training events on metric charts

1 participant