feat: Anomaly Detector toolbox + headless runner + shared toolbox_preview engine#171
Draft
Alvvalencia wants to merge 22 commits into
Draft
feat: Anomaly Detector toolbox + headless runner + shared toolbox_preview engine#171Alvvalencia wants to merge 22 commits into
Alvvalencia wants to merge 22 commits into
Conversation
A toolbox plugin (modeled on toolbox_quaternion) whose dialog buttons create plot markers through the pj.marker_store.v1 service — the "simulator" producer standing in for the future Lua detector library, which will emit markers through the same MarkerStoreHostView API. - Add Region / Add Event: read the first series from the catalog (dataset = topic.source.id, time span from the Arrow series) and markerHost.add(dataset, "__global__", marker) so it renders on every plot of the dataset; Clear: query + remove by id. - markers_plugin.cpp + markers_dialog.ui + manifest.json + conanfile.py + CMakeLists.txt; registered in both dual-mode blocks of the top-level CMakeLists. Building this requires an SDK with pj.marker_store.v1 (currently on the plotjuggler_sdk feature/plot-markers branch; pin once released to Conan). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… set) The pj.marker_store.v1 service was removed. The plugin now owns its marker set and republishes the whole PlotMarkers blob via registerObjectTopicOnDataset + pushOwnedObject on the dataset's global marker object topic. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…lobal scope Replace the three demo buttons with a real producer UI: - Drag timeseries into a table (inputFrame drop target) to mark them per-series; markers publish to the dropped 'topic/field' object topic so the overlay draws them only on plots showing that series. - Tick 'whole dataset (global)' + a dataset selector to target one or all loaded datasets' global marker topic. - A time RangeSlider (spanning the data's time window) positions Region start/end and Event instant. - Markers are kept per (dataset, topic) and republished wholesale on change. The per-series 'topic/field' -> dataset index is built lazily at Add time. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…h, retention & deferred Save Replace the single-range slider with the MarkerTimeline editor: Add time range/event drop draggable marks, edited in place, applied on Save (deferred publish) and removed via Clear. Trash-icon Remove-series; titled Series table. Correctness from the marker review: store the draft in absolute ns (no remap when the window changes); key series_dataset_ via sdk::markerSeriesKey so per-series markers publish to the right dataset; read existing markers back dataset-scoped (lookupTopicOnDataset) and load on every drop; cap each marker topic to keep-latest via setObjectTopicRetention; Clear now republishes empty to the listed + ever-saved topics (honest tooltip). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A GUI toolbox to author Lua anomaly-detection rules that emit plot markers, and a headless anomaly_runner CLI that runs the SAME rules without a GUI for servers/CI. Both link the GUI-free anomaly_core engine (sol2 Lua, kissfft bandPower, 13 builtins, JSON report, portable JSON rules), so a rule authored and tested in the GUI produces identical markers headless. - toolbox_anomaly_detector/: Filter-Editor-style GUI plugin + core/anomaly_core shared static lib + README (GUI guide, builtin table, Lua API). - tools/anomaly_runner/: SDK-only CLI (no Qt). CSV direct ingest, MCAP delegated to the matching parser .so by encoding. JSON report via nlohmann::ordered_json; exit 0=pass / 1=fail / 2=usage-error for pipeline gating. - Series keys use sdk::markerSeriesKey in the runner so GUI and headless resolve the identical series name (no spurious double slash on ROS/MCAP fields). - README plugins table, CLAUDE.md (Task F is new, not a port) updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RcygzPWUrHf9jHGGwzEzZo
…ve sources - Save/Load rule now use native file dialogs (setSaveFilePicker / setFilePicker); the manual rule-path field is gone. Save-as creates a new .json anywhere. - The preview overlays the rule's detected markers live (setChartMarkers) so you see what Apply will publish; markers recompute on source change — a builtin rule re-targets the newly selected curve automatically (hand-edited Lua is preserved). - The source list refreshes live (onTick + a cheap catalog-signature guard) so a dataset loaded after the panel opened appears without reopening the toolbox. - Reworked the dialog .ui: a vertical splitter (preview on top) over grouped, bordered Source / Function / Lua panels. - makeSeriesProvider() removes the duplicated SeriesProvider setup. - README rewritten: concise, usage-focused. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RcygzPWUrHf9jHGGwzEzZo
Migrate the Anomaly Detector (GUI toolbox + headless anomaly_runner) off its own sol2 Lua VM onto the shared pj_scripting_core Luau engine — the same engine PlotJuggler uses for filters. One engine, GUI == headless, on the boss's Luau north star. Behavior-preserving: the runner parity holds via Luau (CSV Spike -> 7/fail/exit 1, MCAP /sensor/value/data -> 36/fail/exit 1). - anomaly_core: SeriesAccessor/SeriesProvider are now aliases of the engine's SeriesView/SeriesProvider; runAnomalyScript() is a thin delegate to PJ::scripting::runMarkerScript(). The sol2 runtime + marker primitives + bandPower (~293 lines) are gone — they live in pj_scripting_core. Kept the GUI-free helpers: builtins, substituteSource, markersToReport, rule<->json. - CMake/Conan: link pj_scripting_core, drop lua/sol2/kissfft from both targets. luau/*:fPIC=True so Luau's static libs link into the plugin .so. - Docs (CLAUDE.md + doc-comments) updated to the shared-Luau reality. The consumer .cpp (anomaly_detector.cpp, runner main.cpp) needed ZERO changes — the alias + delegation kept the anomaly_core surface intact. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RcygzPWUrHf9jHGGwzEzZo
…batch watcher Completes Task F deliverable #6 (notification integration) and the deploy/docs half of #7. Opt-in via --notify <config.json>; without it the runner is identical to before (zero regression). Notifications are a deployment concern, kept OUT of the portable rule JSON so a rule stays shareable and secrets stay on the server. - notify.{hpp,cpp}: NotifyConfig (notify_on policy + sinks), ${ENV} expansion for secrets, pure shouldNotify predicate, dispatch. Sinks: webhook (libcurl HTTP POST), email (libcurl SMTP), command (fork/exec, report on stdin — integrate any existing alerting). Decoupled from anomaly_core: operates on the report JSON. - main.cpp: --notify + --notify-strict (delivery failure -> exit 3, overriding the pass/fail verdict). Config validated up front (exit 2 on a typo). - Build: libcurl/8.10.1 + gtest on the runner; CMake adds an anomaly_notify lib + notify_test (14 cases). Root conanfile.py also gains pj_scripting_core (it was missing from the full-build union since the Luau migration) + libcurl. - deploy/watch.sh: dependency-free batch watcher — processes each new MCAP once (atomic, de-duped), runs the runner with --rule --notify. README documents cron / systemd.path / inotify wrappers. - examples/: ready-to-run rule + notify configs + a webhook listener, so the CLI can be exercised without hand-writing any JSON. - Docs: runner README "Configuring notifications" + "Deploying the batch watcher"; cross-links from the toolbox/root README + CLAUDE.md dep table. Verified: notify_test 14/14; command sink receives the exact report; notify_on fires on fail but not on pass; --notify-strict -> exit 3; a real webhook POST delivered the report with the ${ENV}-expanded Authorization header; watcher dedup. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RcygzPWUrHf9jHGGwzEzZo
…y Detector) The manual marker-timeline editor is removed; PlotMarkers rendering and the sdk::PlotMarkers type stay (used by the Anomaly Detector + renderer). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BeoYQHmF6ixvYv9KXXgyVh
Re-sync to merged SDK 0.12.0 (PlotMarkers builtin). SDK_VERSION 0.10.0 -> 0.12.0; pj_scripting_core now pins 0.12.0. CMakeLists auto-merged (main's plugin list + our toolbox_markers removal). Verified: anomaly_runner CSV Spike -> 7 / fail / exit 1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BeoYQHmF6ixvYv9KXXgyVh
…ore split; unified README
The Anomaly Detector GUI plugin becomes host-driven and Luau-FREE: it submits
rules to the host via pj.markers.v1 (the host runs + recomputes them) and the live
preview is host-driven too (an ephemeral generator read back through the object
store). The headless anomaly_runner is unchanged (standalone engine). Both run the
rule through the same runMarkerScript core → identical markers ("GUI == headless").
- core split: anomaly_helpers (Luau-free: builtin library + JSON report/rule
helpers, linked by the plugin) vs anomaly_core (adds runAnomalyScript on
pj_scripting_core, linked by the runner). The plugin .so no longer links Luau
(~1.67MB → ~471KB, 0 Lua symbols).
- anomaly_detector.cpp: runScript → createMarkerGenerator; previewMarkers →
setPreview + read-back; rule errors surfaced in the status line; clearPreview on
close; removed the dead in-process publish path.
- Global marker scope: per-dataset OR all-datasets (new .ui checkbox; {"scope":"all"}
in params_json).
- README.md rewritten as a single GUI + headless guide; CLAUDE.md Task F paragraph
+ tools/anomaly_runner/README.md updated to the host-driven architecture.
- Bumps the SDK submodule to the pj.markers.v1 commit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QsY1SE2yBsX2qXvkgEnX8U
…eadable marker API
Follows the pj_scripting rename (createMarker + create{Vertical,Horizontal,Point,Band}Marker):
- builtin templates (anomaly_helpers.cpp) now use the explicit shortcuts where they read
best: createVerticalMarker / createHorizontalMarker / createPointMarker / createBandMarker.
- toolbox + runner READMEs document the new family.
- fixes the runner examples (examples/custom_rule.lua, examples/spike_rule.json) that still
called the removed createEvent -> they now use createPointMarker.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QsY1SE2yBsX2qXvkgEnX8U
Pull the SDK 0.11.0 bump + aggregate-build hardening from main into the markers feature branch. Kept SDK_VERSION at 0.12.0 (feature version supersedes main's 0.11.0; merged SDK is a superset). extern/plotjuggler_core recorded at the feature<-main merge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rh6GtzBNRVXXBUhFWhXEam
…vice) The Anomaly Detector submits its rule via GeneratorsHostService (createMarkerGenerator, kind="markers") instead of the removed MarkersHostService. No behavior change: the host still runs the rule and publishes markers; the headless runner is unchanged. - SDK_VERSION -> 0.13.0. - Advance extern/plotjuggler_core gitlink -> 9853aa9 (the pushed SDK commit carrying pj.generators.v1) so a clean from-source build compiles. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013N1nzqco4Y9AQhnQgXo5tQ
…preview freeze
Extract the generator-toolbox preview machinery out of the Anomaly Detector into a new
UI-free common/toolbox_preview static lib (SeriesCatalog, EphemeralPreview, AdaptivePoll)
behind a PreviewBackend seam, so the upcoming Transform Editor toolbox can reuse it. The
Anomaly Detector now supplies only the marker-specific render (submit/read-back/teardown).
Fix the live-preview freeze on large datasets (nissan_zala): the ephemeral marker generator
declared the ENTIRE catalog as inputs, and the host materializes every declared input
whole-series on each run on the GUI thread. parseSeriesRefs now declares only the series the
rule actually reads via series("..."). The marker read-back is change-gated on
SeriesCatalog::dataRevision() (deserialize only when data changed, not every 20 Hz tick), and
the in-dialog preview overlay is capped at 2000 markers via stride-decimation. Committed
(Apply) markers are uncapped. GUI==headless preserved (host runs the same rule engine).
Includes unit tests for the cadence policy (AdaptivePoll) and the change-gated read-back.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013N1nzqco4Y9AQhnQgXo5tQ
…kind=markers) The Anomaly Detector now submits its rule through the unified host service pj.data_processors.v1 with kind="markers": DataProcessorsHostService / DataProcessorsHostView::createMarkers / PJ_DATA_PROCESSOR_FLAG_EPHEMERAL, replacing the interim pj.generators.v1. The shared common/toolbox_preview seam (PreviewBackend) needed no change — it is service-agnostic. - toolbox_anomaly_detector/anomaly_detector.cpp: createMarkers + DataProcessorsHostService. - CLAUDE.md / toolbox_anomaly_detector/README.md: doc aligned to pj.data_processors.v1. - Bump extern/plotjuggler_core to bf7eb0f (the unified SDK). Compiled + linked against the unified SDK 0.13.0; toolbox_preview_test passes; verified end-to-end in the GUI (markers render via the unified service). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FHPK8VduUcoenjL8AtoSVd
# Conflicts: # .gitignore
… contract The "Save rule as..." picker now declares its extension via the new setSaveFilePicker(..., "json") arg (SDK PR #138). The host appends ".json" when the user types a name without one, so the manual extension fix-up in saveRuleTo() is removed — the host is the single source of truth (verified E2E: "a" -> "a.json"). DEPENDENCY: the 5-arg setSaveFilePicker requires the SDK to carry #138. The extern/plotjuggler_core gitlink bump to a #138-containing commit follows once #138 merges into the SDK feature/plot-markers branch; held local (not pushed) until then. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122VUqQqWd1KVQQfhrG7632
… CI is green
The GUI Anomaly Detector is host-driven: it submits its rule to the host via the
SDK data-processors service and the host runs Lua. The plugin .so therefore links
only the Luau-free `anomaly_helpers`, never the Luau engine. But the build still
pulled in `pj_scripting_core` (the private Luau engine package), which:
- broke the Linux per-plugin leg (toolbox_anomaly_detector/conanfile.py declared
a `pj_scripting_core` require the .so never links), and
- broke the macOS/Windows monolithic build, which has no cloudsmith remote and
compiled `tools/anomaly_runner` (the only genuine engine consumer) unconditionally.
Fix — drop the spurious dependency and gate the engine layer behind the runner:
- toolbox_anomaly_detector/conanfile.py: remove `pj_scripting_core` + the now-moot
`luau/*:fPIC` option. The GUI plugin needs only plotjuggler_sdk + nlohmann_json.
- core/CMakeLists.txt: build `anomaly_core` (+ find_package(pj_scripting_core))
ONLY when PJ_BUILD_ANOMALY_RUNNER is set. `anomaly_helpers` always builds.
- CMakeLists.txt (root): add option(PJ_BUILD_ANOMALY_RUNNER OFF) and gate
add_subdirectory(tools/anomaly_runner) in both the subdirectory and standalone lists.
- tools/anomaly_runner/CMakeLists.txt: set(PJ_BUILD_ANOMALY_RUNNER ON) before adding
the core subdir, so building the runner pulls in the engine layer.
- conanfile.py (root): remove `pj_scripting_core` from the full-repo superset.
The runner is unchanged functionally: build it on demand with
`./build.sh tools/anomaly_runner` (its recipe provides pj_scripting_core) or
-DPJ_BUILD_ANOMALY_RUNNER=ON. Verified end-to-end: the runner builds, lists the 12
builtins, detects a spike, and honors the pipeline exit-code contract (fail=1, pass=0,
usage=2). GUI == headless preserved.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0122VUqQqWd1KVQQfhrG7632
…eeds it) The standalone root CMakeLists' find_package(GTest REQUIRED) runs for every per-plugin build, but toolbox_anomaly_detector/conanfile.py never listed gtest (every other plugin recipe does). The omission was masked because the Linux per-plugin leg died earlier resolving pj_scripting_core; with that spurious dependency gone (prev commit), Conan now succeeds and CMake configure failed on "Could NOT find GTest". Add gtest/1.17.0 to the recipe, matching the other plugins. Verified via the per-plugin path (./build.sh toolbox_anomaly_detector): GTest resolves, configure succeeds, the .so links. NOTE: this leg still fails to COMPILE against the published SDK (0.13.0) because anomaly_detector.cpp adopts the 5-arg setSaveFilePicker(default_suffix) contract, which lives in the not-yet-merged SDK default_suffix PR. That is the known, planned convergence dependency — it greens once the unified SDK PR merges and SDK_VERSION is bumped. This commit removes the gtest blocker so #138 is the single remaining one. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122VUqQqWd1KVQQfhrG7632
…th default_suffix) The Anomaly Detector adopted the SDK default_suffix save-file-picker contract (anomaly_detector.cpp:184 calls the 5-arg setSaveFilePicker), but the pinned SDK submodule (4ed853a) predated that API — its setSaveFilePicker took 4 args, so the plugin failed to compile on all three platforms (the last remaining CI blocker, #3). Pablo unified the markers PR (#135) and the default_suffix PR (#138) into #135; a3d15132 = 4ed853a + the single "feat(dialog): add save-file picker (#138)" commit, which adds the 5-arg setSaveFilePicker (default_suffix="") and the saveFilePickerDefaultSuffix accessor. Move the gitlink forward to it. SDK_VERSION stays 0.13.0: cloudsmith has no prebuilt 0.13.0 (confirmed in the acquire-core log), so CI builds the SDK from this submodule — there is no version collision, and macOS/Windows build core from source anyway. Verified with a clean rebuild (not the locally-patched cache): purged 0.13.0 from the Conan cache, rebuilt the SDK from this submodule via conan create, and built toolbox_anomaly_detector — anomaly_detector.cpp compiles and the .so links. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122VUqQqWd1KVQQfhrG7632
#135 (PlotMarkers builtin + unify markers/transforms into pj.data_processors.v1, incl. the default_suffix save-file-picker contract) is now merged to SDK main as the squash commit ca052ce. Move the submodule gitlink off the feature-branch commit (a3d15132) onto the landed main commit. The tree is identical (verified: git diff a3d15132 ca052ce is empty), so this is a pointer-only change — builds stay green, no recompile. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122VUqQqWd1KVQQfhrG7632
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Integration PR for the plot-markers feature on the plugins side. Delivers the Anomaly Detector as a two-mode tool plus the shared preview engine it's built on:
toolbox_anomaly_detector/) — a host-driven, Luau-free plugin. It submits a rule to the host viapj.generators.v1(GeneratorsHostService); the host runs and re-runs it as data changes and publishes the markers. Live preview is host-driven too (ephemeral generator read back through the object store).tools/anomaly_runner/) — standalone analyzer running the same rule in-process via the shared Luau engine (pj_scripting_core). CSV/MCAP in → JSON report + exit code out, for CI; webhook/email/command sinks.runMarkerScriptcore and emit the samePlotMarkers.Latest in this push
common/toolbox_preview/(UI-free):SeriesCatalog(source enumeration + cache + streaming live-refresh),EphemeralPreview(change-gated submit + read-back),AdaptivePoll(cadence policy), behind aPreviewBackendseam. Reusable by the upcoming Transform Editor toolbox; the Anomaly Detector now supplies only the marker-specific render.parseSeriesRefsnow declares only the series the rule actually reads viaseries("..."). The marker read-back is change-gated onSeriesCatalog::dataRevision()(deserialize on data change, not every 20 Hz tick); the in-dialog overlay is stride-capped at 2000 markers (committed markers uncapped).Test plan
toolbox_preview_test— 7/7 green (AdaptivePoll cadence + EphemeralPreview change-gate)../build.sh toolbox_anomaly_detectorbuilds clean (lib + plugin).🤖 Generated with Claude Code