From 61bd1c47380dddfe96c51e29c58de115eef683ff Mon Sep 17 00:00:00 2001 From: Carter Francis Date: Fri, 7 Aug 2026 11:32:03 -0500 Subject: [PATCH 1/3] docs: pin SIGNING/RELEASING/model-release/shutdown-wedge knowledge into code Ahead of deleting the standalone .md docs, move the content that still matters to where it is consumed: - verify_signing_secrets.yml: the cert-acquisition walkthrough (Developer ID Application cert, the G2 intermediate-chain gotcha, .p12 export with the intermediate baked in, App Store Connect API key) + the six-secret table move into the workflow header; release.yml's signing section now points there. - prepare_release.yml: the manual pre-release checks the automated suite cannot cover (distributed navigator drag, real GPU FV/OM run, no-GPU check of the packaged app, IPF colour-key legend, clean shutdown, and the post-tag gh-release-view asset check) become literal checkboxes in the auto-generated release PR body. - runner.ts stopSpyDE: document the open Windows shutdown-wedge bug (quit during a streaming find-vectors batch stops the stdin tick that keeps the hidden backend scheduled; e2e waits for '[fv-batch] finalized' before closing). - spyde/models/registry.py: the ship-a-revised-model contract (train -> validate via benchmark_neural_spots -> upload to HF under a NEW versioned filename, never overwrite -> edit registry.json) moves into the module docstring; models/__init__.py, find_vectors_action.py and pyproject.toml now point at it instead of models/RELEASING.md. --- .github/workflows/prepare_release.yml | 30 +++++++- .github/workflows/release.yml | 7 +- .github/workflows/verify_signing_secrets.yml | 74 +++++++++++++++++++- electron/src/main/runner.ts | 8 +++ pyproject.toml | 2 +- spyde/actions/find_vectors_action.py | 4 +- spyde/models/__init__.py | 2 +- spyde/models/registry.py | 25 ++++++- 8 files changed, 140 insertions(+), 12 deletions(-) diff --git a/.github/workflows/prepare_release.yml b/.github/workflows/prepare_release.yml index 0e3a49da..4310af75 100644 --- a/.github/workflows/prepare_release.yml +++ b/.github/workflows/prepare_release.yml @@ -220,10 +220,28 @@ jobs: ### Review checklist - [ ] Version is correct in \`electron/package.json\` - - [ ] Manual checks from RELEASING.md done on a real machine (GPU path, - distributed navigator, clean shutdown) - [ ] CI passes + ### Manual checks the automated suite cannot cover + Do these on a real machine — CI has no GPU/display and the migrated + suite forces \`SPYDE_NO_DASK=1\`: + - [ ] **Distributed navigator path** — open a multi-GB 4D-STEM scan, drag the + navigator; the diffraction pattern must track without freezing. If the + \`repro_*.py\` distributed scripts are present, run them directly + (\`uv run python -m spyde.tests.repro_\`); they spin a real + \`LocalCluster\` and won't run under pytest. + - [ ] **GPU find-vectors / orientation** — on a CUDA box, run Find Vectors and + Orientation Mapping on a real dataset (e.g. \`pyxem.data.sped_ag()\`); confirm + results look right and nothing segfaults. (The numba subpixel kernel is unit- + tested for arithmetic, but the live CUDA path needs a real run.) + - [ ] **No-GPU fallback** — on a machine without \`numba\`/CUDA, the app must still + launch and Find Vectors must fall back to CPU. (Guarded by + \`test_find_vectors_no_numba.py\`, but verify the PACKAGED app.) + - [ ] **IPF colour-key legend** — open an Orientation map; the colour-key triangle + pins in the corner on the 2-D map and hides on the 3-D view. + - [ ] **Clean shutdown** — quit the app; confirm no orphaned \`python.exe\` / Dask + worker processes remain (Task Manager / \`ps\`). + ### After merging Tag **the merge commit on main** — the tag MUST be exactly \`${TAG}\`: \`\`\`bash @@ -232,4 +250,10 @@ jobs: git push origin ${TAG} \`\`\` The tag push triggers \`release.yml\` (3-platform build → draft release → - publish once all legs pass)." + publish once all legs pass). + - [ ] **Confirm the release actually has installers** before announcing it: + \`gh release view ${TAG} --repo CSSFrancis/spyde --json assets --jq '.assets[].name'\` + — expect the three installers (\`SpyDE Setup *.exe\`, \`SpyDE-*.dmg\`, + \`SpyDE-*.AppImage\`), their \`.blockmap\`s, and the \`latest*.yml\` update + feeds. \`release.yml\`'s \`finalize\` job refuses to un-draft an + asset-less release, but check." diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 43308c64..d1eee523 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -244,7 +244,9 @@ jobs: # ── macOS code signing (only when secrets are present) ───────────────────── # Import the Developer ID Application cert into a throwaway keychain so - # electron-builder can sign + (with notarize:true) notarize. See SIGNING.md. + # electron-builder can sign + (with notarize:true) notarize. The full + # cert-acquisition + secret-setup walkthrough (and the six-secret table) + # lives in .github/workflows/verify_signing_secrets.yml's header comment. # Gated on both runner.os == macOS AND the cert secret being set, so a fork # / secret-less run skips it and electron-builder produces an unsigned build # (rather than failing) — matching electron-builder.yml's no-identity-null. @@ -288,7 +290,8 @@ jobs: # update-feed metadata (latest.yml / latest-mac.yml / latest-linux.yml) # electron-updater's autoUpdater reads at runtime — publish:false builds # never generate these files at all. On macOS the app is signed + notarized - # when MAC_SIGNING_ENABLED (see the signing steps above + SIGNING.md). + # when MAC_SIGNING_ENABLED (see the signing steps above + + # verify_signing_secrets.yml). # -c.publish.owner/.repo pin the publish target to THE REPO THIS WORKFLOW # RUNS IN — the same-repo assumption the draft-release design above already # makes (GITHUB_TOKEN can't publish cross-repo anyway). Without the diff --git a/.github/workflows/verify_signing_secrets.yml b/.github/workflows/verify_signing_secrets.yml index ebb240db..d38f1e30 100644 --- a/.github/workflows/verify_signing_secrets.yml +++ b/.github/workflows/verify_signing_secrets.yml @@ -11,8 +11,78 @@ # Apple with the API key + key-id + issuer + team-id, proving those four all # work together (fails loudly if any is wrong), without submitting a build. # -# Run: Actions tab → "Verify signing secrets" → Run workflow. Delete this file -# once signing is confirmed working end-to-end, or keep it as a pre-release probe. +# Run: Actions tab → "Verify signing secrets" → Run workflow. Keep this file as +# a pre-release probe — it is also the home of the secret-setup walkthrough below. +# +# ═══════════════════════════════════════════════════════════════════════════════ +# HOW THE SIX SECRETS ARE MADE (one-time cert/key setup, done on a Mac) +# ═══════════════════════════════════════════════════════════════════════════════ +# +# Signing an Apple app = two required steps since Catalina: (1) CODE-SIGN the +# .app/.dmg/.zip with a Developer ID Application cert, and (2) NOTARIZE (upload +# to Apple, staple the ticket). electron-builder does both during `npm run dist` +# (notarize: true in electron/electron-builder.yml); release.yml's mac leg +# imports the cert + stages the API key, gated on MAC_SIGNING_ENABLED (true when +# MAC_CERT_P12_BASE64 is set — a fork/secret-less run builds UNSIGNED, not red). +# +# Step 1 — Create the certificate. Distributing OUTSIDE the App Store needs a +# "Developer ID Application" cert — NOT "Mac App Distribution" and NOT +# "Developer ID Installer" (that one is only for .pkg installers, which SpyDE +# doesn't ship). Xcode → Settings → Accounts → your Apple ID → Manage +# Certificates → + → Developer ID Application (or developer.apple.com → +# Certificates, IDs & Profiles → Certificates → +). Lands the cert + private +# key in your login keychain. +# +# Intermediate-chain gotcha: the cert chains up through Apple's CURRENT +# "Developer ID Certification Authority (G2)" intermediate — NOT the older +# non-G2 "Developer ID Certification Authority" (being retired; new certs +# chain to G2): +# Apple Root CA +# └─ Developer ID Certification Authority (G2) ← the intermediate +# └─ Developer ID Application: You (TEAMID) ← your signing cert +# You normally don't pick this by hand, but a CI runner's keychain missing the +# G2 intermediate shows up as "unable to build chain" / CSSMERR_TP_NOT_TRUSTED. +# Guard against it by exporting the full chain into the .p12 (Step 2) — and +# release.yml belt-and-braces fetches +# https://www.apple.com/certificateauthority/DeveloperIDG2CA.cer into the CI +# keychain anyway. +# +# Step 2 — Export as .p12 (CI has no keychain). In Keychain Access, Cmd-click +# BOTH "Developer ID Application: Your Name (TEAMID)" AND "Developer ID +# Certification Authority (G2)" (download + double-click the G2 .cer first if +# it isn't shown), right-click → Export … (2 items) → certificate.p12 with an +# export password. Then: base64 -i certificate.p12 -o certificate.p12.base64 +# +# Step 3 — App Store Connect API key (for notarization; preferred over an +# app-specific password — revocable, no 2FA prompts). appstoreconnect.apple.com +# → Users and Access → Integrations tab → App Store Connect API → + → role +# "Developer". DOWNLOAD THE .p8 — you can only download it once. Note the Key +# ID and (top of that page) the Issuer ID (a UUID). Your Team ID (10 chars) is +# on developer.apple.com → Membership. +# +# Step 4 — The six repo secrets (Settings → Secrets and variables → Actions): +# +# | Secret | Value | +# |----------------------|----------------------------------------------------| +# | MAC_CERT_P12_BASE64 | contents of certificate.p12.base64 (Step 2) | +# | MAC_CERT_PASSWORD | the .p12 export password (Step 2) | +# | APPLE_API_KEY_P8 | full contents of the .p8 file (Step 3) | +# | APPLE_API_KEY_ID | the API Key ID (Step 3) | +# | APPLE_API_ISSUER_ID | the Issuer ID (UUID, Step 3) | +# | APPLE_TEAM_ID | your 10-char Team ID | +# +# Set the .p8 byte-for-byte (`gh secret set APPLE_API_KEY_P8 < AuthKey_X.p8`) +# — never through TextEdit; smart-quote substitution mangles PEM keys (the +# validation below diagnoses exactly that). NB electron-builder wants +# APPLE_API_KEY as a *file path* — release.yml's "Stage notarization API key" +# step writes the secret to disk and exports the path. +# +# Verify a published build on any Mac: +# spctl -a -vvv -t install SpyDE-*.dmg → "accepted, source=Notarized Developer ID" +# codesign -dv --verbose=4 /Applications/SpyDE.app → Authority=Developer ID Application +# stapler validate SpyDE-*.dmg → "The validate action worked!" +# Check the .zip too — it is the auto-update payload and must be notarized, or +# updates re-trigger Gatekeeper. name: Verify signing secrets on: diff --git a/electron/src/main/runner.ts b/electron/src/main/runner.ts index fcccf2b2..9b53c4ab 100644 --- a/electron/src/main/runner.ts +++ b/electron/src/main/runner.ts @@ -180,6 +180,14 @@ export function stopSpyDE(): void { } stopping = true proc = null // every sendAction() after this no-ops; prevents re-entrant kills + // KNOWN BUG (open): quitting while a find-vectors batch is still streaming can + // WEDGE shutdown on Windows. Clearing the tick timer here stops the stdin tick + // that keeps the hidden backend scheduled (the very starvation the tick was + // added for), so a mid-batch backend may never get scheduled long enough to + // process the graceful `quit` — only the 1.5 s taskkill backstop ends it. The + // e2e specs work around it by waiting for the '[fv-batch] finalized' log line + // before closing the app. The app-side fix (keep ticking until the backend + // exits, or force-reap the batch) is still open. if (tickTimer) { clearInterval(tickTimer); tickTimer = null } // 1. Ask the backend to quit gracefully (clean Dask shutdown). diff --git a/pyproject.toml b/pyproject.toml index 41705407..93e3f713 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -63,7 +63,7 @@ dependencies = [ # Remote model registry for the SpotUNet disk detector: pulls upgraded # weights + registry.json from Hugging Face (spyde/models/registry.py). # A hard dep — without it the documented ship-a-model-without-re-releasing - # path (spyde/models/RELEASING.md) silently no-ops. + # path (spyde/models/registry.py module docstring) silently no-ops. "huggingface_hub", ] description = "DE Visualization tool based on HyperSpy" diff --git a/spyde/actions/find_vectors_action.py b/spyde/actions/find_vectors_action.py index e6402bfa..6901c792 100644 --- a/spyde/actions/find_vectors_action.py +++ b/spyde/actions/find_vectors_action.py @@ -1128,8 +1128,8 @@ def fv_models(session, plot, payload) -> None: def fv_refresh_models(session, plot, payload) -> None: """'Check for new models': pull the latest ``registry.json`` from Hugging - Face (the ship-a-model-without-re-releasing path, ``models/RELEASING.md``) - on a worker thread — never the main loop (network) — then re-emit + Face (the ship-a-model-without-re-releasing path — the author-side contract + is in ``spyde/models/registry.py``'s module docstring) on a worker thread — never the main loop (network) — then re-emit ``fv_models`` with ``refreshed: true`` so the wizard dropdown updates in place. Offline-safe: a failed refresh keeps the current merged manifest.""" window_id = (payload or {}).get("window_id", getattr(plot, "window_id", None)) diff --git a/spyde/models/__init__.py b/spyde/models/__init__.py index 407e3c93..1e0de4ab 100644 --- a/spyde/models/__init__.py +++ b/spyde/models/__init__.py @@ -4,7 +4,7 @@ is a self-contained copy of the ``yoloDiffraction`` research project so SpyDE ships and runs without it. ``registry`` resolves which checkpoint to load — a bundled default now, with additional/upgraded models registered (and Hugging-Face-hosted) -later, see ``RELEASING.md``. +later; the ship-a-revised-model contract is in ``registry``'s module docstring. Typical use from the find-vectors detector: diff --git a/spyde/models/registry.py b/spyde/models/registry.py index aae9311f..34765649 100644 --- a/spyde/models/registry.py +++ b/spyde/models/registry.py @@ -23,7 +23,30 @@ - ``refresh_remote_registry()`` → pull the latest manifest from Hugging Face into the user dir (the "check for new models" path). Optional/lazy/offline-safe. -See ``RELEASING.md`` for the author-side workflow of shipping a revised model. +Shipping a revised model (the author-side contract): + +1. Train / iterate in the ``yoloDiffraction`` repo (the checkpoint stores its own + ``base`` / ``in_ch`` / ``levels`` hyperparams). +2. Validate it beats the current default on the real-scale benchmark: + ``python -m spyde.tests.benchmark_neural_spots``. +3. Upload the ``.pt`` to the HF repo (``HF_REPO``) under a NEW versioned + filename (e.g. ``spotunet-base16-v2.pt``) — NEVER overwrite an existing + file: a model ``id`` and its weights are immutable once published, so a + cached/downloaded ``.pt`` is never silently swapped under a user. +4. Add a versioned entry (``id``/``label``/``version``/``arch``/``sha256``/ + ``source``) to the repo's ``registry.json`` and, once accepted as best, set + ``default`` to it. Include the file's ``sha256`` — it is verified after + download and a mismatch falls back to the bundled default. Checkpoints are + loaded with ``torch.load(weights_only=True)``: plain state dicts + scalar + hyperparams only, never pickled objects. + +Users pick the new model up via Find Vectors → Model dropdown → refresh +(``fv_refresh_models`` → ``refresh_remote_registry()``); no SpyDE release +needed. To make a proven model the offline/first-run default for a SpyDE +release instead: copy the ``.pt`` into ``spyde/models/weights/``, add a +``{"type": "bundled", "file": …}`` entry to the bundled ``registry.json`` and +bump its ``default`` (``pyproject.toml`` package-data already globs +``models/weights/*.pt``). """ from __future__ import annotations From 31ded2d19000a7ea26270bfef2b8b8365e36b7ca Mon Sep 17 00:00:00 2001 From: Carter Francis Date: Fri, 7 Aug 2026 11:36:54 -0500 Subject: [PATCH 2/3] docs: delete the standalone .md docs (34 files) Everything that still mattered was pinned into code/workflow comments in the previous commit (signing walkthrough, release manual checks, model release contract, shutdown-wedge bug). The plan/audit/roadmap/benchmark narrative files were stale working documents; per project policy facts live in tests, docstrings and commit messages, not committed .md files. Kept: CLAUDE.md and every README.md (root, spyde/actions, electron/tests). --- AUDIT_defensive_code.md | 171 -- CLEANUP.md | 384 --- DIFFRACTION_VECTORS_PLAN.md | 1432 ---------- DISTRIBUTION_PLAN.md | 169 -- NEURAL_INTEGRATION_PLAN.md | 256 -- NOTEBOOK_PARITY_PLAN.md | 433 --- ORIENTATION_MAPPING_PLAN.md | 173 -- RELEASE_0_3_0_PLAN.md | 310 --- RELEASING.md | 177 -- ROADMAP.md | 297 -- TODO.md | 30 - VECTOR_ORIENTATION_MAPPING_PLAN.md | 686 ----- WALKTHROUGH_PRESENTATION_FIXES.md | 52 - benchmark_3nm_spots_results.md | 36 - benchmarks.md | 1828 ------------- docs/future_tasks.md | 16 - .../plans/2026-05-29-virtual-image.md | 1427 ---------- ...-bug-fixes-close-visibility-placeholder.md | 419 --- .../plans/2026-05-31-line-profile.md | 1875 ------------- ...026-05-31-plotwindow-focus-organization.md | 804 ------ .../2026-05-31-session-scoped-test-fixture.md | 350 --- .../2026-06-01-mainwindow-decomposition.md | 2413 ----------------- .../plans/2026-06-02-orientation-mapping.md | 1110 -------- .../specs/2026-05-28-console-design.md | 175 -- .../specs/2026-05-29-virtual-image-design.md | 432 --- ...-bug-fixes-close-visibility-placeholder.md | 60 - .../specs/2026-05-31-line-profile-design.md | 529 ---- ...31-plotwindow-focus-organization-design.md | 104 - ...5-31-session-scoped-test-fixture-design.md | 85 - ...6-06-01-mainwindow-decomposition-design.md | 304 --- .../2026-06-02-orientation-mapping-design.md | 180 -- electron/PACKAGING.md | 89 - electron/SIGNING.md | 231 -- spyde/models/RELEASING.md | 72 - 34 files changed, 17109 deletions(-) delete mode 100644 AUDIT_defensive_code.md delete mode 100644 CLEANUP.md delete mode 100644 DIFFRACTION_VECTORS_PLAN.md delete mode 100644 DISTRIBUTION_PLAN.md delete mode 100644 NEURAL_INTEGRATION_PLAN.md delete mode 100644 NOTEBOOK_PARITY_PLAN.md delete mode 100644 ORIENTATION_MAPPING_PLAN.md delete mode 100644 RELEASE_0_3_0_PLAN.md delete mode 100644 RELEASING.md delete mode 100644 ROADMAP.md delete mode 100644 TODO.md delete mode 100644 VECTOR_ORIENTATION_MAPPING_PLAN.md delete mode 100644 WALKTHROUGH_PRESENTATION_FIXES.md delete mode 100644 benchmark_3nm_spots_results.md delete mode 100644 benchmarks.md delete mode 100644 docs/future_tasks.md delete mode 100644 docs/superpowers/plans/2026-05-29-virtual-image.md delete mode 100644 docs/superpowers/plans/2026-05-31-bug-fixes-close-visibility-placeholder.md delete mode 100644 docs/superpowers/plans/2026-05-31-line-profile.md delete mode 100644 docs/superpowers/plans/2026-05-31-plotwindow-focus-organization.md delete mode 100644 docs/superpowers/plans/2026-05-31-session-scoped-test-fixture.md delete mode 100644 docs/superpowers/plans/2026-06-01-mainwindow-decomposition.md delete mode 100644 docs/superpowers/plans/2026-06-02-orientation-mapping.md delete mode 100644 docs/superpowers/specs/2026-05-28-console-design.md delete mode 100644 docs/superpowers/specs/2026-05-29-virtual-image-design.md delete mode 100644 docs/superpowers/specs/2026-05-31-bug-fixes-close-visibility-placeholder.md delete mode 100644 docs/superpowers/specs/2026-05-31-line-profile-design.md delete mode 100644 docs/superpowers/specs/2026-05-31-plotwindow-focus-organization-design.md delete mode 100644 docs/superpowers/specs/2026-05-31-session-scoped-test-fixture-design.md delete mode 100644 docs/superpowers/specs/2026-06-01-mainwindow-decomposition-design.md delete mode 100644 docs/superpowers/specs/2026-06-02-orientation-mapping-design.md delete mode 100644 electron/PACKAGING.md delete mode 100644 electron/SIGNING.md delete mode 100644 spyde/models/RELEASING.md diff --git a/AUDIT_defensive_code.md b/AUDIT_defensive_code.md deleted file mode 100644 index 61770502..00000000 --- a/AUDIT_defensive_code.md +++ /dev/null @@ -1,171 +0,0 @@ -# SpyDE backend audit — bug-hiding except / dead legacy / over-defensive getattr - -**Scope:** `spyde/` Python backend (not tests, not electron, not anyplotlib), extra depth on `spyde/actions/report/*.py`. **Read-only** — no edits applied. Every CONFIRMED finding below was re-verified against source after the auditors reported. - -**Confidence tags:** `CONFIRMED` = I traced it to source and the smell holds. `SUSPECTED` = real but lower blast-radius or more defensible. - ---- - -## TL;DR — what's actually worth fixing - -The backend is in good shape on the two things you were most worried about. There is **no dual-format serialization, no `_migrate_*` shims, no old-vs-new parse branch** anywhere — the report format is a single never-yet-shipped `SCHEMA_VERSION = 1`. And the report's *top-level* save/open/export all fail loud (`emit_error`). The genuine issues cluster into three buckets: - -1. **Two real data-loss swallows in the report load/save path** (P1) — a malformed field silently voids a whole figure spec on open; a bake failure silently drops a figure's pixels from a "successful" save. Both are `except Exception` + `log.debug`/silent in a path where the failure means corruption. -2. **~25 `getattr(cell/spec/panel/doc, 'field', default)` on guaranteed dataclass fields** (P2) — redundant defensiveness that would mask a "this isn't a Cell" bug instead of raising. Directly in line with the split-cell design goal: let a wrong object fail loud. -3. **A pile of dead Qt-migration vestiges** (P3, mechanical) — no-op stub methods/classes/aliases from the pyqtgraph era, zero callers. Safe deletions, mostly in `spyde/drawing/selectors/`. - -Broad `except Exception` scope is **mostly correct-by-design** here (worker boundaries, best-effort emits, anyplotlib build). Only a handful have a clearly-narrower intended error type. - ---- - -## P1 — Bug-hiding swallows in the report data path (fix these) - -### 1. CONFIRMED — one malformed field silently voids a whole figure spec on open -`spyde/actions/report/model.py:1547-1551` and `:1576-1579` (both in `read_report`) -```python -try: - c.spec = FigureSpec.from_yaml(zf.read(spec_name).decode("utf-8")) -except Exception: - c.spec = None -``` -`FigureSpec.from_yaml → from_dict → PanelSpec.from_dict → LayerSpec.from_dict`, and **none of the inner parsers has a try/except**. `LayerSpec.from_dict` does `float(d.get("alpha", 1.0))` / `float(linewidth)`; `PanelSpec.from_dict` does `list(...)`/`float(...)`. So one bad scalar in a `figures/.yaml` (a hand-edit, a truncated write, `alpha: "--"`) raises `ValueError`/`TypeError` — and this catch throws away the **entire** `FigureSpec`, not just the bad field. - -**What it hides:** the loud `emit_error` in `report_open` (handlers.py:1832) is never reached — the exception was already eaten *inside* `read_report`. The cell is pushed onto `_offline` (handlers.py:1861 skips it since `c.spec is None`), the user sees the baked PNG with **no indication their spec was corrupted**, and edit/refresh/re-slice are silently dead for that figure forever. On the next save the spec is gone from disk. If there's no baked PNG either, the figure is fully lost — with not even a log line (this is a silent `pass`-equivalent: no `log` call at all). - -**Fix:** (a) narrow to `except (yaml.YAMLError, ValueError, TypeError, KeyError)` so a real bug in `from_dict` (a typo'd attr → `AttributeError`) surfaces instead of masquerading as "corrupt file"; (b) at minimum `log.warning` with the cell id + exception; (c) ideally mark the cell "spec-parse-failed" and surface a per-cell warning so the user knows that figure lost its editability. - -### 2. CONFIRMED — a bake failure silently drops a figure's pixels from a "successful" save -`spyde/actions/report/handlers.py:797-803` (`assemble_assets`) -```python -if arr is not None and c.spec is not None: - try: - png = _bake_primary_snapshot(c, arr, max_edge=1200) - except Exception as e: - log.debug("asset bake failed for cell %s: %s", c.id, e) -if not png: - png = self._baked.get(c.id) -``` -On save/export, when the renderer harvest returned no PNG (headless save, slow/absent renderer reply past the 3 s timeout, unmounted window), this bakes the held snapshot. If `_bake_primary_snapshot` raises (bad dtype/shape, an mpl/agg error, a missing colormap the spec references) it's swallowed to `log.debug`; if `_baked` also has nothing (a freshly-added figure never harvested), `png` stays falsy and the cell is **omitted from `assets`** — while `write_report` still writes an `assets/.png` image ref into `report.md`. Result: a **dangling image ref** on next open, the figure's pixels silently lost, and the save reports success (`report_saved`). - -The code even comments the dangling-ref hazard for the empty-harvest case (handlers.py:787-790), but the bake-failure branch re-opens it. - -**Fix:** keep the fallback chain, but if the final `png` is still empty for a non-scene3d figure cell, collect those cell ids and `emit_error`/warn ("Report saved, but N figures could not be rendered and were omitted") instead of silently writing dangling refs. At minimum raise the bake failure from `log.debug` to `log.warning`. - -### 3. SUSPECTED — `count_map()` failure → blank-black navigator in the embedded vectors explorer -`spyde/actions/report/vectors_embed.py:214-218` — `except Exception → cm = np.zeros(...)`. If the CSR count-map aggregation raises, the embedded explorer renders a blank navigator the user may read as "no vectors here" rather than "the count map crashed." Degraded-not-corrupt. **Fix:** narrow the except, `log.warning`, and ideally mark "count map unavailable." Low priority. - -### 4. SUSPECTED — a 3-D IPF panel silently vanishes on render failure -`spyde/actions/report/figure_builder.py:334-337` — `except Exception → return None`, and the caller skips a None panel. scene3d is genuinely fragile (Agg can't render 3-D, hence the separate path), so this is a defensible best-effort, but a panel the user asked for disappears invisibly. **Fix:** keep the fallback, raise the log to `warning`, ideally draw a "3-D panel unavailable" placeholder tile. Borderline. - -### 5. SUSPECTED (minor) — example-data calibration silently wrong -`spyde/backend/_session_files.py:92-97` — applying the hardcoded scale/offset for a built-in example dataset under `except Exception → log.debug`. Scoped to example data (known dict, axes known to exist), so a failure here is really a regression being hidden. `except Exception` is too broad for a should-always-succeed op. Same file: the metadata/axis carry-over swallows at `:441, :562, :570` silently degrade calibration on stacked/reshaped derived signals (`log.debug` best-effort, primary data still loads — lower severity). **Fix:** narrow or let it propagate in dev. - ---- - -## P2 — Over-defensive `getattr`/defaults on GUARANTEED dataclass fields - -Every field of `Cell`, `ReportDoc`, `FigureSpec`, `PanelSpec`, `LayerSpec` is **default-valued** (model.py:711-724 Cell, 736-742 ReportDoc, 520-531 PanelSpec, 590-594 FigureSpec) → **always present on a real instance**. So `getattr(cell, 'field', default)` can only fire its default if `cell` isn't the dataclass it's typed as (None / a dict / wrong type) — i.e. it masks a contract violation. The tell throughout: the *same block* reads `c.id`, `c.spec`, `c.cell_type`, `c.source` as plain attributes but wraps sibling fields in `getattr`. Recommended fix everywhere: **drop the default, access the attribute directly**, so a wrong object fails loud (exactly the split-cell design goal). - -### CONFIRMED — report state / serialization (a wrong object here corrupts the document) - -| # | Site | Code | Fix | -|---|---|---|---| -| 1 | `handlers.py:302-310` | `getattr(c, "slide_break"/"live_action"/"slide_kind"/"slide_style"/"notes", …)` in `state()` | `c.slide_break` etc. | -| 2 | `handlers.py:319, 323, 330` | `getattr(c, "split_layout", "text-left")`, `getattr(c, "image_ext", "")` | `c.split_layout`, `c.image_ext` | -| 3 | `handlers.py:372, 2024, 2040` | `getattr(self.doc/mgr.doc, "doc_type", "report")`, `getattr(c, "split_layout", …)` in template save | direct | -| 4 | `handlers.py:2355` | `getattr(cell, "slide_break", False)` in `report_toggle_slide_break` (cell already non-None) | `not cell.slide_break` | -| 5 | `model.py:942-965, 873` | `getattr(c, …)` for `slide_break/slide_kind/slide_style/notes/live_action/split_layout` in `serialize_report_md` + `move_slide` | direct (serialization twins of #1/#2) | -| 6 | `model.py:804, 823-825, 834` | `getattr(c, "cell_type", "")`, `getattr(first, "slide_kind"/"slide_style"/"notes", "")` in `slide_columns`/`slide_meta`/`slide_notes` | keep the `first is None` guard, drop the attr-absent default: `first.slide_kind if first else ""` | -| 7 | `export_html.py:174, 239, 287` | `getattr(cell, "image_ext")`, `getattr(cell.spec, "vectors_mode")` (spec already non-None), `getattr(cell, "split_layout")` | direct | -| 8 | `handlers.py:425` | `getattr(spec, "vectors_mode", "")` (spec non-None, guarded above) | `spec.vectors_mode` | - -### CONFIRMED — `PanelSpec.kind` read via `getattr` (redundant even inside `str(...)`) -`kind` is a guaranteed `PanelSpec` field (default `"image"`). The codebase uses `str(panel.kind)` in most places and `str(getattr(panel, "kind", ""))` in these — the getattr form is the smell: -- `handlers.py:68, 80` (`_is_scene3d_panel`/`_is_line_panel`), `figure_builder.py:454, 456`, `export_html.py:777` (in `report_paste_cell` — `spec` was *just built* by `FigureSpec.from_dict` two lines up, so `kind` is guaranteed). → `str(panel.kind)`. - -### SUSPECTED — `_is_*`/`_cell` gate helpers -`handlers.py:63-64` (`_is_figure_like`), `:75` (`_is_scene3d_cell`), `compose.py:84-85` (`_cell`), `overlay_embed.py:100`, `vectors_embed.py:917-918` — `getattr(cell, "cell_type"/"spec", …)`. The `cell is None` guard is legitimate; the attr-absent default is not (in `_is_figure_like` the None check is on the line above, so `getattr(cell, "cell_type", "")` is pure redundancy). Fix: after the None check, use `cell.cell_type`/`cell.spec`. - -**NOT flagged (correct):** ~450 other `getattr` across `spyde/` read genuinely external/dynamic shapes — hyperspy axes (`scale/offset/units/size`), dask internals (`_lazy`, `chunksize`, `client`), set-late plot/session/tree attrs (`_plot2d`, `plot_state`, `_om_wizard`), torch/cupy device probes. Those are correct defensive reads. Also correct: all the `*.from_dict` `d.get(key, default)` — those parse external YAML with genuinely-optional fields. - ---- - -## P3 — Broad `except Exception` where narrow is intended - -The report area's `except Exception` is **overwhelmingly correct-by-design** (event-handler wiring that must never crash compute, anyplotlib build/emit best-effort, snapshot harvest, save/export top-level — all log). Only these have a clearly-narrower intended error: - -| Site | Code | Narrow to | Hides | -|---|---|---|---| -| `handlers.py:2524-2527` | `count = int(len(vecs.flat_buffer))` → `except Exception: count = 0` | `(TypeError, AttributeError)` or drop | a renamed/broken `flat_buffer` silently reports `count=0` forever | -| `handlers.py:1435-1438, 1577-1580` | `clim = [float(lv[0]), float(lv[1])]` → `except Exception: clim = None` | `(TypeError, ValueError, IndexError)` (matches `compose.py:1048` which does it right) | an unrelated error becomes "no clim" | -| `handlers.py:1338-1349, 1376-1386` | per-field `str()/float()` of renderer line-style state → `except Exception: pass` | `(TypeError, ValueError)` | an `AttributeError` from a mistyped dict access | -| `model.py:1188-1190` | `yaml.safe_load(m_live.group("payload"))` → `except Exception` | `yaml.YAMLError` | a regex-group `AttributeError` (low sev — external marker text) | - -**NOT flagged (correct broad catches):** `_wire_*`/`_make_*_handler` bodies, `build_cell_figure`, all `figure_builder.py` anyplotlib calls, `export_html` interactive→static degradation, report open/save/export top-level (all `emit_error`), the atomic `write_report` cleanup (`model.py:1420` — swallows only the cleanup OSError and **re-raises** the real error, textbook-correct), and every compute/dask/worker/GPU boundary CLAUDE.md mandates. - ---- - -## P4 — Dead legacy / Qt vestiges (mechanical, safe deletions) - -No serialization-format legacy exists. What's dead is pyqtgraph/QWidget scaffolding left from the Qt→Electron migration — no-op stubs with **zero callers** (all grep-verified repo-wide). None touches a format, so none can reintroduce a split-cell-style edge bug. - -**Group A — the `spyde/drawing/selectors/` dead subsystem (one sweep):** -- `selectors/selection_selector2d.py` — **entire file** dead (`SelectionSelector2D`, "kept for import compatibility", zero importers). Delete the file. -- `selectors/utils.py:32-49` — `no_return_update_function`, `create_linked_rect_roi`, `create_linked_linear_region`, `create_linked_infinite_line` (pyqtgraph linked-ROI stubs, zero callers). Keep `broadcast_rows_cartesian` (live). Delete the four + the misleading "pyqtgraph signal chaining" comment. -- `base_selector.py:124-128, 186` — `_StubWidget` + `self.widget = _StubWidget()` — nothing ever *reads* `selector.widget`. Delete both. -- `selector2d.py:602-603` — `IntegratingSSelector2D = IntegratingSSelector2D` (self-assignment no-op). Delete. -- `selector2d.py:469-470` + `selectors/__init__.py:4` — `LineSelector = LineProfileSelector` alias, no consumer uses the name. Delete + drop from `__init__` export. -- `add_linked_roi(self, plot): pass` stubs across `selector2d.py` (70,166,214,254,424,598), `selector1d.py` (81,179,317), `base_selector.py` (489) — zero call sites. Delete all. - -**Group B — Qt method-name no-op shims on Plot/PlotWindow:** -- `plot_window.py` — `set_graphics_layout_widget` (182), `_build_new_layout` (186), `setGraphicsEffect` (143), `raise_` (118), `lower` (122), `setGeometry` (136), `previous_subplots_pos`/`previous_subplot_added` (72-73), and the `x/y/width/height` lambda properties (189-204) — QMdiSubWindow/QWidget shims, no callers. (Keep `show`/`hide`/`isVisible`/`move`/`resize`/`close` — those have real Electron-emit bodies.) -- `plot.py:1222-1235` — `addItem`, `removeItem`, `normalize_axes`, `update_range` (all `pass`, pyqtgraph PlotItem names, zero callers). Delete + the section header. -- *Risk note:* CONFIRMED-dead within the Python backend; these are Python object methods, not IPC verbs, so the electron boundary is unaffected — but a 10-second grep of the electron side for these exact names before deleting is cheap insurance. - -**Group C — the one true dead compat branch in the report/overlay path:** -- `spyde/actions/overlay.py:413-417` — `_apply_pending_layer_frames` tolerates a "legacy (handle, frame) 2-tuple", but the only writer (`_enqueue_layer_push`) always builds a 3-tuple `(layer, layer.handle, frame)`. The `else` is unreachable. Collapse to `layer, handle, frame = entry`. - -**Group D — low-value dead kwargs:** -- `multiplot_manager.py:48-52` — `main_window=None` "legacy" kwarg; the sole construction site passes `session=`. Drop the param + `or main_window`. (The `main_window=` *parameter name* on the compute functions is a live `Session` carrier — a rename, not dead code; leave it.) -- `update_functions.py:926, 943-947` — `cache_in_shared_memory` DEPRECATED no-op kwarg; only a benchmark passes it. Drop once you confirm no electron caller sets it. - -**Housekeeping (not code):** -- `mdi_manager.py:4` references `_qt_main_legacy.py ("Phase 4 reference")` — **that file doesn't exist**. Stale comment; drop the reference. (`MDIManager` itself is fully live.) -- No Qt imports remain anywhere in `spyde/` — clean. Only migration *comments* reference it. - ---- - -## Explicitly NOT dead / NOT a smell (traced — don't waste time removing) - -The report code's docstrings say "back-compat"/"legacy" in several places that are actually **correct optional-field tolerance within the single v1 schema**, or live fallbacks still triggered by current inputs: -- The whole `model.py` "SCHEMA_VERSION stays 1 / older files → default" family (`LayerSpec.tint/color/linewidth`, `PanelSpec.scene/text_sizes`, `FigureSpec.vectors_mode`, `Cell.slide_break/…/split_layout`, `ReportDoc.doc_type`) — features added *within* v1 with emit-when-set + `.get()`. No prior format ever shipped. **Correct new-optional-field design** — the opposite of the split-cell bug. -- `read_report` `.png`-without-yaml → image promotion — the current, intentional figure-vs-photo disambiguation. Both are live cell types. -- `_normalize_doc_type` "movie" — reserved-*forward* value, forward-compat not backward. -- `compose.py` "legacy edge-of-grid placement" (289,401,427,443) — live fallback when the renderer omits `target_panel_id`. -- `export_html.py` "backward compatible" token-omission (591,643,693) — optional field in the *live* backend↔renderer IPC (they ship together). -- `session.py:447`, `playback.py DEFAULT_FPS`, `live_overlay.py "sync"`, `_session_files.py _EXAMPLE_CALIBRATION`, `spotunet-base16-v1` weights — all live; "legacy" there is lineage-describing. -- `diffraction_vectors.py offsets` "legacy single-level" (226,253,347,748,1014) — a redundant alias of `nav_offsets[-1]` that's still *produced and consumed*; a de-dup *simplification* opportunity, not dead code. - ---- - -## Suggested order - -1. **P1 #1 + #2** — the two data-loss swallows. Highest stakes, smallest diff. Narrow the except + escalate to a user-visible warning. -2. **P2** — the ~25 guaranteed-field `getattr` sites. Mechanical, and it makes wrong-object bugs fail loud (your stated split-cell goal). One focused pass over `handlers.py state()/serialize`, `model.py serialize/slide_*`, `export_html.py`, `figure_builder.py`, `compose.py`. -3. **P4 Group A** — the selectors dead-subsystem sweep. Pure deletion, removes the most "looks like it still does something" misdirection. -4. **P3** + **P4 B/C/D** — narrow the four excepts; delete the remaining Qt shims + the overlay 2-tuple branch + the two dead kwargs. - ---- - -## Execution outcome (all four applied) - -All four buckets were applied, with per-site verification. Two auditor claims were **corrected during execution** by re-checking callers myself: - -- **`_StubWidget` / `self.widget` in `base_selector.py` is NOT dead — kept.** The auditor grepped only within the selectors package and concluded nothing reads `selector.widget`. It missed `plot.py:1125` `sel.widget.hide()`, a live call during node-switching. `_StubWidget` provides that `.hide()` no-op; removing it would raise. Left in place. -- **`raise_` / `lower` / `setGeometry` / `setGraphicsEffect` on `PlotWindow` are functional IPC emitters, not no-op shims.** The auditor listed them among dead no-op shims; they actually emit `window_raise`/`window_lower`/etc. They have no current callers but are a real API surface, so I left them (deleting uncalled-but-working emitters is a judgment call, not "dead code"). Only the genuine `pass`-body shims (`set_graphics_layout_widget`, `_build_new_layout`, the `x/y/width/height` fake-geometry properties, `previous_subplot*` attrs) were removed. - -Everything else was applied as reported: -- **P1:** narrowed both `read_report` excepts (+ new `Cell.spec_error`, user warning on open), narrowed the `assemble_assets` bake except (+ `_dropped_assets` tracking, user warning on save), and the three SUSPECTED (count_map / scene3d / example-calibration) narrowed + bumped to `log.warning`. 7 new regression tests. -- **P2:** all ~25 guaranteed-field `getattr` sites → direct attribute access across `handlers.py`, `model.py`, `export_html.py`, `figure_builder.py`, `compose.py`, `overlay_embed.py`, `vectors_embed.py`. -- **P3:** narrowed the `flat_buffer` count, both `clim` conversions, the six line-style per-field conversions, and the `yaml.safe_load` marker parse. -- **P4:** deleted `selection_selector2d.py`, the 4 `create_linked_*`/`no_return_update_function` stubs, all 9 `add_linked_roi` stubs, the `LineSelector`/`IntegratingSSelector2D` aliases, the `Plot` axis no-op shims, the `PlotWindow` no-op shims, the overlay 2-tuple dead branch, the `main_window` + `cache_in_shared_memory` dead kwargs, and the stale `_qt_main_legacy.py` doc reference. diff --git a/CLEANUP.md b/CLEANUP.md deleted file mode 100644 index 7f843545..00000000 --- a/CLEANUP.md +++ /dev/null @@ -1,384 +0,0 @@ -# SpyDE cleanup runbook — fully remove Qt + kill error-hiding `except`s - -A **mechanical, verifiable** plan to (1) delete the now-dead Qt/pyqtgraph layer, -(2) drop the Qt dependencies, and (3) eliminate the `except: pass` anti-pattern. -Each step has an exact command, a verification gate, and a rollback. Work in -**small commits** (one logical step each) so any regression is caught immediately -and `git revert`/`git reset` is trivial. - -> Status legend used below: ✅ proven this session · ⚠️ verify before acting. - ---- - -## 0. Why this is safe (proven facts) - -The Electron app's Python backend is launched as `uv run python -m spyde` -(`electron/src/main/runner.ts`) → `spyde/__main__.py:main()` → `spyde.backend.app.run`. - -These were **measured** (not assumed) this session: - -- ✅ Importing `spyde.backend.session` + `spyde.backend.app` pulls **zero** - `PySide6` / `pyqtgraph` modules. -- ✅ Importing the **entire live action + figure surface** (all 7 staged-handler - modules + the IPF/figure path + `spyde.drawing.plots.plot`) pulls **zero** Qt. -- ✅ The only real importers of `spyde.actions.pyxem` are `line_profile.py` and - `vector_orientation_action.py` — both themselves dead. Nothing live imports it. -- ✅ `spyde/backend/app.py` and `spyde/backend/session.py` mention Qt only in - **comments/docstrings** (no import). `vector_orientation_gpu.py` likewise - (a comment about `QApplication.processEvents`). Leave these as-is. - -**Conclusion:** because the live surface is Qt-free, *no live module imports any -Qt module*. Therefore every file that imports `PySide6`/`pyqtgraph` is dead with -respect to the Electron app — **with one exception**: `find_vectors.py` is live -but lazy-imports Qt inside a few legacy widget functions (trim those, keep the -module). See §4. - -The 7 live staged-handler modules (from `spyde/backend/session.py`): -`center_zero_beam`, `composition`, `find_vectors_action`, `ipf_view`, -`orientation_action`, `vector_orientation_om`, `views`. - ---- - -## 1. Safety harness — run after EVERY step - -Define these three invariants. A step is only "done" when all three pass. - -**INV-1 — live surface stays Qt-free** (catches an accidental live coupling): - -```bash -uv run python - <<'PY' -import sys, importlib -for m in ['spyde.backend.app','spyde.backend.session','spyde.actions.center_zero_beam', - 'spyde.actions.composition','spyde.actions.find_vectors_action','spyde.actions.ipf_view', - 'spyde.actions.orientation_action','spyde.actions.vector_orientation_om', - 'spyde.actions.views','spyde.actions.ipf_density','spyde.actions.ipf_refine', - 'spyde.actions.ipf_refine_render','spyde.actions.orientation_compute', - 'spyde.actions.find_vectors','spyde.actions.vector_overlay', - 'spyde.actions.find_vectors_torch','spyde.drawing.plots.plot']: - importlib.import_module(m) -qt=[m for m in sys.modules if 'PySide6' in m or m=='pyqtgraph' or m.startswith('pyqtgraph.')] -assert not qt, f'LIVE SURFACE PULLED QT: {qt[:6]}' -print('INV-1 OK: live surface Qt-free') -PY -``` - -> Note: the harness intentionally uses a heredoc with `uv run python`. If a -> tooling hook blocks heredocs, paste the body into `scripts/_inv1.py` and run -> `uv run python scripts/_inv1.py` instead. - -**INV-2 — the Qt-free test suite is green** (THE liveness oracle — if a deletion -broke something live, a migrated test goes red): - -```bash -uv run pytest spyde/tests/migrated/ -q -p no:cacheprovider # ~2–3 min, must be all-pass -``` - -> ⚠️ Live actions have **two** entry points, not one — trust INV-2 over static -> grep: (a) the 7 string handlers in `session.py`; (b) the **YAML-wired actions** -> in `spyde/toolbars.yaml`, `spyde/actions/hyper_signal_actions/*.yaml`, -> `spyde/actions/plot_actions/*.yaml` (modules: `base`, `center_zero_beam`, -> `fft_action`, `find_vectors_action`, `line_profile_action`, `orientation_action`, -> `vector_orientation_om`, `vector_virtual_imaging`, `virtual_image`). A module -> reachable from either — including via a **lazy** import inside an action that -> only fires at runtime — is LIVE even though INV-1 (import-only) shows it Qt-free. -> This is how `line_profile` was caught. - -**INV-3 — the backend boots** (import the entry, don't spawn a UI): - -```bash -uv run python -c "import spyde.__main__; import spyde.backend.app; print('INV-3 OK')" -``` - -**E2E smoke (run at phase boundaries, not every step — it's heavy, one at a time):** - -```bash -cd electron && npm run build && npx playwright test tests/orientation_lazy.spec.ts \ - tests/spyde.spec.ts --workers=1 --reporter=line -``` - -**Baseline:** before starting, run INV-1/2/3 + the E2E smoke and record that they -pass. If the baseline isn't green, fix that first — do not start deleting. - -**Branch:** `git checkout -b chore/remove-qt` off `main`. One step = one commit. - ---- - -## 2. The KEEP set (never delete) - -- `spyde/backend/**`, `spyde/signals/**`, `spyde/workers/**` -- `spyde/__init__.py`, `spyde/__main__.py` -- `spyde/actions/**` **except** the dead Qt actions in §3 (and trim §4) -- `spyde/drawing/plots/plot.py`, `spyde/drawing/selectors/**`, - `spyde/drawing/update_functions.py`, `spyde/drawing/colormaps.py`, - `spyde/drawing/__init__.py` -- `spyde/*.yaml`, `spyde/actions/**/*.yaml` -- `spyde/tests/migrated/**`, `spyde/conftest.py`, `spyde/qt/shared.py`'s - replacement helpers ⚠️ (see §5 — `shared.py` itself is Qt; check whether any - migrated test imports `open_window`/`create_data`/`wait_until` from it and, if - so, move those Qt-free helpers into `spyde/tests/migrated/_helpers.py` first). - -Everything in §3 is dead and goes. - ---- - -## 3. Delete the dead Qt packages (the bulk of the work) - -These all import `PySide6`/`pyqtgraph` and nothing live imports them. Delete in -the batches below; after **each batch** run INV-1, INV-2, INV-3. - -**Batch 3a — pure Qt UI packages (zero live refs):** - -```bash -git rm -r spyde/qt spyde/live spyde/misc spyde/external/pyqtgraph spyde/external/qt -``` -(If `spyde/external/` is now empty except `__init__.py`, remove it too.) - -**Batch 3b — the legacy Qt MainWindow + scratch + Qt-only top-level modules:** - -```bash -git rm spyde/_qt_main_legacy.py spyde/qt_scrapper.py spyde/_conftest_legacy.py \ - spyde/dock_manager.py -``` -> ✅ Verified: `dock_manager.py` is imported only by the two legacy files (dead). -> ⚠️ Do **NOT** delete `spyde/mdi_manager.py` or `spyde/metadata_extract.py` — -> both are **live and Qt-free**. `session.__init__` instantiates `MDIManager` -> (the Qt-free window abstraction; `PlotWindow` *replaces* `QMdiSubWindow`, it is -> not Qt), and `session` calls `metadata_extract.build_metadata_dict/_axes_list` -> for the dock. (`metadata_extract`'s only "PySide6" is a docstring; importing it -> pulls no Qt.) - -**Batch 3c — the Qt drawing layer (toolbars + presenter):** - -```bash -git rm spyde/drawing/toolbars/caret_group.py spyde/drawing/toolbars/toolbar.py \ - spyde/drawing/toolbars/popout_toolbar.py spyde/drawing/toolbars/stylized_toolbar.py \ - spyde/drawing/toolbars/floating_button_trees.py spyde/drawing/toolbars/utils.py -git rm spyde/drawing/signal_tree_presenter.py -``` -> ⚠️ **`spyde/drawing/toolbars/` is NOT all dead.** Do **NOT** `git rm -r` the -> whole dir — it also holds LIVE, Qt-free assets that the running app needs: -> `plot_control_toolbar.py` (`get_toolbar_config_for_plot` — builds the per-window -> toolbar action list emitted by `plot_states._send_toolbar_config`; without it -> EVERY toolbar action button vanishes), `__init__.py`, and `icons/*.svg` -> (referenced by `toolbars.yaml`). Delete only the six Qt modules above. (Both -> were over-deleted once and restored — neither headless nor the smoke E2Es catch -> it, because migrated tests don't render the toolbar and the smoke specs don't -> click toolbar action buttons. A `vector_*_lazy` / `strain_lazy` E2E that clicks -> an `action-btn-*` is the guard.) -⚠️ Then verify the rest of `spyde/drawing/plots/` is Qt-free and live: -`grep -rn "PySide6\|pyqtgraph\|QMdiSubWindow\|QWidget" spyde/drawing/plots/`. -If `plot_window.py` / `multiplot_manager.py` / `plot_states.py` import Qt **and** -no migrated test imports them, `git rm` them too; if a migrated test imports -them, they're live — keep and open a follow-up to de-Qt them. (INV-2 will tell -you: delete, run it, and if a test errors on import, revert that one file.) - -**Batch 3d — the dead Qt action modules:** - -```bash -git rm spyde/actions/vector_orientation_action.py # the dead pyqtgraph caret -``` -> ✅ Verified dead: `vector_orientation_action` is only named in comments; the -> live path is `vector_orientation_om.py`. -> ⚠️ **`pyxem.py`, `line_profile.py`, `line_profile_action.py` are NOT dead.** -> `line_profile_action` is wired in `spyde/toolbars.yaml` (a LIVE entry point — -> see §1) and lazy-imports the Qt `line_profile.py`, which imports -> `pyxem._start_progress_poll` + pyqtgraph + PySide6. Deleting them turns -> `test_template_actions::test_line_profile_opens_output_window` red. These are -> **Phase-6 de-Qt targets** (§4b), not deletions. - -After 3a–3d: run INV-1/2/3, then the **E2E smoke**. Commit each batch separately, -e.g. `git commit -m "chore(qt): remove dead Qt UI packages (qt/ live/ misc/ external/)"`. - -> Rollback for any batch: `git restore --staged --worktree ` (before -> commit) or `git revert ` (after). - ---- - -## 4. Trim the dead Qt tail of `find_vectors.py` (live module) - -`find_vectors.py` is **live** (the Find-Vectors compute) but its tail -(≈ lines 2780→EOF) is the legacy Qt overlay widget — it lazy-imports -`pyqtgraph` / `PySide6` inside functions, so it doesn't pull Qt at import. - -1. Identify the Qt functions: `grep -n "pyqtgraph\|PySide6\|QtCore\|CircleROI\|ScatterPlotItem\|ImageItem" spyde/actions/find_vectors.py`. -2. For each such function, confirm **no live caller**: - `grep -rn "" spyde --include=*.py | grep -v /tests/` (and check it - isn't referenced by a YAML action or a session handler). The Electron overlay - is `spyde/actions/vector_overlay.py` (Qt-free) — the Qt versions here are the - old pyqtgraph ones. -3. Delete those functions. Keep everything above the Qt tail (the compute). -4. Gate: `grep -c "pyqtgraph\|PySide6" spyde/actions/find_vectors.py` → **0**. - Run INV-1/2/3 + `uv run pytest spyde/tests/migrated/test_find_vectors_port.py -q`. - -Repeat the same liveness check for any other module flagged by -`grep -rln "PySide6\|pyqtgraph" spyde/actions --include=*.py` that turns out to be -live-with-lazy-Qt. - -### 4b. De-Qt the live line-profile action (`line_profile.py` + `pyxem.py`) - -`line_profile_action.py` (live, YAML-wired) lazy-imports `line_profile.py`, which -is a **pyqtgraph LineROI widget** that also pulls `pyxem._start_progress_poll` and -`spyde.qt.compute_status_indicator`. This is the last live action still on Qt. - -1. **Extract the Qt-free helper:** `pyxem.py` is ~3.2k lines of mostly-dead Qt - UI; the only thing live code needs from it is `_start_progress_poll`. Move that - (and any pure-compute helpers `line_profile.py` uses) into a Qt-free module - (e.g. `spyde/actions/_progress.py`), repoint the import. Confirm with - `uv run python -c "import sys,importlib; importlib.import_module('spyde.actions.line_profile_action'); ..."` - style INV-1. -2. **Port the line-profile UI to the action template** the way the other actions - were (Electron/anyplotlib `RegionAction`), so `line_profile.py` no longer needs - pyqtgraph/PySide6/`spyde.qt`. The migrated test - `test_template_actions::test_line_profile_opens_output_window` is the gate. -3. Once `grep -c "PySide6\|pyqtgraph\|spyde.qt" spyde/actions/line_profile.py` → 0 - and `find_vectors.py` is trimmed (§4), **`pyxem.py` is fully dead** — delete it - — and **`spyde/qt/` has no importer** — delete it: - ```bash - git rm spyde/actions/pyxem.py - git rm -r spyde/qt - ``` - Gate: INV-1/2/3 + E2E + `grep -rn "PySide6\|pyqtgraph" spyde --include=*.py | grep -v /tests/` → empty. - ---- - -## 5. Remove the legacy `pytest-qt` test suite - -`spyde/tests/` (root, ~28 files) tests the **old Qt app** (`qtbot`, the legacy -`MainWindow`, `from spyde.qt …`). The Qt-free suite is `spyde/tests/migrated/`. - -1. List them: `grep -rln "PySide6\|qtbot\|_qt_main_legacy\|from spyde.qt\|MainWindow\|_conftest_legacy" spyde/tests/*.py`. -2. ⚠️ **Before deleting**, rescue anything still referenced by migrated tests: - - `grep -rn "from spyde.qt.shared import\|import spyde.qt.shared" spyde/tests/migrated/` - — if migrated tests use `open_window`/`create_data`/`wait_until`, copy those - **Qt-free** helpers into `spyde/tests/migrated/_helpers.py` and repoint imports. - - Keep data fixtures used by migrated tests (e.g. `Silver__0011135.cif`, - `*.hspy` test inputs) — `grep -rn "Silver__0011135\|" spyde/tests/migrated/`. -3. `git rm` the legacy test files (and `spyde/tests/conftest*.py` that only - serves them). Run INV-2 (now only migrated runs) — must stay green. -4. `git rm` any now-orphaned legacy fixtures not referenced by migrated tests. - ---- - -## 6. Drop the Qt dependencies (the acceptance that Qt is gone) - -1. Edit `pyproject.toml`: remove `PySide6` and `pyqtgraph` from `dependencies` - (and any Qt entry in `[project.optional-dependencies]`/`[tool.*]`). Also remove - stale Qt packaging if unused: the root `spyde.spec` (PyInstaller) and any - `[tool.pycrucible]` block — confirm the Electron build (`npm run build`) is the - only shipping path first. -2. Recreate the environment **without** Qt to prove nothing needs it: - ```bash - uv sync --reinstall # or: uv lock && uv sync - uv pip list | grep -iE "pyside6|pyqtgraph" # must print NOTHING - ``` -3. Gate: INV-1/2/3 + full E2E. If anything imports Qt now, it will fail loudly — - that's the point. Fix by porting (rare) or deleting the offending import. -4. Final Qt grep — must be **empty**: - ```bash - grep -rn "PySide6\|import pyqtgraph\|from pyqtgraph" spyde --include=*.py | grep -v /tests/ - ``` - ---- - -## 7. Eliminate error-hiding `except`s - -> **STATUS: DONE (2026-06).** All silent `except … : pass` in live source are -> gone — **176 → 0** (AST finder below = 0). Converted module-by-module, one -> commit each, compute/data-path modules first and UI-glue last, exactly as the -> process prescribes. Every handler now either logs (`log.debug`, or -> `log.warning`/`log.exception` where a swallow would otherwise erase a -> user-visible failure — e.g. `write_shared_array`, the threaded navigator load), -> let-raises, or is a narrow membership guard (the `except ValueError: pass` on -> `list.remove` became `if x in lst`). Two latent bugs surfaced and were handled: -> -> - **`log` vs `logger` shadow** — `ipf_density.build_ipf_density_figure` takes a -> `log: bool` param that shadows a module-level `log`, so `log.debug(...)` would -> crash *only when the except fired*. Module loggers in plotting modules are now -> named `logger`. Guard added: `tools/scan_logger_shadow.py` (AST check, 0 hits). See -> the `logger-name-shadow` memory. -> - **anyplotlib `Axes.set_title` doesn't exist** — several multi-panel titles -> (`ipf_density`, `ipf_refine_render`, `views`) were wrapped in `except: pass` -> and so *silently never rendered*. Now logged; the real fix is to add -> `set_title` to anyplotlib (tracked in §9). - -228 of 457 `except` handlers (49%) are silent `except … : pass` — they hide real -failures. Fix systematically; **never** leave a bare swallow. - -**Find them (AST-accurate, prints file:line):** - -```bash -uv run python - <<'PY' -import ast, pathlib -for p in pathlib.Path('spyde').rglob('*.py'): - if '/tests/' in str(p): continue - try: t = ast.parse(p.read_text()) - except SyntaxError: continue - for n in ast.walk(t): - if isinstance(n, ast.ExceptHandler) and len(n.body)==1 and isinstance(n.body[0], ast.Pass): - typ = ast.unparse(n.type) if n.type else 'BARE' - print(f'{p}:{n.lineno}: except {typ}: pass') -PY -``` - -**Remediation rule (apply per occurrence):** - -1. **Never** `except:` or `except BaseException:` — narrow to the specific - exception(s) actually expected. -2. **Never** `except Exception: pass`. Choose one: - - *Genuinely optional / best-effort* (e.g. a cosmetic UI nicety, an optional - metadata field): keep going but **log it** — - `except SpecificError as e: log.debug("…: %s", e)` (module-level - `log = logging.getLogger(__name__)`). It must be traceable. - - *Could mask a real bug* (compute, data, anything in a hot/analysis path): - **let it raise** — delete the `try/except`, or re-raise after logging. - - *Control flow* (`except ImportError`, `except KeyError` with a default, - `except (FileNotFoundError, ...)`): keep, but narrowed and commented with - *why* it's safe to continue. -3. Add `import logging` + `log = logging.getLogger(__name__)` to any module that - gains a logged handler. - -**Process:** one module per commit (start with the live compute modules — -`orientation_compute.py`, `ipf_refine.py`, `find_vectors*.py`, `vector_*` — where -hidden errors are most dangerous; UI-glue modules last). After each module: -`uv run pytest spyde/tests/migrated/ -q` + INV-1. Re-run the finder; the count -must monotonically drop. Target: **0** silent `except: pass` in non-test code, -and `grep -rn "except:" spyde --include=*.py` (bare) = 0. - -> While here: the audit also found **142 bare `print()`** in non-test source -> (`grep -rn "^\s*print(" spyde --include=*.py | grep -v /tests/`). Convert to -> `logging` in the same per-module passes (optional but recommended). - ---- - -## 8. Acceptance criteria (the cleanup is "done" when ALL hold) - -- [x] `grep -rn "PySide6\|import pyqtgraph\|from pyqtgraph" spyde --include=*.py | grep -v /tests/` → only 2 *comments*, no imports *(done 2026-06)* -- [x] `uv pip list | grep -iE "pyside6|pyqtgraph"` → **empty** (deps removed) *(done 2026-06)* -- [x] AST finder (§7) → **0** silent `except: pass`; `grep -rn "except:" spyde --include=*.py` → **0** bare *(done 2026-06)* -- [x] INV-1, INV-2 (320 tests), INV-3 all green *(done 2026-06)* -- [~] E2E: migrated suite (320) + `app_log` spec green this round; re-run the full Playwright set (`orientation_lazy`/`spyde`/`ipf_*`/`composition`) before release -- [x] `spyde/tests/` contains only the migrated (Qt-free) suite — legacy retired *(done 2026-06)* -- [x] App still launches (Playwright launches the real backend each run) *(done 2026-06)* - ---- - -## 9. Deferred / related follow-ups (track separately, not blockers) - -- **Port the IPF-refine panel off matplotlib** (`ipf_refine_render.py` still uses - a matplotlib-Agg raster). The anyplotlib prerequisite is already merged - (1-D / PlotXY `double_click` now reports `xdata`/`ydata`). Plan: `PlotXY` + - `pcolormesh(clip_path=…)` (mirror `ipf_density.py`) with live `marker.set` - repaint per navigator move; the double-click mask uses the new data-coord event. -- **Split the 3.8k-line `find_vectors.py`** once its Qt tail is gone (compute core - vs action glue). -- **Broader dead-code sweep** (non-Qt): use INV-2 + E2E as the oracle — delete a - candidate, run the suite, keep iff green. Do this *after* Qt removal so the dead - Qt files don't confuse the graph. -- **Distribution story** *(resolved 2026-06)*: the locked decision in - `DISTRIBUTION_PLAN.md` is **PyCrucible + uv** (self-extracting exe with embedded - uv) as the portable/offline path, with a uv-managed installer as the primary. - So pycrucible config stays. Deleted the genuinely-stale **PyInstaller** - `spyde.spec` (untracked, referenced the removed `spyde.qt` icons, used by - nothing) and gitignored `*.spec` so it can't recur. The remaining distribution - work is the phased plan in `DISTRIBUTION_PLAN.md` (installer / GPU-readiness / - auto-update), tracked there, not here. diff --git a/DIFFRACTION_VECTORS_PLAN.md b/DIFFRACTION_VECTORS_PLAN.md deleted file mode 100644 index a87766b7..00000000 --- a/DIFFRACTION_VECTORS_PLAN.md +++ /dev/null @@ -1,1432 +0,0 @@ -# Diffraction Vector Finding — Feature Spec & Implementation Plan - -## Overview - -Add a **Find Diffraction Vectors** workflow to SpyDE that: - -1. Provides a caret popout with real-space Gaussian blur (σ), disk kernel radius (linked to a draggable `CircleROI`), threshold, min-distance separation, and subpixel refinement toggle — all auto-populated from the current diffraction pattern. -2. Creates a live preview window showing the processed image (Gaussian → window-normalized cross-correlation → thresholded) with vector markers overlaid on both the transformed image and the raw diffraction pattern. -3. On "Compute", produces a new signal tree node whose rendering overlays circles (radius = kernel radius) with `+` centers on the parent diffraction pattern. -4. Backs the vectors in a flat-buffer nested-tensor layout (like PyTorch NestedTensor) as `SpyDEDiffractionVectors`, gating strain mapping, virtual imaging through vectors, and density-based clustering. - ---- - -## Architecture Overview - -``` -ElectronDiffraction2D (4D-STEM: nav=[y,x], sig=[ky,kx]) - │ or 5D-STEM: nav=[time,y,x], sig=[ky,kx] - │ - ├── [Centered] ← existing node - │ │ - │ └── [Diffraction Vectors] ← NEW node (SpyDEDiffractionVectors) - │ │ rendering: circles(r=kernel_r) + '+' markers overlaid on parent - │ │ data: flat buffer + offsets (CSR / PyTorch NestedTensor layout) - │ │ - │ ├── [Strain Maps] ← existing pyxem workflow, now gated here - │ ├── [Virtual Images] ← vector-based virtual image creation - │ └── [Cluster Analysis] ← DBSCAN / HDBSCAN on vector positions -``` - ---- - -## Timing Budget (benchmarked on development machine) - -### Live Preview (per frame, warm cache) - -| Operation | 128×128 sig | 256×256 sig | -|---|---|---| -| Nav blur — `NavBlurCache` lookup (cached chunk) | **~0 ms** | **~0 ms** | -| Nav blur — single-frame fallback (cold chunk) | ~0.4 ms | ~1.2 ms | -| `match_template` disk r=10 | ~1.5 ms | ~5.6 ms | -| `peak_local_max` | ~2.0 ms | ~6.5 ms | -| Subpixel CoM refinement | ~0.1 ms | ~0.1 ms | -| **Total (warm)** | **~4 ms** | **~13 ms** | -| **Total (cold / first frame after chunk load)** | **~4 ms** | **~14 ms** | - -Live preview target: **<20 ms per frame** on a 256×256 pattern → 50 fps achievable on CPU. - -### NavBlurCache background cost (one-time per chunk change) -| Chunk size | Signal size | Pad+blur time (async background) | -|---|---|---| -| 16×16 nav | 128×128 sig | ~120 ms | -| 16×16 nav | 256×256 sig | ~490 ms | -| 32×32 nav | 128×128 sig | ~400 ms | - -These run in a daemon thread triggered by chunk-load events; the UI never waits for them. - -### Batch Compute (16×16 nav, 128×128 sig) -- Nav blur (`map_overlap` on full dataset): ~620 ms -- Template match + subpixel (256 patterns): ~790 ms -- Flat buffer assembly: ~1 ms -- **Total**: ~1.4 s - ---- - -## Part 1: The Real-Space Gaussian Blur — Two Paths - -The nav-space Gaussian blur has two completely different implementations depending on whether it's serving the **live preview** (single frame, fast) or the **batch compute** (full dataset, correct at all boundaries). - -### 1.1 How SpyDE's Plot System Already Loads Data - -`CachedDaskArray` (in `hyperspy._signals.lazy`) is the chunk-caching layer that `update_from_navigation_selection` calls via `_get_cache_dask_chunk`. It maintains: - -- **`core_cached_blocks`**: numpy arrays of the current navigation chunk(s) — already in memory -- **`surrounding_cached_blocks`**: numpy arrays of adjacent chunks — pre-fetched in the background (`cache_padding=1` when a Dask client is running) - -This means when the user moves the navigator, the neighboring patterns are already resident in memory with **zero disk I/O**. The live preview can exploit this directly. - -### 1.2 Live Preview Fast Path — `NavBlurCache` - -Instead of blurring a single frame (which ignores neighbors) or triggering a full `map_overlap` compute (which re-reads from disk), the live preview uses a **`NavBlurCache`** that piggybacks on the existing chunk cache. - -**Algorithm:** - -``` -On chunk change (new chunk loaded by CachedDaskArray): - 1. Fetch the raw chunk numpy array: shape (chunk_ny, chunk_nx, ky, kx) - 2. Reflect-pad by depth=ceil(3σ) in both nav dims: - padded shape = (chunk_ny + 2·depth, chunk_nx + 2·depth, ky, kx) - 3. Apply gaussian_filter(padded, sigma=(σ, σ, 0, 0)) in a daemon thread - 4. Trim: blurred_chunk = blurred_padded[depth:-depth, depth:-depth] - -> shape (chunk_ny, chunk_nx, ky, kx), correct at ALL positions - -On nav position change (within cached chunk): - if blurred_chunk ready: - return blurred_chunk[iy_local, ix_local] # O(1), zero copy - else: - # Cold: chunk just loaded, blur not done yet - # Fallback: gaussian_filter on the single pattern only - return gaussian_filter(raw_pattern, sigma=(σ, σ)) # ~1.2ms for 256×256 -``` - -**Why this is correct**: The reflect-pad ensures that even edge patterns of the chunk see real (reflected) neighbor data rather than a hard boundary. The interior patterns see actual neighboring patterns from the chunk. Cross-chunk boundary accuracy is limited to reflection artifacts, which are negligible for the param-tuning use case. - -**Why the async blur doesn't block the UI**: It runs in a daemon thread. For the first few nav moves after a chunk boundary (while blur is computing), the single-frame fallback takes ~1.2ms — fast enough that the user won't notice. - -```python -# spyde/actions/find_vectors.py - -class NavBlurCache: - """ - Async per-chunk Gaussian blur cache for live diffraction vector preview. - - Hooks into the chunk-loading lifecycle: - - Call update_chunk(chunk_array, chunk_id) when a new dask chunk becomes available. - - Call get_blurred(iy_local, ix_local) to retrieve the nav-blurred pattern. - """ - - def __init__(self, sigma: float): - self.sigma = sigma - self._depth = int(np.ceil(3 * sigma)) - self._blurred: Optional[np.ndarray] = None # (cy, cx, ky, kx) - self._raw_chunk: Optional[np.ndarray] = None # (cy, cx, ky, kx) - self._chunk_id: Optional[tuple] = None - self._blur_thread: Optional[threading.Thread] = None - self._lock = threading.Lock() - - def update_chunk(self, chunk_array: np.ndarray, chunk_id: tuple): - """Call when CachedDaskArray loads a new chunk. Starts async blur.""" - with self._lock: - if chunk_id == self._chunk_id: - return # already have this chunk - self._chunk_id = chunk_id - self._raw_chunk = chunk_array - self._blurred = None - - # Cancel previous blur thread (it will check chunk_id and exit early) - t = threading.Thread(target=self._do_blur, args=(chunk_array, chunk_id), daemon=True) - self._blur_thread = t - t.start() - - def _do_blur(self, chunk_array: np.ndarray, chunk_id: tuple): - from scipy.ndimage import gaussian_filter - d = self._depth - # Reflect-pad in nav dims so edge patterns see real (mirrored) neighbors - padded = np.pad(chunk_array, ((d, d), (d, d), (0, 0), (0, 0)), mode='reflect') - blurred_padded = gaussian_filter(padded, sigma=(self.sigma, self.sigma, 0, 0)) - trimmed = blurred_padded[d:-d, d:-d] - with self._lock: - if self._chunk_id == chunk_id: # still the active chunk - self._blurred = trimmed - - def get_blurred(self, iy_local: int, ix_local: int, raw_pattern: np.ndarray) -> np.ndarray: - """ - Return nav-blurred pattern at (iy_local, ix_local). - Uses cached blurred chunk if ready; falls back to single-frame blur. - """ - from scipy.ndimage import gaussian_filter - with self._lock: - blurred = self._blurred - if blurred is not None: - return blurred[iy_local, ix_local] - # Cold fallback: single-frame blur (ignores true neighbors, ~1.2ms) - return gaussian_filter(raw_pattern, sigma=(self.sigma, self.sigma)) - - def invalidate(self, sigma: float): - """Call when σ changes; clears cache and updates sigma.""" - with self._lock: - self.sigma = sigma - self._depth = int(np.ceil(3 * sigma)) - self._blurred = None - self._chunk_id = None -``` - -**Hooking into the chunk lifecycle**: The `NavBlurCache.update_chunk()` is called from the live-preview `_do_refit()` function. The current chunk can be obtained by accessing `signal.cached_dask_array` and checking which blocks are currently in `core_cached_blocks`. Since this is internal to hyperspy, the simpler approach is to call `signal._get_cache_dask_chunk(current_nav_indices)` which returns (or triggers) the chunk load, then read `signal.cached_dask_array.core_cached_blocks[0]` as a numpy array. - -### 1.3 Batch Compute Path — `map_overlap` (Correct for Full Dataset) - -The batch compute path uses the standard `dask.array.map_overlap` approach. This is correct at all boundaries (including dataset edges) because `map_overlap` loads ghost zones from disk before processing each chunk. - -**Why `map_blocks` is wrong** — verified experimentally: with a spike at nav position [4,0] and a chunk boundary at row 4, `map_blocks` gives 0.43 at the neighbor cell [3,0]; `map_overlap` gives 7.06, matching scipy's reference of 7.06 on the full array. - -```python -import dask.array as da -from scipy.ndimage import gaussian_filter - -depth_px = int(np.ceil(3 * sigma_nav)) -blurred = da.map_overlap( - gaussian_filter, - da_data, # (nav_y, nav_x, ky, kx) - depth=(depth_px, depth_px, 0, 0), # ghost zones only in nav dims - boundary='reflect', - sigma=(sigma_nav, sigma_nav, 0, 0), - dtype=np.float32, -) -``` - -Memory cost per chunk with ghost zones — `(C_y + 2·depth) × (C_x + 2·depth) × ky × kx × 4 bytes`: - -| σ | depth | Chunk | Padded size | RAM/chunk (256×256 sig) | -|---|---|---|---|---| -| 1.0 | 3 | 32×32 | 38×38 | 150 MB | -| 1.5 | 5 | 16×16 | 26×26 | 176 MB | -| 2.0 | 6 | 16×16 | 28×28 | 204 MB | - -```python -def _nav_chunk_size(sigma: float, max_ram_mb: float = 200, sig_shape: tuple = (256, 256)) -> int: - depth = int(np.ceil(3 * sigma)) - sig_pixels = sig_shape[0] * sig_shape[1] - max_padded = int(np.sqrt(max_ram_mb * 1e6 / (sig_pixels * 4))) - return max(depth + 1, max_padded - 2 * depth) -``` - -### 1.4 Axis Selection for 4D vs 5D - -In HyperSpy, for a signal of shape `(nav_0, ..., nav_k, sig_0, sig_1)`: -- The last two array axes are always the signal axes (ky, kx) -- The leading axes are navigation: for 4D-STEM `(nav_y, nav_x, ky, kx)`; for 5D-STEM `(time, nav_y, nav_x, ky, kx)` - -Gaussian blur should target only the **spatial navigation axes** (the last two navigation axes), never the time axis and never the signal axes: - -```python -nav_dim = signal.axes_manager.navigation_dimension # 2 for 4D, 3 for 5D -sig_dim = signal.axes_manager.signal_dimension # always 2 - -# sigma tuple: zeros for time (if present) and signal axes -sigma_tuple = tuple([0.0] * (nav_dim - 2) + [sigma_nav, sigma_nav] + [0.0] * sig_dim) -depth_tuple = tuple([0] * (nav_dim - 2) + [depth_px, depth_px] + [0] * sig_dim) -``` - -For 4D: `sigma=(σ, σ, 0, 0)`, `depth=(d, d, 0, 0)` -For 5D: `sigma=(0, σ, σ, 0, 0)`, `depth=(0, d, d, 0, 0)` - -The **5D UI** shows a selector for which axes to blur (pre-selected to the last two navigation axes). This is a `QCheckBox` row in the caret, auto-built from `axes_manager.navigation_axes[:-2]`. - -### 1.4 GPU Acceleration - -When a GPU worker is available (`main_window.dask_manager.gpu_worker_address` is not None): - -```python -# GPU path via CuPy (if installed) -try: - import cupy as cp - from cupyx.scipy.ndimage import gaussian_filter as gpu_gaussian - HAS_GPU = True -except ImportError: - HAS_GPU = False - -def _nav_blur_gpu(data: np.ndarray, sigma_tuple: tuple) -> np.ndarray: - """Apply navigation-space Gaussian blur on GPU.""" - arr = cp.asarray(data) - out = gpu_gaussian(arr, sigma=sigma_tuple) - return cp.asnumpy(out) -``` - -For the batch compute, the Dask scheduler dispatches the `map_overlap` task to the GPU worker if available. The live preview always runs on CPU (the GPU round-trip overhead negates the benefit for a single 256×256 frame at ~14ms). - ---- - -## Part 2: Live Preview Pipeline - -### 2.1 Core Compute Function (`spyde/actions/find_vectors.py`) - -```python -def _find_vectors_single_frame( - frame: np.ndarray, # (ky, kx) float32 — already nav-blurred - kernel_radius: int, # disk kernel radius in pixels - threshold: float, # correlation threshold in [0, 1] - min_distance: int, # minimum peak separation (pixels) - *, - subpixel: bool = True, # apply center-of-mass subpixel refinement - use_gpu: bool = False, -) -> tuple[np.ndarray, np.ndarray, np.ndarray]: - """ - Returns: - corr_map: (ky, kx) thresholded correlation image for display - raw_corr: (ky, kx) pre-threshold correlation (full range [-1,1]) - peaks: (N, 3) float32 — [ky_subpx, kx_subpx, intensity] - integer coords if subpixel=False - """ - disk = _make_disk(kernel_radius) # cached on first call per radius - raw_corr = match_template(frame, disk, pad_input=True) - corr_map = np.where(raw_corr >= threshold, raw_corr, 0.0) - peaks_px = peak_local_max(corr_map, min_distance=min_distance, threshold_abs=threshold) - - if len(peaks_px) == 0: - return corr_map, raw_corr, np.zeros((0, 3), dtype=np.float32) - - if subpixel: - refined = _subpixel_com(raw_corr, peaks_px) - else: - refined = np.column_stack([peaks_px.astype(np.float32), - raw_corr[peaks_px[:, 0], peaks_px[:, 1]]]) - return corr_map, raw_corr, refined - - -@functools.lru_cache(maxsize=16) -def _make_disk(radius: int) -> np.ndarray: - """Build a normalized flat-disk kernel; cached by radius.""" - disk = np.zeros((2*radius+1, 2*radius+1), dtype=np.float32) - yy, xx = np.ogrid[-radius:radius+1, -radius:radius+1] - disk[yy**2 + xx**2 <= radius**2] = 1.0 - disk /= disk.sum() - return disk - - -def _subpixel_com(corr: np.ndarray, peaks_px: np.ndarray, half_win: int = 2) -> np.ndarray: - """Center-of-mass subpixel refinement within ±half_win of each integer peak.""" - from scipy.ndimage import center_of_mass - out = np.empty((len(peaks_px), 3), dtype=np.float32) - for i, (py, px) in enumerate(peaks_px): - y0, y1 = max(0, py - half_win), min(corr.shape[0], py + half_win + 1) - x0, x1 = max(0, px - half_win), min(corr.shape[1], px + half_win + 1) - patch = corr[y0:y1, x0:x1] - dy, dx = center_of_mass(patch) - out[i, 0] = y0 + dy # ky subpixel - out[i, 1] = x0 + dx # kx subpixel - out[i, 2] = float(corr[py, px]) - return out -``` - -The live preview uses `NavBlurCache.get_blurred(iy_local, ix_local, raw_pattern)` (see §1.2) which returns the correctly nav-blurred pattern from the cached blurred chunk, or falls back to a single-frame approximation if the async blur hasn't completed yet. The batch compute uses the full `map_overlap` path (§1.3). - -### 2.2 Auto-population of Parameters - -```python -def _auto_params(frame: np.ndarray) -> dict: - """Estimate reasonable starting params from the current pattern.""" - # Kernel radius: 5% of shorter dimension, min 3 px - r_px = max(3, int(min(frame.shape) * 0.05)) - return dict( - sigma=1.5, - kernel_radius=r_px, - threshold=0.3, - min_distance=2 * r_px, - subpixel=True, - ) -``` - ---- - -## Part 3: UI — Caret Popout - -### 3.1 Caret Structure - -Following the virtual imaging / orientation mapping pattern: - -``` -[Find Vectors icon] → CaretGroup titled "Find Diffraction Vectors" - ├── Row: "Real-space σ" [slider + spinbox, range 0.1–10.0 px] - ├── Row: "Kernel radius" [slider + spinbox, range 1–50 px] ← linked to CircleROI - ├── Row: "Threshold" [slider + spinbox, range 0.0–1.0] - ├── Row: "Min distance" [slider + spinbox, range 1–100 px] - ├── [Subpixel CoM] [QCheckBox, default ON] - ├── [Live (ON)] [Compute] ← button_row - └── Status label: "N peaks found · X.Xms" - - [5D only: tab "Blur Axes"] - └── [time ☐] [y ☑] [x ☑] ← QCheckBoxes per nav axis -``` - -### 3.2 CircleROI Linking - -A `CircleROI` at the diffraction origin, radius = kernel_radius in data units: - -```python -r_data = kernel_radius_px * sig_ax[0].scale # pixels → Å⁻¹ -cx = sig_ax[1].size / 2.0 * sig_ax[1].scale + sig_ax[1].offset -cy = sig_ax[0].size / 2.0 * sig_ax[0].scale + sig_ax[0].offset - -circle_roi = CircleROI( - pos=(cx - r_data, cy - r_data), - size=(2 * r_data, 2 * r_data), - pen=mkPen("c", width=1.5), -) -plot.addItem(circle_roi) - -def _roi_to_spinbox(): - r = circle_roi.size().x() / 2.0 / sig_ax[0].scale - radius_spin.blockSignals(True) - radius_spin.setValue(r) - radius_spin.blockSignals(False) - _schedule_recompute() - -def _spinbox_to_roi(r_px): - r_d = r_px * sig_ax[0].scale - cx2, cy2 = (cx, cy) - circle_roi.blockSignals(True) - circle_roi.setPos(cx2 - r_d, cy2 - r_d) - circle_roi.setSize((2 * r_d, 2 * r_d)) - circle_roi.blockSignals(False) - _schedule_recompute() - -circle_roi.sigRegionChanged.connect(_roi_to_spinbox) -radius_spin.valueChanged.connect(_spinbox_to_roi) -``` - -### 3.3 Live Preview Window - -Two-panel `GraphicsLayoutWidget` MDI subwindow: - -``` -┌─────────────────────────────────────────────────┐ -│ Correlation map (thresholded) │ Raw pattern │ -│ [corr_map image] │ [frame image]│ -│ [+ markers at peaks] │ [○ + markers]│ -└─────────────────────────────────────────────────┘ -``` - -```python -# Build preview MDI window -preview_window = main_window.add_plot_window(is_navigator=False, signal_tree=plot.signal_tree) -preview_window.setWindowTitle("Vector Finding — Preview") - -glw = pg.GraphicsLayoutWidget() -left_plot = glw.addPlot(row=0, col=0, title="Correlation") -right_plot = glw.addPlot(row=0, col=1, title="Diffraction Pattern") - -left_img = pg.ImageItem() -right_img = pg.ImageItem() -left_plot.addItem(left_img) -right_plot.addItem(right_img) - -# ScatterPlotItem for '+' markers on correlation image -corr_scatter = pg.ScatterPlotItem(symbol='+', size=12, pen=mkPen('c', width=1.5), brush=None) -# ScatterPlotItem with circle symbols on raw pattern -raw_scatter = pg.ScatterPlotItem(symbol='o', size=kernel_radius_px*2, pen=mkPen('c', width=1), brush=None) -raw_plus = pg.ScatterPlotItem(symbol='+', size=8, pen=mkPen('c', width=1.5), brush=None) - -left_plot.addItem(corr_scatter) -right_plot.addItem(raw_scatter) -right_plot.addItem(raw_plus) -``` - -**Relay pattern** (identical to orientation mapping): - -```python -class _VectorRelay(QtCore.QObject): - vectors_ready = QtCore.Signal(object, object, object) # corr_map, raw_corr, peaks(N,3) - -relay = _VectorRelay() - -def _apply_results(corr_map, raw_corr, peaks): - left_img.setImage(corr_map.T) - raw_img_data = plot.current_data # grab current frame for right panel - if raw_img_data is not None: - right_img.setImage(np.asarray(raw_img_data).T) - spots = [{"pos": (p[1], p[0])} for p in peaks] # kx, ky for scene coords - corr_scatter.setData(spots) - raw_scatter.setData(spots) - raw_plus.setData(spots) - status_label.setText(f"{len(peaks)} peaks · {elapsed_ms:.1f}ms") - -relay.vectors_ready.connect(_apply_results) -``` - -### 3.4 Debounce + Generation Counter - -50ms debounce timer (identical to orientation mapping): - -```python -refit_timer = QTimer() -refit_timer.setInterval(50) -refit_timer.setSingleShot(True) -refit_generation = [0] - -def _schedule_recompute(): - refit_timer.start() - -def _do_refit(): - # Grab current nav indices and raw pattern on the GUI thread - nav_indices = _get_current_nav_indices(plot) - raw_frame = np.asarray(plot.current_data).copy() - - # Update NavBlurCache with the current chunk (triggers async blur if chunk changed) - cached_dask = getattr(signal, 'cached_dask_array', None) - if cached_dask is not None and cached_dask.core_cached_blocks: - # core_cached_blocks[0] is a Future or numpy array for the current chunk - block = cached_dask.core_cached_blocks[0] - if not isinstance(block, Future): - chunk_id = tuple(cached_dask.core_cached_block_inds[0]) - nav_blur_cache.update_chunk(block, chunk_id) - - refit_generation[0] += 1 - my_gen = refit_generation[0] - sigma = sigma_spin.value() - - def _run(): - if refit_generation[0] != my_gen: - return - t0 = time.perf_counter() - # Get nav-blurred pattern: O(1) from cache, or ~1.2ms single-frame fallback - iy_local = nav_indices[0] % signal.data.chunks[0][0] # local position in chunk - ix_local = nav_indices[1] % signal.data.chunks[1][0] - blurred = nav_blur_cache.get_blurred(iy_local, ix_local, raw_frame) - corr_map, raw_corr, peaks = _find_vectors_single_frame( - blurred, kernel_radius_spin.value(), threshold_spin.value(), - min_distance_spin.value(), subpixel=subpixel_check.isChecked() - ) - elapsed_ms = (time.perf_counter() - t0) * 1000 - if refit_generation[0] == my_gen: - relay.vectors_ready.emit(corr_map, raw_corr, peaks, elapsed_ms) - - threading.Thread(target=_run, daemon=True).start() - -refit_timer.timeout.connect(_do_refit) - -for spin in [sigma_spin, radius_spin, threshold_spin, mindist_spin]: - spin.valueChanged.connect(_schedule_recompute) -``` - ---- - -## Part 4: Batch Compute - -### 4.1 Algorithm - -```python -def _do_compute_vectors(signal, params, main_window, signal_tree): - """ - Full batch compute: - 1. Build sigma/depth tuples from nav_dim - 2. Rechunk for map_overlap - 3. Apply nav Gaussian via map_overlap - 4. Collect blurred data - 5. Run template match + subpixel per frame (in worker thread) - 6. Assemble flat buffer on main thread - 7. Build SpyDEDiffractionVectors + count map signal - 8. Add to signal tree - """ - data = signal.data # dask or numpy array - nav_dim = signal.axes_manager.navigation_dimension # 2 (4D) or 3 (5D) - sig_dim = signal.axes_manager.signal_dimension # 2 - sig_shape = signal.data.shape[-2:] - - sigma = params["sigma"] - depth_px = int(np.ceil(3 * sigma)) - sigma_tuple = tuple([0.0]*(nav_dim - 2) + [sigma, sigma] + [0.0]*sig_dim) - depth_tuple = tuple([0]*(nav_dim - 2) + [depth_px, depth_px] + [0]*sig_dim) - - # Determine chunk size so ghost-padded chunk fits in ~200 MB - chunk_nav = _nav_chunk_size(sigma, max_ram_mb=200, sig_shape=sig_shape) - - # 4D: chunks=(chunk_nav, chunk_nav, ky, kx) - # 5D: chunks=(1, chunk_nav, chunk_nav, ky, kx) — one time step per chunk - if nav_dim == 2: - chunks = (chunk_nav, chunk_nav) + sig_shape - nav_shape = signal.data.shape[:2] - else: - chunks = (1, chunk_nav, chunk_nav) + sig_shape - nav_shape = signal.data.shape[:nav_dim] - - if not hasattr(data, 'dask'): - da_data = da.from_array(data.astype(np.float32), chunks=chunks) - else: - da_data = data.astype(np.float32).rechunk(chunks) - - # Step 1: blurred is a lazy dask array; compute() loads chunks as needed - blurred_lazy = da.map_overlap( - gaussian_filter, - da_data, - depth=depth_tuple, - boundary='reflect', - sigma=sigma_tuple, - dtype=np.float32, - ) - - # Step 2: Collect results per partition - # Compute blurred_lazy into memory, then iterate frames - # For large datasets: compute chunk-by-chunk using dask futures - blurred = blurred_lazy.compute() # triggers actual disk reads + blur - - # Step 3: Template match + subpixel, frame by frame - # Flatten to (N_patterns, ky, kx) for uniform iteration - flat_blurred = blurred.reshape(-1, sig_shape[0], sig_shape[1]) - n_patterns = flat_blurred.shape[0] - kernel_r = params["kernel_radius"] - threshold = params["threshold"] - min_dist = params["min_distance"] - subpixel = params["subpixel"] - - frame_results = [] # list of (N_i, 3) float32 arrays: [ky_sub, kx_sub, intensity] - for i in range(n_patterns): - _, _, peaks = _find_vectors_single_frame( - flat_blurred[i], kernel_r, threshold, min_dist, subpixel=subpixel - ) - frame_results.append(peaks) - - # Step 4: Assemble flat buffer on main thread - sig_ax = signal.axes_manager.signal_axes - ky_scale = sig_ax[1].scale; ky_offset = sig_ax[1].offset - kx_scale = sig_ax[0].scale; kx_offset = sig_ax[0].offset - - counts = np.array([len(r) for r in frame_results], dtype=np.int64) - offsets = np.zeros(n_patterns + 1, dtype=np.int64) - np.cumsum(counts, out=offsets[1:]) - N_total = int(offsets[-1]) - - flat_buffer = np.zeros((N_total, 5), dtype=np.float32) - # columns: [nav_x, nav_y, kx_data, ky_data, intensity] - # nav_shape is (nav_y, nav_x) for 4D, (time, nav_y, nav_x) for 5D - nav_2d_shape = nav_shape[-2:] # always (nav_y, nav_x) - - for flat_idx, peaks in enumerate(frame_results): - if len(peaks) == 0: - continue - # Recover nav coordinates from flat_idx - # For 4D: flat_idx = iy * nav_x + ix - # For 5D: flat_idx = it * nav_y * nav_x + iy * nav_x + ix - s, e = offsets[flat_idx], offsets[flat_idx + 1] - iy = (flat_idx % (nav_2d_shape[0] * nav_2d_shape[1])) // nav_2d_shape[1] - ix = flat_idx % nav_2d_shape[1] - ky_data = peaks[:, 0] * ky_scale + ky_offset - kx_data = peaks[:, 1] * kx_scale + kx_offset - flat_buffer[s:e, 0] = ix - flat_buffer[s:e, 1] = iy - flat_buffer[s:e, 2] = kx_data - flat_buffer[s:e, 3] = ky_data - flat_buffer[s:e, 4] = peaks[:, 2] # intensity - - return SpyDEDiffractionVectors( - flat_buffer=flat_buffer, - offsets=offsets, - nav_shape=nav_2d_shape, - full_nav_shape=nav_shape, - sig_shape=sig_shape, - sig_axes=signal.axes_manager.signal_axes, - kernel_radius_px=float(kernel_r), - kernel_radius_data=float(kernel_r) * sig_ax[0].scale, - params=params, - ) -``` - -### 4.2 Dask Worker Dispatch - -The compute runs in a background thread (not blocking the GUI). Progress uses the existing `ComputeStatusIndicator` pattern from virtual imaging: - -```python -def _on_compute_clicked(): - btn.setEnabled(False) - status_label.setText("Computing…") - - def _run(): - vecs = _do_compute_vectors(signal, _get_params(), main_window, signal_tree) - # Marshal count map signal to GUI thread via pending_signal_queue - count_map = vecs.count_map() - count_signal = hs.signals.Signal2D(count_map) - count_signal.metadata.vectors = vecs - count_signal.metadata.Signal.signal_type = "diffraction_vectors" - _copy_nav_axes(signal, count_signal) - main_window._pending_signal_queue.append(count_signal) - QtCore.QMetaObject.invokeMethod( - main_window, "_flush_pending_signals", - QtCore.Qt.ConnectionType.QueuedConnection, - ) - # _flush_pending_signals calls signal_tree.add_node(signal, count_signal, "Diffraction Vectors") - - threading.Thread(target=_run, daemon=True).start() -``` - -### 4.3 Large Dataset Strategy (Dask Futures) - -For datasets that don't fit in RAM after blurring, the batch compute can be split into chunks of time steps or nav tiles and dispatched as Dask futures. The `frame_results` list is then built by collecting futures in order. This is the same polling pattern used by virtual imaging and orientation mapping. - ---- - -## Part 5: `SpyDEDiffractionVectors` Data Layout - -### 5.1 Design (PyTorch NestedTensor analogy) - -PyXEM's `DiffractionVectors2D` stores ragged data in a numpy object array `(nav_y, nav_x)` where each element is a `(N_i, 2)` array — memory-inefficient and slow for slicing. - -The new layout uses a **flat buffer + CSR offset array**: - -``` -flat_buffer: shape (N_total, 5) float32 - columns: [nav_x, nav_y, kx_data, ky_data, intensity] - -offsets: shape (n_patterns + 1,) int64 (CSR row pointer) - offsets[i] = start of position i in flat_buffer - offsets[-1] = N_total - -Slicing position (iy, ix): - flat_idx = iy * nav_shape[1] + ix - rows = flat_buffer[offsets[flat_idx] : offsets[flat_idx+1]] -``` - -### 5.2 Class Definition (`spyde/signals/diffraction_vectors.py`) - -```python -from __future__ import annotations -import numpy as np -from dataclasses import dataclass, field -from typing import Optional - - -@dataclass -class SpyDEDiffractionVectors: - flat_buffer: np.ndarray # (N_total, 5) float32: [nav_x, nav_y, kx, ky, intensity] - offsets: np.ndarray # (n_patterns+1,) int64 - nav_shape: tuple # (nav_y, nav_x) — the 2D spatial nav grid - full_nav_shape: tuple # same as nav_shape for 4D; (time, nav_y, nav_x) for 5D - sig_shape: tuple # (ky_size, kx_size) - sig_axes: object # hyperspy AxesManager signal_axes - kernel_radius_px: float - kernel_radius_data: float # in Å⁻¹ - params: dict = field(default_factory=dict) - _dense_cache: Optional[np.ndarray] = field(default=None, repr=False) - - # ── Indexing ───────────────────────────────────────────────────────────── - - def at(self, iy: int, ix: int) -> np.ndarray: - """(N, 5) array at navigation position (iy, ix).""" - i = iy * self.nav_shape[1] + ix - return self.flat_buffer[self.offsets[i]:self.offsets[i+1]] - - def kxy_at(self, iy: int, ix: int) -> np.ndarray: - """(N, 2) [kx, ky] in data units at (iy, ix).""" - return self.at(iy, ix)[:, 2:4] - - def intensities_at(self, iy: int, ix: int) -> np.ndarray: - return self.at(iy, ix)[:, 4] - - def count_map(self) -> np.ndarray: - """(nav_y, nav_x) int32 — vector count at each position.""" - return np.diff(self.offsets).reshape(self.nav_shape).astype(np.int32) - - def flatten(self) -> np.ndarray: - """Full (N_total, 5) flat buffer.""" - return self.flat_buffer - - # ── Dense conversion ───────────────────────────────────────────────────── - - def to_dense(self, fill_value: float = np.nan, max_vectors: int = None) -> np.ndarray: - """(nav_y, nav_x, max_n, 5) dense array; cached after first call.""" - if self._dense_cache is not None: - return self._dense_cache - counts = np.diff(self.offsets) - max_n = max_vectors or int(counts.max()) - nav_y, nav_x = self.nav_shape - dense = np.full((nav_y, nav_x, max_n, 5), fill_value, dtype=np.float32) - for flat_idx in range(nav_y * nav_x): - iy, ix = divmod(flat_idx, nav_x) - s, e = self.offsets[flat_idx], self.offsets[flat_idx+1] - n = e - s - if n > 0: - dense[iy, ix, :n] = self.flat_buffer[s:e] - self._dense_cache = dense - return dense - - # ── Unique vectors ──────────────────────────────────────────────────────── - - def get_unique_vectors(self, distance_threshold: float = 0.01) -> np.ndarray: - """(M, 2) [kx, ky] — unique vectors across entire scan.""" - kxy = self.flat_buffer[:, 2:4] - if distance_threshold == 0: - return np.unique(kxy, axis=0) - # iterative distance-comparison (same algorithm as pyxem) - from scipy.spatial.distance import cdist - unique = list(kxy[:1]) - for v in kxy[1:]: - dists = cdist([v], unique)[0] - if dists.min() >= distance_threshold: - unique.append(v) - return np.array(unique, dtype=np.float32) - - # ── PyXEM compatibility ──────────────────────────────────────────────────── - - def to_pyxem(self): - """Convert to pyxem DiffractionVectors2D (object-array form).""" - from pyxem.signals import DiffractionVectors2D - nav_y, nav_x = self.nav_shape - ragged = np.empty((nav_y, nav_x), dtype=object) - for iy in range(nav_y): - for ix in range(nav_x): - ragged[iy, ix] = self.kxy_at(iy, ix) - return DiffractionVectors2D(ragged) - - # ── Downstream gateways ─────────────────────────────────────────────────── - - def get_strain_maps(self, unstrained_vectors: np.ndarray, distance: float = 0.5): - """Delegate to pyxem after converting to DiffractionVectors2D.""" - dv = self.to_pyxem() - return dv.get_strain_maps(unstrained_vectors, distance=distance) - - def cluster(self, eps: float = 0.02, min_samples: int = 5): - """DBSCAN clustering on all kx,ky vectors. Returns labels array (N_total,).""" - from sklearn.cluster import DBSCAN - kxy = self.flat_buffer[:, 2:4] - return DBSCAN(eps=eps, min_samples=min_samples).fit_predict(kxy) - - # ── Markers for overlay rendering ───────────────────────────────────────── - - def spots_at(self, iy: int, ix: int) -> list: - """Return list of pyqtgraph ScatterPlotItem spot dicts for (iy, ix).""" - kxy = self.kxy_at(iy, ix) - # scene coords: scene_x = ky, scene_y = kx (pyqtgraph col-major) - r_scene = self.kernel_radius_data * 2 # diameter for 'o' symbol size - return [{"pos": (float(ky), float(kx)), "size": r_scene} - for kx, ky in kxy] - - # ── Constructors ────────────────────────────────────────────────────────── - - @classmethod - def from_ragged(cls, ragged: np.ndarray, nav_shape: tuple, **kwargs) -> SpyDEDiffractionVectors: - """Build from pyxem-style (nav_y, nav_x) object array of (N_i, 2) [kx, ky] arrays.""" - nav_y, nav_x = nav_shape - counts = np.array([len(ragged[iy, ix]) for iy in range(nav_y) for ix in range(nav_x)], dtype=np.int64) - offsets = np.zeros(nav_y * nav_x + 1, dtype=np.int64) - np.cumsum(counts, out=offsets[1:]) - N_total = int(offsets[-1]) - - flat_buffer = np.zeros((N_total, 5), dtype=np.float32) - for flat_idx in range(nav_y * nav_x): - iy, ix = divmod(flat_idx, nav_x) - s, e = offsets[flat_idx], offsets[flat_idx+1] - if e > s: - arr = ragged[iy, ix] # (N, 2) - flat_buffer[s:e, 0] = ix - flat_buffer[s:e, 1] = iy - flat_buffer[s:e, 2:4] = arr # kx, ky - return cls(flat_buffer=flat_buffer, offsets=offsets, - nav_shape=nav_shape, full_nav_shape=nav_shape, **kwargs) -``` - ---- - -## Part 6: Signal Tree Node & Overlay Rendering - -### 6.1 Node Representation (Option A) - -The vectors node stores a `(nav_y, nav_x)` count-map `Signal2D` with `metadata.vectors = SpyDEDiffractionVectors(...)`. This slots into `signal_tree.add_node()` without any changes to `SignalNode` or `BaseSignalTree`. - -```python -import hyperspy.api as hs - -count_signal = hs.signals.Signal2D(vecs.count_map().astype(np.float32)) -count_signal.metadata.vectors = vecs -count_signal.metadata.Signal.signal_type = "diffraction_vectors" -# Copy navigation axes from parent -for i, ax in enumerate(signal.axes_manager.navigation_axes): - count_signal.axes_manager.navigation_axes[i].scale = ax.scale - count_signal.axes_manager.navigation_axes[i].offset = ax.offset - count_signal.axes_manager.navigation_axes[i].units = ax.units - count_signal.axes_manager.navigation_axes[i].name = ax.name - -signal_tree.add_node(signal, count_signal, "Diffraction Vectors") -``` - -### 6.2 Overlay on Signal Plot - -When the user activates the vectors node in the signal tree, the signal plot switches to show the **parent diffraction pattern** with vector overlays. The existing `plot.set_current_signal()` machinery handles the parent image display. The overlay layer is added as pyqtgraph items: - -```python -def _activate_vector_overlay(plot, vecs: SpyDEDiffractionVectors): - scatter_circles = pg.ScatterPlotItem( - symbol='o', - pen=mkPen('c', width=1.0), - brush=None, - ) - scatter_plus = pg.ScatterPlotItem( - symbol='+', - size=8, - pen=mkPen('c', width=1.5), - brush=None, - ) - plot.addItem(scatter_circles) - plot.addItem(scatter_plus) - - def _update(nav_idx): - iy, ix = nav_idx - spots = vecs.spots_at(iy, ix) - scatter_circles.setData(spots) - scatter_plus.setData(spots) - - plot.sigNavigatorMoved.connect(_update) - _update(plot.current_nav_index) - return scatter_circles, scatter_plus -``` - ---- - -## Part 7: Action Registration - -New YAML entry in `spyde/actions/hyper_signal_actions/` (or appended to the existing pyxem config): - -```yaml -- name: "Find Diffraction Vectors" - icon: "peak_finding.svg" - function: "spyde.actions.find_vectors.find_diffraction_vectors" - signal_types: ["electron_diffraction"] - toggle: true -``` - -The `find_diffraction_vectors(toolbar, action_name, ...)` function follows the orientation mapping guard pattern: - -```python -_FV_BUILT_TOOLBARS: set = set() - -def find_diffraction_vectors(toolbar, action_name="Find Diffraction Vectors", *args, **kwargs): - tid = id(toolbar) - if tid in _FV_BUILT_TOOLBARS: - return - _FV_BUILT_TOOLBARS.add(tid) - # ... build CaretGroup, ROI, preview window, state dict ... -``` - ---- - -## Part 8: PyXEM Upstream Suggestions - -1. **`ElectronDiffraction2D.find_vectors_wncc(sigma_nav, kernel_radius, threshold, min_distance, subpixel=True)`** — new method exposing the window-normalized cross-correlation pipeline as a first-class operation, returning `DiffractionVectors2D`. The current `find_peaks(method='template_matching')` wraps this in an interactive widget that isn't scriptable cleanly. - -2. **`ElectronDiffraction2D.filter()` doesn't use `map_overlap`** — the current implementation calls `func(self.data, **kwargs)` directly. For lazy datasets with `dask_image.ndfilters.gaussian_filter`, this works because `dask_image` uses `map_overlap` internally. But with `scipy.ndimage.gaussian_filter` on a dask array it silently produces wrong results at chunk boundaries. The method should warn or document this. - -3. **`DiffractionVectors2D.from_flat_buffer(flat, offsets, nav_shape)`** — classmethod mirroring the CSR design above; submit as a PR to pyxem. - -4. **`DiffractionVectors2D.get_strain_maps` blocks on lazy input** — calls `.compute()` internally without returning a lazy result; document as a breaking limitation or fix. - -5. **`subpixel_refine` as a pipeline step** — the current API buries subpixel refinement in interactive find_peaks; expose as `DiffractionVectors2D.subpixel_refine(method='com'|'gaussian', half_win=2)` returning a new `DiffractionVectors2D`. - ---- - -## Part 9: Tests - -### 9.1 Unit Tests (`spyde/tests/test_find_vectors.py`) - -```python -import numpy as np -import pytest -import functools -from scipy.ndimage import gaussian_filter -from spyde.actions.find_vectors import ( - _find_vectors_single_frame, _auto_params, _nav_chunk_size, _subpixel_com -) -from spyde.signals.diffraction_vectors import SpyDEDiffractionVectors - - -# ── Core algorithm ──────────────────────────────────────────────────────────── - -def test_detects_known_peaks(): - frame = np.zeros((128, 128), dtype=np.float32) - expected = [(30, 40), (80, 70), (60, 100)] - for ky, kx in expected: - frame[ky-3:ky+3, kx-3:kx+3] = 10.0 - frame = gaussian_filter(frame, sigma=1.5) - _, _, peaks = _find_vectors_single_frame(frame, kernel_radius=5, threshold=0.3, min_distance=8) - for ey, ex in expected: - dists = np.hypot(peaks[:, 0] - ey, peaks[:, 1] - ex) - assert dists.min() < 3, f"Peak at ({ey},{ex}) not found; got {peaks[:, :2]}" - - -def test_threshold_controls_count(): - frame = np.zeros((128, 128), dtype=np.float32) - for ky, kx in [(20, 20), (60, 60), (100, 100)]: - frame[ky-3:ky+3, kx-3:kx+3] = 10.0 - frame = gaussian_filter(frame, sigma=1.5) - _, _, p_low = _find_vectors_single_frame(frame, 5, 0.05, 8) - _, _, p_high = _find_vectors_single_frame(frame, 5, 0.90, 8) - assert len(p_low) >= len(p_high) - - -def test_min_distance_prevents_duplicates(): - frame = np.zeros((64, 64), dtype=np.float32) - frame[30:34, 30:34] = 10.0 - frame[32:36, 32:36] = 10.0 - frame = gaussian_filter(frame, sigma=0.5) - _, _, peaks = _find_vectors_single_frame(frame, 4, 0.2, 10) - assert len(peaks) <= 1 - - -def test_output_shapes(): - frame = np.random.rand(64, 64).astype(np.float32) - corr, raw, peaks = _find_vectors_single_frame(frame, 4, 0.5, 5) - assert corr.shape == frame.shape - assert raw.shape == frame.shape - assert peaks.ndim == 2 and peaks.shape[1] == 3 - - -def test_zero_frame_no_peaks(): - frame = np.zeros((64, 64), dtype=np.float32) - _, _, peaks = _find_vectors_single_frame(frame, 4, 0.3, 5) - assert len(peaks) == 0 - - -def test_subpixel_refinement_moves_peaks(): - """Subpixel CoM should shift integer peaks to fractional positions.""" - frame = np.zeros((64, 64), dtype=np.float32) - # Off-center peak: blob centered at (30.3, 40.7) - for dy in range(-3, 4): - for dx in range(-3, 4): - dist = np.hypot(dy - 0.3, dx - 0.7) - frame[30 + dy, 40 + dx] = max(0, 5.0 - dist) - frame = gaussian_filter(frame, sigma=0.5) - _, _, peaks_sub = _find_vectors_single_frame(frame, 4, 0.1, 6, subpixel=True) - _, _, peaks_int = _find_vectors_single_frame(frame, 4, 0.1, 6, subpixel=False) - # Subpixel peaks should have non-integer coordinates - assert len(peaks_sub) > 0 - assert any(p % 1 != 0 for p in peaks_sub[0, :2]) - # Integer peaks should be whole numbers - assert all(p % 1 == 0 for p in peaks_int[0, :2]) - - -def test_disk_kernel_cached(): - from spyde.actions.find_vectors import _make_disk - d1 = _make_disk(8) - d2 = _make_disk(8) - assert d1 is d2 # lru_cache hit - - -def test_auto_params_valid_ranges(): - frame = np.random.rand(128, 128).astype(np.float32) - p = _auto_params(frame) - assert 0 < p["sigma"] <= 10 - assert 1 <= p["kernel_radius"] < 64 - assert 0 < p["threshold"] < 1 - assert p["min_distance"] >= 1 - assert isinstance(p["subpixel"], bool) - - -# ── NavBlurCache ────────────────────────────────────────────────────────────── - -def test_nav_blur_cache_warm_hit(): - """After update_chunk, get_blurred returns the correctly blurred pattern.""" - from spyde.actions.find_vectors import NavBlurCache - from scipy.ndimage import gaussian_filter - - sigma = 1.5 - cache = NavBlurCache(sigma=sigma) - chunk = np.random.rand(8, 8, 64, 64).astype(np.float32) - chunk[4, 4, 30, 30] = 20.0 # spike at center of chunk - - cache.update_chunk(chunk, chunk_id=(0, 0)) - cache._blur_thread.join() # wait for async blur to finish - - result = cache.get_blurred(4, 4, raw_pattern=chunk[4, 4]) - # Reference: full-array blur - ref = gaussian_filter(chunk, sigma=(sigma, sigma, 0, 0))[4, 4] - np.testing.assert_allclose(result, ref, atol=1e-3) - - -def test_nav_blur_cache_cold_fallback(): - """Before async blur completes, get_blurred falls back to single-frame blur.""" - from spyde.actions.find_vectors import NavBlurCache - from scipy.ndimage import gaussian_filter - - sigma = 1.5 - cache = NavBlurCache(sigma=sigma) - chunk = np.random.rand(16, 16, 64, 64).astype(np.float32) - cache._chunk_id = (0, 0) # pretend chunk is loaded - cache._raw_chunk = chunk - cache._blurred = None # but blur not done yet - - raw_pattern = chunk[8, 8] - result = cache.get_blurred(8, 8, raw_pattern=raw_pattern) - ref = gaussian_filter(raw_pattern, sigma=(sigma, sigma)) - np.testing.assert_allclose(result, ref, atol=1e-6) - - -def test_nav_blur_cache_invalidate_clears(): - """invalidate() clears cached state and updates sigma.""" - from spyde.actions.find_vectors import NavBlurCache - cache = NavBlurCache(sigma=1.5) - cache._blurred = np.zeros((8, 8, 64, 64), dtype=np.float32) - cache._chunk_id = (0, 0) - cache.invalidate(sigma=2.0) - assert cache._blurred is None - assert cache._chunk_id is None - assert cache.sigma == 2.0 - - -def test_nav_blur_cache_chunk_id_guards_stale_blur(): - """A blur started for chunk (0,0) should not overwrite results for chunk (1,0).""" - from spyde.actions.find_vectors import NavBlurCache - import time - - sigma = 1.5 - cache = NavBlurCache(sigma=sigma) - chunk_a = np.zeros((4, 4, 16, 16), dtype=np.float32) - chunk_b = np.ones((4, 4, 16, 16), dtype=np.float32) - - cache.update_chunk(chunk_a, (0, 0)) - # Immediately switch to chunk_b before blur of chunk_a finishes - cache.update_chunk(chunk_b, (1, 0)) - cache._blur_thread.join() # wait for blur_b to finish - - # Result should be for chunk_b (ones), not chunk_a (zeros) - with cache._lock: - assert cache._chunk_id == (1, 0) - if cache._blurred is not None: - assert cache._blurred.mean() > 0.5 # chunk_b was ones - - -def test_nav_blur_cache_edge_accuracy(): - """Edge patterns of the chunk should be within 5% of the full-array reference.""" - from spyde.actions.find_vectors import NavBlurCache - from scipy.ndimage import gaussian_filter - - sigma = 1.5 - # Simulate: chunk is surrounded by actual data (not zeros) - full = np.random.rand(24, 24, 32, 32).astype(np.float32) - full[12, 12, 16, 16] = 20.0 - - ref_blurred = gaussian_filter(full, sigma=(sigma, sigma, 0, 0)) - - # NavBlurCache sees only the middle 8x8 chunk - chunk = full[8:16, 8:16].copy() - cache = NavBlurCache(sigma=sigma) - cache.update_chunk(chunk, (0, 0)) - cache._blur_thread.join() - - # At the chunk EDGE (position 0,0 in local = global (8,8)): - result_edge = cache.get_blurred(0, 0, raw_pattern=chunk[0, 0]) - ref_edge = ref_blurred[8, 8] - # Reflect-pad gives different boundary than true neighbors; allow 5% error - rel_err = np.max(np.abs(result_edge - ref_edge)) / (np.max(np.abs(ref_edge)) + 1e-6) - assert rel_err < 0.05, f"Edge pattern error too large: {rel_err:.3f}" - - -def test_nav_blur_cache_speed(): - """Warm cache lookup must be sub-millisecond.""" - import time - from spyde.actions.find_vectors import NavBlurCache - - cache = NavBlurCache(sigma=1.5) - chunk = np.random.rand(16, 16, 256, 256).astype(np.float32) - cache.update_chunk(chunk, (0, 0)) - cache._blur_thread.join() - - raw_pattern = chunk[8, 8] - N = 200 - t0 = time.perf_counter() - for _ in range(N): - cache.get_blurred(8, 8, raw_pattern) - avg_ms = (time.perf_counter() - t0) / N * 1000 - assert avg_ms < 1.0, f"Warm cache lookup too slow: {avg_ms:.2f}ms" - - -# ── Chunk size calculation ──────────────────────────────────────────────────── - -def test_nav_chunk_size_respects_memory_limit(): - chunk = _nav_chunk_size(sigma=2.0, max_ram_mb=200, sig_shape=(128, 128)) - depth = int(np.ceil(3 * 2.0)) - ram_mb = (chunk + 2*depth)**2 * 128 * 128 * 4 / 1e6 - assert ram_mb <= 200, f"Chunk uses {ram_mb:.0f} MB, limit 200 MB" - - -def test_nav_chunk_size_larger_than_depth(): - for sigma in [0.5, 1.0, 2.0, 3.0]: - chunk = _nav_chunk_size(sigma, max_ram_mb=200, sig_shape=(256, 256)) - depth = int(np.ceil(3 * sigma)) - assert chunk > depth, f"sigma={sigma}: chunk={chunk} <= depth={depth}" - - -# ── Navigation Gaussian blur with map_overlap ───────────────────────────────── - -def test_map_overlap_correct_at_chunk_boundary(): - """Verify map_overlap produces the same result as full-array gaussian_filter.""" - import dask.array as da - data = np.zeros((8, 8, 32, 32), dtype=np.float32) - data[4, 0, 16, 16] = 100.0 # spike straddling chunk boundary at row 4 - - sigma = 1.5 - depth = int(np.ceil(3 * sigma)) - da_data = da.from_array(data, chunks=(4, 4, 32, 32)) - - result = da.map_overlap( - gaussian_filter, da_data, - depth=(depth, depth, 0, 0), boundary='reflect', - sigma=(sigma, sigma, 0, 0), dtype=np.float32, - ).compute() - - reference = gaussian_filter(data, sigma=(sigma, sigma, 0, 0)) - # Value at [3,0,16,16] must match reference (spike bleeds across boundary) - np.testing.assert_allclose(result[3, 0, 16, 16], reference[3, 0, 16, 16], rtol=1e-4) - - -def test_map_overlap_wrong_without_overlap(): - """Confirm that map_blocks (no overlap) gives wrong result at chunk boundaries.""" - import dask.array as da - data = np.zeros((8, 8, 32, 32), dtype=np.float32) - data[4, 0, 16, 16] = 100.0 - - sigma = 1.5 - da_data = da.from_array(data, chunks=(4, 4, 32, 32)) - - wrong = da_data.map_blocks(gaussian_filter, sigma=(sigma, sigma, 0, 0), dtype=np.float32).compute() - reference = gaussian_filter(data, sigma=(sigma, sigma, 0, 0)) - - # The two should differ at the boundary - assert abs(wrong[3, 0, 16, 16] - reference[3, 0, 16, 16]) > 0.1, \ - "Expected chunk boundary artifact but result matched reference" - - -def test_sigma_tuple_4d(): - """4D signal: sigma tuple is (s, s, 0, 0).""" - import hyperspy.api as hs - s = hs.signals.Signal2D(np.zeros((4, 4, 16, 16))) - nav_dim = s.axes_manager.navigation_dimension # 2 - sig_dim = s.axes_manager.signal_dimension # 2 - sigma_nav = 1.5 - sigma_tuple = tuple([0.0]*(nav_dim-2) + [sigma_nav, sigma_nav] + [0.0]*sig_dim) - assert sigma_tuple == (1.5, 1.5, 0.0, 0.0) - - -def test_sigma_tuple_5d(): - """5D signal: sigma tuple is (0, s, s, 0, 0) — time axis gets zero.""" - import hyperspy.api as hs - s = hs.signals.Signal2D(np.zeros((3, 4, 4, 16, 16))) - nav_dim = s.axes_manager.navigation_dimension # 3 - sig_dim = s.axes_manager.signal_dimension # 2 - sigma_nav = 1.5 - sigma_tuple = tuple([0.0]*(nav_dim-2) + [sigma_nav, sigma_nav] + [0.0]*sig_dim) - assert sigma_tuple == (0.0, 1.5, 1.5, 0.0, 0.0) - - -# ── SpyDEDiffractionVectors ─────────────────────────────────────────────────── - -def _make_vecs(nav_shape=(4, 4), n_per_pos=3): - nav_y, nav_x = nav_shape - n_nav = nav_y * nav_x - counts = np.full(n_nav, n_per_pos, dtype=np.int64) - offsets = np.zeros(n_nav + 1, dtype=np.int64) - np.cumsum(counts, out=offsets[1:]) - N = int(offsets[-1]) - flat = np.random.rand(N, 5).astype(np.float32) - return SpyDEDiffractionVectors( - flat_buffer=flat, offsets=offsets, - nav_shape=nav_shape, full_nav_shape=nav_shape, - sig_shape=(128, 128), sig_axes=None, - kernel_radius_px=5.0, kernel_radius_data=0.05, - ) - - -def test_at_returns_correct_rows(): - vecs = _make_vecs((3, 3), n_per_pos=4) - for iy in range(3): - for ix in range(3): - assert vecs.at(iy, ix).shape == (4, 5) - - -def test_kxy_at_correct_columns(): - vecs = _make_vecs() - assert vecs.kxy_at(0, 0).shape == (3, 2) - - -def test_count_map(): - vecs = _make_vecs((4, 4), n_per_pos=3) - cm = vecs.count_map() - assert cm.shape == (4, 4) - assert (cm == 3).all() - - -def test_to_dense_shape_and_cache(): - vecs = _make_vecs((2, 3), n_per_pos=5) - d1 = vecs.to_dense() - assert d1.shape == (2, 3, 5, 5) - d2 = vecs.to_dense() - assert d1 is d2 # cache hit - - -def test_flatten_full_buffer(): - vecs = _make_vecs((2, 2), n_per_pos=3) - assert vecs.flatten().shape == (12, 5) - - -def test_from_ragged_roundtrip(): - nav_shape = (3, 4) - nav_y, nav_x = nav_shape - ragged = np.empty(nav_shape, dtype=object) - for i in range(nav_y): - for j in range(nav_x): - n = np.random.randint(1, 8) - ragged[i, j] = np.random.rand(n, 2).astype(np.float32) - - vecs = SpyDEDiffractionVectors.from_ragged( - ragged, nav_shape, - full_nav_shape=nav_shape, sig_shape=(128, 128), - sig_axes=None, kernel_radius_px=5.0, kernel_radius_data=0.05, - ) - for i in range(nav_y): - for j in range(nav_x): - assert len(vecs.at(i, j)) == len(ragged[i, j]) - - -def test_to_pyxem_type(): - from pyxem.signals import DiffractionVectors2D - vecs = _make_vecs() - dv = vecs.to_pyxem() - assert isinstance(dv, DiffractionVectors2D) - - -# ── Performance ─────────────────────────────────────────────────────────────── - -def test_single_frame_pipeline_under_20ms(): - import time - frame = np.random.rand(256, 256).astype(np.float32) - frame = gaussian_filter(frame, sigma=1.5) - _find_vectors_single_frame(frame, 12, 0.3, 10) # warm up - t0 = time.perf_counter() - for _ in range(10): - _find_vectors_single_frame(frame, 12, 0.3, 10) - avg_ms = (time.perf_counter() - t0) / 10 * 1000 - assert avg_ms < 20, f"Pipeline too slow: {avg_ms:.1f}ms (limit 20ms)" -``` - -### 9.2 Integration Tests (`spyde/tests/test_find_vectors_integration.py`) - -```python -import pytest -import numpy as np - - -@pytest.mark.usefixtures("qapp") -def test_caret_builds_on_4d_dataset(stem_4d_dataset, qtbot): - """find_diffraction_vectors caret builds without error.""" - window = stem_4d_dataset["window"] - subwindows = stem_4d_dataset["subwindows"] - signal_pw = next(sw for sw in subwindows if not sw.plot.is_navigator) - toolbar = signal_pw.plot.toolbar - - from spyde.actions.find_vectors import find_diffraction_vectors - find_diffraction_vectors(toolbar) - assert hasattr(toolbar, "_fv_state") - - -@pytest.mark.usefixtures("qapp") -def test_compute_adds_vectors_node(stem_4d_dataset, qtbot): - """Batch compute adds a SpyDEDiffractionVectors-backed node to the signal tree.""" - window = stem_4d_dataset["window"] - trees = stem_4d_dataset["signal_trees"] - tree = trees[0] - n_before = sum(1 for _ in tree.walk()) - - from spyde.actions.find_vectors import _do_compute_vectors - from spyde.signals.diffraction_vectors import SpyDEDiffractionVectors - import hyperspy.api as hs - - signal = tree.root - params = dict(sigma=1.0, kernel_radius=4, threshold=0.3, min_distance=8, subpixel=True) - vecs = _do_compute_vectors(signal, params, window, tree) - - assert isinstance(vecs, SpyDEDiffractionVectors) - assert vecs.flat_buffer.shape[1] == 5 - assert vecs.count_map().shape == signal.data.shape[:2] - - -@pytest.mark.usefixtures("qapp") -def test_sigma_tuple_5d(stem_5d_dataset): - """5D dataset: sigma tuple has zero in time axis position.""" - trees = stem_5d_dataset["signal_trees"] - signal = trees[0].root - nav_dim = signal.axes_manager.navigation_dimension - sig_dim = signal.axes_manager.signal_dimension - sigma = 1.5 - sigma_tuple = tuple([0.0]*(nav_dim-2) + [sigma, sigma] + [0.0]*sig_dim) - assert sigma_tuple[0] == 0.0 # time axis = 0 - assert sigma_tuple[1] == sigma - assert sigma_tuple[2] == sigma - - -@pytest.mark.usefixtures("qapp") -def test_chunk_boundary_blur_correctness_on_real_signal(stem_4d_dataset): - """map_overlap result matches full-array gaussian_filter for a 4D signal.""" - import dask.array as da - trees = stem_4d_dataset["signal_trees"] - signal = trees[0].root - data = np.asarray(signal.data).astype(np.float32) - - sigma = 1.5 - depth = int(np.ceil(3 * sigma)) - da_data = da.from_array(data, chunks=(4, 4) + data.shape[2:]) - - overlap_result = da.map_overlap( - gaussian_filter, da_data, - depth=(depth, depth, 0, 0), boundary='reflect', - sigma=(sigma, sigma, 0, 0), dtype=np.float32, - ).compute() - - reference = gaussian_filter(data, sigma=(sigma, sigma, 0, 0)) - np.testing.assert_allclose(overlap_result, reference, rtol=1e-4, - err_msg="map_overlap blur differs from reference") -``` - ---- - -## Part 10: Implementation Sequence - -### Phase 1 — Core algorithm + data class (no UI) -1. `spyde/signals/diffraction_vectors.py`: `SpyDEDiffractionVectors` -2. `spyde/actions/find_vectors.py`: `_find_vectors_single_frame`, `_make_disk` (cached), `_subpixel_com`, `_auto_params`, `_nav_chunk_size`, sigma/depth tuple helpers -3. Pass all unit tests from §9.1 - -### Phase 2 — Batch compute -4. `_do_compute_vectors`: `map_overlap` blur → template match → flat buffer assembly -5. Wire to `_pending_signal_queue` / `_flush_pending_signals` -6. Pass integration tests from §9.2 - -### Phase 3 — Live preview caret UI -7. `find_diffraction_vectors(toolbar, ...)`: `CaretGroup` + parameter rows + `CircleROI` + `QCheckBox` for subpixel -8. Debounce timer, generation counter, `_VectorRelay`, two-panel preview window -9. Auto-populate on caret open via `_auto_params` -10. 5D axis selection UI (`QCheckBox` per extra nav axis) - -### Phase 4 — Signal tree overlay -11. `_activate_vector_overlay`: `ScatterPlotItem` circles + plus markers -12. Wire to `sigNavigatorMoved` / navigation index changes -13. Connect "Compute" button → `_on_compute_clicked` → background thread - -### Phase 5 — Downstream gateways -14. Strain mapping toolbar action on vectors node -15. Virtual image from vectors toolbar action -16. Clustering toolbar action - -### Phase 6 — Polish & GPU -17. GPU path (CuPy) for `_do_compute_vectors` nav blur step -18. Subpixel refinement with Gaussian fitting option (alternative to CoM) -19. Benchmark and profile on real 4D-STEM data - ---- - -## Open Questions (Resolved) - -1. **Nav blur scope for 5D**: Blur only spatial nav axes (last 2 of nav). Time axis gets σ=0. User can override via checkbox row in caret. ✓ - -2. **Ragged gather strategy**: Collect frame results into `frame_results = list of (N_i, 3) arrays` in main thread after `blurred.compute()`. Assemble `flat_buffer` on main thread. For very large datasets (blurred array > available RAM), process in time-step chunks using Dask futures with the existing progress polling pattern. ✓ - -3. **Signal tree node type**: Option A — count map `Signal2D` + `metadata.vectors`. Clean fit with existing `add_node()` / `update_plot_states()` infrastructure. ✓ - -4. **Subpixel refinement**: Center-of-mass in a ±2 px window around each integer peak. Toggle in caret (ON by default). Adds negligible time (<0.1ms for typical N peaks). ✓ - -5. **PyXEM upstream**: Submit `from_flat_buffer` + `find_vectors_wncc` as PR to pyxem. SpyDE carries locally until merged. ✓ diff --git a/DISTRIBUTION_PLAN.md b/DISTRIBUTION_PLAN.md deleted file mode 100644 index 4d657e86..00000000 --- a/DISTRIBUTION_PLAN.md +++ /dev/null @@ -1,169 +0,0 @@ -# SpyDE Distribution, Installer & Auto-Update Plan - -Scope: make SpyDE easy to **install**, **update**, and **run with a correctly -set-up GPU**, with `uv` as the engine. This is a design doc — no code yet. - -## 1. Where we are today - -| Concern | Current state | -|---|---| -| Bundling | **PyCrucible**: one self-extracting exe with an embedded `uv` (0.9.21) + `uv.lock`; on first run uv resolves deps into a venv and launches `main.py`. | -| Release | Tag-triggered CI (`release.yml`): builds `SpyDE.exe`→zip (Win), `.dmg` (mac), `.AppImage` (Linux); publishes a GitHub Release. | -| Install (Win) | Zip + `create_shortcut.ps1` run manually. **No installer, no Add/Remove Programs entry, no fixed install dir.** | -| Updates | **None.** No version check, no channel, no in-app update. | -| GPU | `torch` is a dep with `[tool.uv] torch-backend = "auto"`; `vector_orientation_gpu.select_device()` picks CUDA→MPS→CPU at runtime. **No first-run validation, no diagnostics surfaced, no driver guidance.** | -| Version | Duplicated: `pyproject.toml` and `spyde/__init__.py` (`0.0.1`). | - -The PyCrucible + uv foundation is good — uv already does runtime dependency -resolution, which is exactly what an update mechanism can reuse. The gaps are -**install UX**, **update flow**, **GPU readiness**, and **version hygiene**. - -## 2. Goals & non-goals - -**Goals** -1. A real **installer** per platform (Win: MSI/NSIS w/ Add-Remove entry, Start-Menu + optional desktop shortcut, per-user default; mac: signed/notarized `.dmg`→`.app`; Linux: AppImage + optional `.deb`). -2. **Check for updates** (in-app, manual + optional auto-check on startup) against GitHub Releases, with a clear "update available → download/apply" flow. -3. **GPU readiness**: first-run (and on-demand) detection of CUDA/MPS, the right `torch` wheel installed, a diagnostics panel, and graceful CPU fallback with an explanation — never a silent slow path. -4. `uv`-powered throughout: dependency resolution, the GPU-correct wheel selection, and (where feasible) the update install step. -5. Robust: signed where possible, atomic updates (no half-written installs), offline-friendly first run optional, reproducible via `uv.lock`. - -**Non-goals (this round)** -- An app store / winget / Homebrew cask (can layer on later). -- Delta/binary-patch updates (full-artifact updates are fine at this size cadence). -- Background silent auto-update without consent (we prompt). - -## 3. Design decisions (the important ones) - -### 3a. Two viable architectures — pick per the size/speed tradeoff - -**Option A — "uv-managed app" (recommended).** The installer lays down a small -launcher + the bundled `uv` + the project (`pyproject.toml`/`uv.lock`). First -launch runs `uv sync` into a managed venv next to the install. Updates = -fetch the new `pyproject.toml`/`uv.lock` (or a versioned source bundle) and -`uv sync` again — fast, incremental, and **this is where uv shines**: -`uv sync` only changes what the lock changed, and `--torch-backend=auto` -fetches the right GPU wheel. -- *Pros:* tiny installer, fast incremental updates, GPU wheel handled by uv, no - 3 GB torch baked into the installer, reproducible from the lock. -- *Cons:* first run needs network (mitigate: optional "offline bundle" build - that pre-seeds the uv cache); venv lives on the user's disk. - -**Option B — "fully bundled".** Keep PyCrucible's single self-contained exe -(torch baked in) and just wrap it in an installer. Updates replace the whole exe. -- *Pros:* fully offline, one file. -- *Cons:* ~3 GB artifact per platform/GPU variant, slow updates (re-download - everything), GPU variant matrix explodes (cu121/cu124/cpu/mps). - -**Recommendation: Option A.** It's the natural fit for "powered by uv", keeps -artifacts small, makes GPU-correct installs automatic, and makes updates cheap. -Keep Option B's single-exe as a fallback "portable" download. - -### 3b. Update transport: GitHub Releases as the channel -- A small `latest.json` manifest published per release (`version`, per-platform - artifact URLs, sha256, min-supported-version, release notes URL, channel). -- App checks `https://github.com//spyde/releases/latest` (or the manifest) - → compares semver → prompts. -- Channels: `stable` (tags `vX.Y.Z`) and optional `beta` (tags `vX.Y.Z-rc.N`). - -### 3c. Versioning: single source of truth -- Make `spyde/__init__.__version__` the source; have `pyproject.toml` read it - dynamically (`[tool.setuptools.dynamic]` or hatch), OR generate both from a - git tag at build (`uv version` / `hatch-vcs`). Update checks compare - `__version__` to the manifest. - -### 3d. GPU readiness as a first-class step -- A `spyde/gpu_setup.py` module: detect platform + NVIDIA driver (via `nvidia-smi` - / `torch.cuda`), decide the correct backend, and verify a real torch op runs. -- Installer/first-run: run `uv sync --torch-backend=auto` so the matching wheel - is fetched (cu12x on Win/Linux+NVIDIA, MPS wheel on mac arm64, CPU otherwise). -- In-app **GPU diagnostics** panel (Help → GPU Status): device name, torch - build, CUDA/MPS availability, a "re-run GPU setup" button, and the - `gpu_unavailable_reason()` string we already expose. Surfaces driver-missing / - CPU-only situations instead of the silent slow path the vector-OM work hit. - -## 4. Concrete deliverables (phased) - -### Phase 0 — Foundations (low risk, do first) -- [ ] Single-source version (`__version__` → pyproject dynamic). -- [ ] `spyde/_build_info.py` written at build time (version, git sha, channel, - build date) for the About box + update checks. -- [ ] `tools/release.py` (uv-run) to cut a release: bump version, tag, push. - -### Phase 1 — GPU readiness -- [ ] `spyde/gpu_setup.py`: `detect()`, `ensure_backend()` (wraps - `uv sync --torch-backend=auto`), `diagnostics()`. -- [ ] Help → "GPU Status…" dialog (reuses `vector_orientation_gpu.select_device` - / `gpu_unavailable_reason`). -- [ ] First-run check: if an accelerated device exists but torch is CPU-only, - offer to fetch the GPU wheel via uv. -- [ ] Tests: `gpu_setup` detection logic (mock `nvidia-smi`/torch), subprocess- - isolated as the existing GPU tests are. - -### Phase 2 — uv-managed install + update core -- [ ] `spyde/updater.py`: `check_for_updates(channel)` → parse manifest, - compare semver; `download_and_stage()`; `apply_update()` (atomic swap of - the source bundle + `uv sync`, then relaunch). -- [ ] Help → "Check for Updates…" (manual) + optional startup check with a - "remind me later / skip this version" choice (persisted in the session - settings store, `~/.spyde/settings.json` — the app is Qt-free now). -- [ ] `latest.json` manifest generation added to `release.yml`. -- [ ] Robustness: verify sha256, stage to a temp dir, swap on success only, - keep the previous version for rollback, single-instance guard during apply. - -### Phase 3 — Real installers -- [ ] **Windows**: NSIS or WiX MSI — installs launcher+uv+source under - `%LOCALAPPDATA%\Programs\SpyDE` (per-user, no admin), Start-Menu + desktop - shortcuts, Add/Remove Programs entry, file associations (`.hspy`, `.zspy`, - `.mrc`), uninstaller. First-launch runs `uv sync`. -- [ ] **macOS**: `.app` in a signed+notarized `.dmg` (needs Apple Developer ID; - flag as a prerequisite/cost). Sparkle-style update or our updater. -- [ ] **Linux**: keep AppImage; add optional `.deb`. AppImage self-update via - `appimageupdate` or our updater. -- [ ] CI: extend `release.yml` to produce installers + manifest; matrix stays - per-OS. Add code-signing secrets (Win EV cert, Apple ID) as a follow-up. - -### Phase 4 — Polish -- [ ] About box (version, sha, GPU summary, licenses). -- [ ] Offline-install bundle variant (pre-seeded uv cache) for air-gapped labs - (microscope PCs are often offline — worth it for this audience). -- [ ] Crash/first-run telemetry opt-in (optional). - -## 5. Risks & mitigations -- **First-run network dependency (Option A).** Mitigate with an optional - offline bundle and a clear progress UI during the initial `uv sync`. -- **Code signing.** Unsigned Win/mac apps trigger SmartScreen/Gatekeeper. - Win EV cert (~$300/yr) and Apple Developer ($99/yr) are prerequisites for a - clean install UX — call out as a decision/cost, ship unsigned in the interim - with install instructions. -- **GPU wheel size/index.** `--torch-backend=auto` needs the PyTorch index - reachable; pin a known-good torch version in the lock so updates are - deterministic. -- **Update atomicity.** Never overwrite the running install in place — stage + - swap + relaunch, keep N-1 for rollback. -- **Microscope-PC constraints.** Often locked-down/offline/older GPUs — per-user - install (no admin), offline bundle, and CPU fallback all matter here. - -## 6. Decisions (locked 2026-06-15; update-channel superseded 2026-07-01) - -| Question | Decision | -|---|---| -| Architecture | **Both: uv-managed (primary) + portable single-exe (offline fallback).** Build the uv-managed installer as the main path; keep PyCrucible's self-contained exe as a "portable/offline" download for air-gapped microscope PCs. | -| Windows installer | **NSIS** (.exe installer) — per-user, no-admin, Start-Menu + desktop shortcuts, Add/Remove entry, uninstaller. | -| Code signing | **Ship unsigned for now**; document SmartScreen/Gatekeeper click-through. Wire signing into CI later when certs are procured (Win EV, Apple Developer). | -| Update checking | **Startup check + manual, WIRED (2026-07-01).** `electron-updater` checks on launch (~5s delay) and via Help → Check for Updates…; `autoDownload=false` — user clicks Download, then Restart to Install. **Beta channel is now live** (superseding the original "deferred" decision): tag suffix (`-rc.N`/`-beta.N`/`-alpha.N`) determines `stable` vs `beta`, selectable in the update dialog, persisted to `~/.spyde/settings.json` + an Electron-side `update-channel.json`. See `electron/PACKAGING.md` "Auto-update + beta channel". | -| Offline bundle | In scope (Phase 4) — the "portable single-exe" doubles as the offline path; optionally also a uv-cache-seeded bundle. | - -### Resulting build matrix -- **uv-managed installer** (primary): Win NSIS `.exe`, mac `.dmg` (.app launcher - + uv), Linux AppImage/`.deb`. Tiny; first run / updates = `uv sync`. -- **Portable single-exe** (fallback): the existing PyCrucible artifacts, renamed - `SpyDE-portable-` in the release. - -## 7. Recommended starting point -Given "ship unsigned now" + "uv-managed", the lowest-risk first slice is -**Phase 0 + Phase 1 + the update *check*** (not yet auto-apply): -single-source version, `_build_info`, `gpu_setup.py` + GPU Status dialog, and a -Help → Check for Updates that compares `__version__` to the GitHub latest release -and links the download. That delivers visible value (version hygiene, GPU -diagnostics, update awareness) with no installer/signing dependencies, and the -NSIS installer + auto-apply land in Phases 2–3. diff --git a/NEURAL_INTEGRATION_PLAN.md b/NEURAL_INTEGRATION_PLAN.md deleted file mode 100644 index 90ba836e..00000000 --- a/NEURAL_INTEGRATION_PLAN.md +++ /dev/null @@ -1,256 +0,0 @@ -# Neural (SpotUNet) Integration Plan - -Status of the neural disk-detector integration and the phased plan to make it -complete. Written 2026-07-15 after an audit of `spyde/models/`, -`spyde/actions/find_vectors*`, the wizard UI, and the packaging config. - -> **Status 2026-07-16: Phases 0 and 1 are IMPLEMENTED** (G1, G2, G3, G7 fixed; -> the concurrent-download half of G5 fixed via client-side `ensure_local`). -> **Phase 2 partially done**: the SPYDE_FV_GPU policy now governs the neural -> path (neural unset-default "all" = today's behaviour; flipping it needs the -> multi-worker benchmark, still open) and `load_model` uses the CUDA→MPS→CPU -> chain with a load-time MPS smoke test (needs a Mac to validate for real). -> **Phase 3 partially done**: `persistence` is wired end-to-end behind a -> default-off "Neighbor refine" wizard toggle (refine.py is live code now); -> the eval promotion gate is still open. Phase 4 not started. -> Verified: pytest (`test_model_registry.py`, `test_find_vectors_neural.py`) -> + Playwright (`fv_neural_calibration.spec.ts` — screenshots show High-pass σ -> auto-calibrating 12→4 on si-grains, live preview re-tuning, model-list -> refresh, and the vectors window opening). Full migrated suite green. - -## Where we are - -The core integration is in good shape: - -- `spyde/models/` is a self-contained vendored copy of the yoloDiffraction - detector (U-Net + preprocess + GPU decode + refine), so SpyDE ships without - the research repo. -- The **model registry** (`spyde/models/registry.py`) merges bundled < user - manifests, caches loaded models, and falls back to the bundled default on any - failure — the wizard can never crash offline. Two bundled models ship - (`spotunet-production-v2` default). -- **"neural" is the default find-vectors method**: wizard Model dropdown - (populated by `fv_models`), per-chunk batched torch forward pass on GPU with a - per-frame CPU fallback, beam-stop rejection and disk-mean intensity parity - with NXCORR/DoG. -- Wiring tests (`test_find_vectors_neural.py`, `test_neural_detect.py`) and a - real-scale benchmark (`benchmark_neural_spots.py`, sped_ag 13k patterns) - exist. `spyde/models/RELEASING.md` documents the model-upgrade workflow. - -## The gaps (audit findings) - -Ranked by how much they break a documented or advertised behaviour. - -### G1 — The remote-model upgrade path is inert (documented but cannot work) - -`RELEASING.md` path A ("ship a new model WITHOUT re-releasing SpyDE") relies on -two things that don't exist: - -- **`huggingface_hub` is not a dependency** — absent from `pyproject.toml` and - `uv.lock`. `registry._resolve_hf` and `refresh_remote_registry` import it - lazily and swallow the `ImportError`, so in a shipped app every remote - resolve/refresh silently no-ops and the bundled default is used forever. -- **`fv_refresh_models` does not exist.** `RELEASING.md:27` says the wizard's - Model-dropdown refresh calls it; there is no such action in - `actions/registry.py` and no refresh control in `FindVectorsWizard.tsx`. - `registry.refresh_remote_registry()` has zero callers in app code. - -Net effect: a newly trained model uploaded to HF can never reach a user. - -### G2 — Auto-calibration is dead code (the "parameter-free" promise) - -`calibrate_neural` (`find_vectors_neural.py:246`) — the one-shot optimiser for -`bg_sigma` (diffuse/beam-stop backgrounds) and a lowered threshold (faint-peak -data) — **is never called**. `orchestrate.py:281` reads `params["bg_sigma"]` -but nothing sets it (the wizard has no such field), so every run uses the -default `bg_sigma=12.0`, `thresh=0.3`. The registry notes advertise -"parameter-free with auto-calibration"; only the disk-size auto-scale actually -runs. - -### G3 — Preview/batch parameter divergence - -The live-preview dispatch `_find_peaks_single_frame` (`detectors.py:697`) -forwards `model_id` but **not `bg_sigma`** to the single-frame neural detector. -Once calibration (G2) is wired, the preview would silently use different -parameters than the batch run. Fix together with G2. - -### G4 — The refine/persistence stage is fully dead - -`models/refine.py` and the propose-then-refine machinery -(`_persistence_filter` / `_refine_block`, `find_vectors_neural.py:100-157`) -are vendored, parameterised (`persistence=` on `_neural_block` / -`_find_vectors_chunk_neural`) — and never enabled: `chunk.py:221` doesn't pass -`persistence`, no wizard control exists. Either wire it (it encodes real -physics: scan-neighbour persistence + Friedel) or delete it. - -### G5 — Neural GPU use ignores the cluster GPU policy - -The numba NXCORR path gates GPU use per worker (`_gpu_task_allowed`, -`SPYDE_FV_GPU`, default = worker "1" only) so CPU workers keep contributing. -`_neural_block` checks only `torch_gpu_device() is not None` — so on a -multi-process cluster **every worker builds a CUDA context and pushes batches -at the same GPU** (context VRAM × N workers, kernel contention). Relatedly, -each worker resolves the model itself: an HF-sourced `model_id` would be -concurrently downloaded by every worker into the same directory (G1 makes this -theoretical today, real after G1 ships). - -### G6 — No MPS: Macs run the model on CPU while taking the "GPU" branch - -`infer.load_model` picks `cuda`-else-`cpu` (`infer.py:27`), but -`torch_gpu_device()` supports MPS — so on Apple Silicon the batch path is taken -("GPU" branch) with a CPU-resident model. Works, but mislabeled and leaves the -Mac GPU idle. - -### G7 — Checkpoint loading is neither hardened nor verified - -`torch.load(ckpt_path, map_location=device)` (`infer.py:28`) without -`weights_only=True`: arbitrary-code-execution risk for downloaded checkpoints -on torch < 2.6, and a behaviour flip (potential load failure) on torch ≥ 2.6 -where the default changed. Registry entries carry no `sha256`, so a corrupted -or tampered download is undetectable. Downloads also have no progress/status -surfacing — a first use of a remote model blocks wherever `get_model` was -called. - -### G8 — No path from SpyDE usage back to training data - -`RELEASING.md` says the detector "is meant to be revised indefinitely", trained -in yoloDiffraction — but there is no way to get labelled examples out of SpyDE. -Every user scan where the detector under- or over-fires is training signal we -currently discard. - -## The plan - -Each phase is independently shippable. Phases 0–1 fix documented-but-broken -behaviour; 2–4 extend. Per CLAUDE.md discipline, anything touching the live -compute path changes default behaviour only with a benchmark on real-scale data -(`benchmark_neural_spots.py` on sped_ag) and gets an Electron/Playwright -screenshot verification, not just pytest. - -### Phase 0 — Make the model-upgrade loop real (G1, G7) — DONE 2026-07-15 - -1. Add `huggingface_hub` to `pyproject.toml` dependencies + re-lock (decision - 2026-07-15: hard dep, not an optional extra — it's a small pure-Python - package and the registry code already treats it as the download backend). -2. Implement **`fv_refresh_models`**: staged verb that runs - `registry.refresh_remote_registry()` via `run_on_worker` (network — never on - the main loop), then re-emits `fv_models`. Wizard: a small ↻ button beside - the Model dropdown + "checked — N models" status text. -3. **Resolve weights once, client-side**: `registry.ensure_local(model_id)` + - `is_cached(model_id)`, called in the `fv_open` preview worker and at the top - of the batch worker (`_start_batch._work`) before any compute is submitted; - a "downloading model…" status line is emitted only when the file isn't - cached. Workers then only ever read a locally-present file (also removes the - N-way concurrent-download race, G5b). Per-byte `emit_progress` can be added - later if model files grow beyond a few MB. -4. Harden loading: `torch.load(..., weights_only=True)` (verify both bundled - checkpoints load — they store plain state dicts + scalar hyperparams); - optional `sha256` field per registry entry, verified after download, with a - clear log + fallback-to-bundled on mismatch. -5. Tests: `fv_refresh_models` emits the payload (hub monkeypatched); merge + - checksum + `ensure_local` unit tests, all offline. E2E: wizard shows the - refresh control (screenshot). - -### Phase 1 — Wire auto-calibration end-to-end (G2, G3) — DONE 2026-07-15 - -1. When the wizard is opened with (or switched to) the neural method, run - `calibrate_neural` on a handful of sample frames — reuse the NavBlurCache - chunk plus a few frames spread across the scan — on a worker thread, - cancellable via the tree cancel registry (wizard close must kill it). -2. Emit `fv_calibration {bg_sigma, thresh, confidence, scale_factor}`; the - wizard shows the values as auto-filled (user-overridable) fields and includes - them in `params` for **both** preview and run. -3. Thread `bg_sigma` through `_find_peaks_single_frame` so preview == batch - (fixes G3 independently of the UI). -4. Stamp `model_id` + the effective calibrated params into the committed - vectors' provenance dict (mechanism already exists in `commit.py`). -5. Tests: calibration-dispatch parity unit test (preview and chunk fn receive - identical params); an e2e run on `load_test_data_si_grains` asserting the - calibration payload arrives and the run completes ("Found N diffraction - vectors"). - -Decision (2026-07-15): calibration **auto-runs on wizard-open** (parameter-free -is the selling point) with visible values + override; it's ~8 forward passes on -~4 frames. - -### Phase 2 — Runtime parity + platform reach (G5, G6) — PARTIAL 2026-07-16 - -1. ~~Wire the policy~~ DONE. The neural unset-default is **"4"** GPU-feeding - workers (user-picked starting point; their first "=2 beats all" A/B turned - out not to have applied the env var — the freeze cure was the sigma-0 - copy-elimination + backpressure work, and with GPU-ONLY dispatch the CPU - workers idle anyway, so more feeders are cheap). Consistent in - `_neural_block`'s `_gpu_task_allowed` AND orchestrate's lane split - (threaded through `dispatch_chunks`' mid-run lane refresh; chunks pinned - HARD — `allow_other_workers=False` — since torch-CPU inference is 10-50× - slower and soft placement leaks chunks there). NXCORR keeps "one" - (serialising kernels). `SPYDE_FV_GPU=one/N/all/off` overrides; a proper - measured default is still worth a benchmark pass. -2. ~~MPS~~ DONE (code): `load_model` picks cuda → mps → cpu and smoke-tests one - forward on MPS at load, degrading to CPU if an op is unsupported. OPEN: - validate on real Apple-Silicon hardware. -3. OPEN (benchmark-gated): move `detect_batch`'s per-frame CPU zoom+normalize - loop into batched torch ops if the CPU preprocess dominates GPU batch time. - -### Phase 3 — Quality: propose-then-refine + a promotion gate (G4) — PARTIAL 2026-07-16 - -1. ~~Wire persistence~~ DONE: `persistence` flows wizard → `_coerce` → - orchestrate → chunk → `_refine_block` (refine.py is live code now), behind - the default-off "Neighbor refine" checkbox (neural only, batch-only — the - preview has no scan neighbours). NB `refine()` normalises the col-2 value - internally, so the raw-intensity column works as the relative-confidence - term; the hard `min_persist=0.5` floor does the real filtering. -2. OPEN: add an **eval mode** to `benchmark_neural_spots.py`: precision/recall - against synthetic ground truth + persistence-consensus pseudo-labels on - sped_ag. Reference it from `RELEASING.md` as the required promotion gate — - and use it to decide whether "Neighbor refine" should default ON. - -### Phase 4 — Training-data flywheel (G8) - -1. **"Export training data"** action on a vectors result: sample N frames + - their detected labels into the yoloDiffraction dataset format (npz/json - patches + scan fingerprint/metadata). v1 needs no UI beyond a menu entry and - closes the loop RELEASING.md assumes. -2. Later: in-app correction (add/remove spots on the vector overlay) exporting - hard examples — the highest-value labels are exactly the frames users had to - fix. - -## UX simplification (user feedback 2026-07-16) — DONE - -The neural pane was over-parameterised ("too many options"). Decisions, all -implemented: - -- **Neural controls = Spot size + Threshold only.** Spot size (px radius, - auto-seeded from the pattern's LoG estimate) is THE scale knob: it overrides - the model's autocorrelation disk-size estimate (`spot_diameter = 2·radius` → - `scale_to_canonical(diameter=…)`, still upsample-only), drives the NMS - min-distance (~radius/2) and the preview marker radius. `spot_radius=0` - (scripting/api default) keeps the model's own auto-estimate. -- **Nav blur is NEVER applied for neural** (`_coerce` forces `sigma=0` — the - net is trained on single frames) **and defaults to 0 for NXCORR/DoG too** - (slider remains for those methods; `toolbars.yaml` default updated in - lockstep). -- **High-pass (bg σ) is fully automatic** — no control; `fv_calibration` is - adopted invisibly (threshold adoption still respects a user override). -- **Subpixel checkbox hidden for neural** (the net always emits subpixel — it - was a no-op). -- **Every dropdown is themed like the File/Examples/Help menus**: new - `Dropdown.tsx` (menubar palette; NOT a native `