feat(seg): the fast segmentation engine, non-rigid drift, and a test suite that can fail

.github/workflows/build.yml

@@ -78,6 +78,19 @@ jobs:
       # DaskManager.shutdown() sleeping 0.5s per Session teardown; see
       # test_dask_shutdown_fast.py.
       - name: Run tests
+        # NUMBA_BOUNDSCHECK=1 on the legs where numba/llvmlite (0.65.1 AND
+        # 0.66.0) MISCOMPILES the particle kernels: cp310/cp311 wheels on every
+        # OS, and macOS on every python. Uninstrumented builds corrupt the heap
+        # (0xc0000374 / SIGABRT in hull/intensity/contours/instances — wherever
+        # the corruption is DETECTED, the corruptor is the compiled kernel);
+        # the minimal pair on a Windows py3.11 venv: cold compile crashes,
+        # cold compile with boundscheck passes 13/13 with ZERO bounds
+        # violations — the kernels' indexing is in-bounds, the instrumented
+        # codegen is what dodges the miscompile. The shipped app is unaffected
+        # (the payload pins Python 3.12, a clean leg). Remove when a fixed
+        # numba lands and these legs run clean without it.
+        env:
+          NUMBA_BOUNDSCHECK: ${{ (matrix.python-version == '3.10' || matrix.python-version == '3.11' || startsWith(matrix.os, 'macos')) && '1' || '0' }}
         run: uv run pytest spyde/tests/migrated ${{ (matrix.os == 'ubuntu-latest' && matrix.python-version == '3.12') && '--cov=spyde --cov-report=xml' || '' }}
 
       - name: Upload coverage to Codecov

benchmarks.md

@@ -1746,6 +1746,107 @@ next chunk so a ~30x-faster GPU lane and the CPU lane drain one pool and finish
 together — real work stealing the scheduler cannot do, and the reason this
 module exists. Only the unpinned lane changed.
 
+### Batch segmentation paused dask workers — the per-FRAME peak, not the graph (2026-08-01)
+
+Reported from a real run: workers pausing at 80% of a 9.24 GiB limit, restarting
+at 95%, and `Unmanaged memory: 6.47 GiB`.
+
+The graph was never the problem. `segment_movie` is a plain `map_blocks` over
+time chunks and nothing calls `.compute()` on the movie — it is genuinely
+embarrassingly parallel. What was wrong is that ONE unit of that parallel work
+is enormous. Measured, 4096² frame, classical engine:
+
+| stage | peak | time |
+|---|---|---|
+| input frame (float32) | 64 MB | — |
+| `_prepare` | 24 MB | 0.04 s |
+| threshold | 7 MB | 0.02 s |
+| **`split_instances` (watershed)** | **546 MB** | **4.15 s** |
+| **whole frame** | **852 MB** | 4.4 s |
+
+The cluster runs `threads_per_worker=4`, so that is **~3.4 GB of concurrent peak
+on one worker** before frames in flight or allocator fragmentation. And
+"unmanaged" is the correct label: the rasters are ours, inside the task, so dask
+can neither account for them nor spill them.
+
+The watershed is 64% of the peak because the distance transform, its smoothed
+copy, the marker labelling and the watershed each materialise a full-frame
+float32/int32 raster.
+
+**Fix: watershed each connected component in its own bbox.** This is EXACT, not
+an approximation — a component is surrounded by background by definition, so a
+1 px pad contains every pixel the distance transform, the markers and the
+watershed can depend on, and no watershed can flow between components that do
+not touch. It is the same shape `measure.py` already uses to trace contours.
+
+| 4096², 400 overlapping discs | peak | time |
+|---|---|---|
+| whole-frame | 538 MB | 1.59 s |
+| **per-component** | **272 MB** | **1.09 s** |
+
+**2x less memory and 1.5x faster.** Faster because the distance transform now
+runs over the particles' own bboxes instead of 16.7 M pixels of mostly
+background.
+
+**One behaviour change, and it is an improvement.** The counts differ on large
+frames (395 vs 388 above) because `_split_factor` decimates the whole-frame
+split geometry 4x at 4096² (the earlier 7.1 s -> 2.8 s optimisation), while a
+per-component crop is a few hundred pixels and so gets factor 1 — full
+resolution markers. At 1024², where neither route decimates, the two agree
+PIXEL FOR PIXEL, which is what the parity test pins. So the per-component route
+is more accurate as well as smaller; the decimation that bought the original
+speedup is simply no longer needed on this path.
+
+Gated at `_COMPONENT_ROUTE_PX` (4 MP): below it the per-crop bookkeeping costs
+more than the memory it saves.
+
+---
+
+## Segmentation preview on a LOW-CONTRAST frame (2026-08-02)
+
+Reported from a real in-situ movie: "14028 particles in this region", a solid
+green preview window, and 4.1 s per tune. Reproduced on a synthetic stand-in —
+noisy support film, 8 faint dark particles, 1024², `invert=True`, otsu.
+
+**Otsu has no bimodal histogram to find here, so no caret knob rescues it:**
+
+| settings | instances | coverage | block-ANY coverage at 1/4 | time |
+|---|---|---|---|---|
+| defaults (`min_size=20`, watershed) | 4873 | 39.0% | 51.1% | 810 ms |
+| `min_size=200` | 208 | 14.4% | 16.5% | 595 ms |
+| `min_size=2000` | 17 | 7.2% | 7.9% | 577 ms |
+| watershed off | 751 | 39.8% | 52.3% | 67 ms |
+| `gaussian=2` | 431 | 52.7% | 55.0% | 392 ms |
+| rolling ball 64 + `gaussian=2` | 2280 | 26.4% | 39.4% | 11191 ms |
+| `gaussian=2` + `min_size=200` + no watershed | 8 | 52.4% | 54.2% | 101 ms |
+
+The last row is why `_threshold_failed` tests count AND coverage: 8 instances
+looks like the 8 real particles, but at 52% coverage those 8 bodies are the
+film. And note the block-ANY column — the overview reduction the tiled overlay
+uses turns 39% coverage into 51%, so a shattered frame composites into a sheet.
+
+**Where the preview's time goes** (same frame, warm):
+
+| stage | over-segmented (n=4873) | filtered (n=17) |
+|---|---|---|
+| `segment_frame` (threshold + watershed + filter) | 706 ms | 569 ms |
+| `measure_frame` (props + contours) | 647 ms | 62 ms |
+| **total** | **1353 ms** | **631 ms** |
+
+`watershed=True` is 700 ms of that vs 61 ms off — the split itself is ~639 ms
+and is inherent to a mask covering 39% of the frame, so it is NOT the part to
+optimise; the fix is to stop producing such a mask.
+
+The contours are, though: they are the bulk of `measure_frame` at high instance
+counts and are thrown away above the overlay's draw cap. `want_contours=False`
+takes the preview **1513 -> 969 ms (36% faster)** at n=4873 with the measured
+rows **bit-identical** (`np.array_equal`), so the count, histogram, median and
+confidence filter are unaffected. The saving scales with instance count, so the
+reported 14028-instance frame gains proportionally more.
+
+Reproduce in the app: `load_test_data_particles {noise: 0.35, size: [1200,1200]}`
+(`seg_oversegment.spec.ts`). At the default `noise=0.015` the fixture is clean,
+a global threshold works on it, and none of this is visible.
 
 ## Apple-MPS for the 0.3.0 compute paths (2026-07-31)
 

electron/src/renderer/src/App.tsx

@@ -6,6 +6,7 @@ import { ReportSidebar } from './components/ReportSidebar'
 import { ConsoleBar } from './components/ConsoleBar'
 import { StatusBar } from './components/StatusBar'
 import { LogPanel } from './components/LogPanel'
+import { BottomDock } from './components/BottomDock'
 import { Tour } from './components/Tour'
 import { NavShapeGate } from './components/NavShapeGate'
 import { StackGate } from './components/StackGate'
@@ -65,6 +66,11 @@ export function App() {
           {sidebarOpen && <PlotControlDock />}
           {reportOpen && <ReportSidebar />}
         </div>
+        {/* Table dock above the log: a data surface belongs next to the plots,
+            diagnostics below it. Both are flexShrink:0 siblings of the body —
+            BottomDock's own maxHeight:50% is what stops the pair starving the
+            MDI area. Visibility comes from the context (View menu / StatusBar). */}
+        <BottomDock />
         <LogPanel open={logOpen} onClose={() => setLogOpen(false)} />
         <ConsoleBar />
         <StatusBar logOpen={logOpen} onToggleLog={() => setLogOpen(v => !v)} />