docs(presentations): "SpyDE — an overview" — a 12-minute talk, authored in SpyDE - #104
Open
CSSFrancis wants to merge 19 commits into
Open
docs(presentations): "SpyDE — an overview" — a 12-minute talk, authored in SpyDE#104CSSFrancis wants to merge 19 commits into
CSSFrancis wants to merge 19 commits into
Conversation
…SpyDE A ~12-minute conference talk about SpyDE, built AS a SpyDE presentation document rather than a PDF — dogfooding the Report Builder's Present mode. doc/presentations/spyde-overview.spyde-report the deck (18 slides, 499 KB) doc/presentations/build_spyde_overview.py the generator (the real source) doc/presentations/media/ real app screenshots it embeds doc/presentations/README.md how to open, edit, re-capture The deck covers what SpyDE is, HyperSpy as the data model, the signal tree, the expansion packages (pyxem core; exspy/kikuchipy/atomap extras; orix transitive), anyplotlib, the Electron/Python split, the live-navigator and GPU work, and the report builder itself. Two "Active development" slides are left as OPEN placeholders, and every unverifiable claim is marked [TODO: ...] in the slide or its speaker notes rather than guessed at. Slides are markdown/image/split cells only — no live figure bindings — so the deck opens standalone with no data loaded. Each slide carries speaker notes and a time budget; the generator prints the total so the talk stays in its slot. Screenshots come from the real app via electron/tests/talk_screenshots.spec.ts (a capture run). electron/tests/talk_present.spec.ts opens the committed deck in the app, pages through all 18 slides in Present mode, and fails on a slide that renders no text or overflows — all 18 verified visually.
CSSFrancis
force-pushed
the
docs/spyde-presentation
branch
from
July 31, 2026 14:11
1aa7cac to
acf20e0
Compare
…t, a theme
The deck read like a feature tour written by someone who had never given the
talk: on-slide [TODO] markers, two slides that were deliberately blank, and no
answer to the question a conference audience actually asks first, which is who
is this person and why did they build it.
WHERE IT COMES FROM is now slide 2, because the credibility argument is
biographical: a PhD measuring disorder in glasses with 4D-STEM, then HyperSpy
and pyxem maintenance because the analysis did not exist yet, then Direct
Electron and the same complaint from the detector side.
OPEN, OR IT ISN'T REPRODUCIBLE and DIRECT ELECTRON'S BET ON OPEN SOURCE are the
two slides the old deck was missing entirely. The first is the Apple-vs-Linux
framing — polished-and-closed against readable-and-still-running-in-ten-years —
landing on the point that a number you cannot re-derive is an anecdote. The
second says plainly that DE funds this and gives it away, GPL-3.0, for any
detector, with the work going upstream. Both are positions, so they are stated
as positions rather than implied.
ANYPLOTLIB gets two slides instead of half of one: what it is (matplotlib's OO
API rendered in the browser, deliberately light, no matplotlib required) and
what it is FOR — a bridge that runs in Jupyter, in Electron, behind deapi, as a
HyperSpy backend, and as a self-contained HTML export. The HyperSpy backend is
labelled a goal, not a feature, because it is one.
ROADMAP is quantem: PyTorch-native ptychography and tomography, which makes the
same bet SpyDE's GPU paths already make. ACTIVE DEVELOPMENT is now the real tree
as of 0.3.0 rather than four empty bullets.
A DECK THEME, using the front-matter support that landed on main: the app's own
dark palette (so the embedded screenshots aren't framed in bright boxes), a
footer bar carrying name / email / affiliation / slide number, and the SpyDE
mark embedded as a data: URL so the file survives being emailed. Title cards
carry no footer.
NOTHING ON A SLIDE IS A PLACEHOLDER any more. Everything still to be decided
moved into the speaker notes, where a projector can't show it to the room.
SPEAKER is one table at the top of the generator — venue and date included, and
a blank one drops its line instead of leaving a dangling separator.
Held to the slot: 21 slides, 745 s = 12.4 min against the old 18 / 12.3, by
merging the data-model and tree slides and trimming the engineering ones. The
build now ASSERTS the budget rather than only printing it.
Two things found by looking rather than by asserting:
* the repo-root path was one dirname short, so the logo lookup silently
missed and the theme serialized with no logo. The import worked anyway
because the script is normally run from the repo root.
* present-footer matches EVERY mounted slide, not the active one — Present
mode keeps them all in the DOM. The first version of the assertion passed
for the wrong reason. It is now scoped to the stage, plus a count check
that pins footers to exactly the non-title slides.
Verified in the real app: talk_present.spec.ts pages all 21 slides, asserts the
themed background / accent / heading colour / footer / logo, and every shot was
looked at. 76 report tests still pass.
CSSFrancis
changed the title
…NG that lands where you drop it
Three things, all reported from real use.
SPLIT LAYOUTS — the model has had four (`text-left`, `text-right`, `text-top`,
`text-bottom`) and the EDITOR has offered all four in its layout picker, but
Present mode tested the value with a bare `!== 'text-right'`. So a stacked split
round-tripped through the document perfectly and then rendered SIDE BY SIDE: the
editor offered a layout the deck could not show. Present now resolves the same
two booleans the editor does (`stacked`, `textFirst`) through one shared helper,
and a stacked split gets one column with the text row sized to its content —
`1fr 1fr` there would hand a two-line caption half the slide.
The HTML export already handled all four correctly; only its docstring (and
`report_set_split_layout`'s) still claimed two. Fixed so they stop misleading.
The GENERATOR gated on the left/right pair too, so `text-top` fell through to the
full-width path — a markdown cell plus a detached image cell, with no error to
say why. It now accepts `_SPLIT_LAYOUTS` and RAISES on an unrecognised value,
because the silent fallback is the whole complaint.
TEXT SLIDES FILL THE STAGE. A heading and four bullets rendered at prose size,
vertically centred in a 60rem column — a small block adrift in a large dark
rectangle on a projector. Content slides now anchor their heading at the TOP
(centring is right for a title card, but on a content slide it floats the
heading at a height that depends on how much text follows, so consecutive slides
visibly jump), and a text-only slide scales its type by CONTENT LENGTH: sparse
gets the large tier, dense keeps prose size, which is what stops it overflowing
into the pager. Scaled by length rather than by measuring — deterministic, no
reflow loop. The column widens with the type, because `max-width` is in `rem`
and leaving it at 60rem while the body goes 18px to 27px would SHORTEN the
measure to ~45 characters and undo the point.
A DROPPED PNG NOW FILLS THE SLOT. Dragging an image onto a split cell's figure
side embedded it BELOW instead. Both drop zones gated on `isComposeDrag`, which
only tests the figure/window pill mimes — a file drag carries `Files`, failed
the test, bubbled to the sidebar body, and was appended as a new image cell. And
there was no verb that COULD fill a slot that already existed.
* `report_set_cell_image` — the image counterpart of report_set_split_figure.
A split takes the photo as its figure side, text and layout untouched. A
figure PLACEHOLDER converts IN PLACE to an image cell: same cell id, so the
slide attributes riding on a slide's first cell (break, kind, style, speaker
notes) survive — re-creating the cell would silently drop the notes and
merge the slide into the previous one. There is a test for exactly that.
* The image-file helpers move to `components/imageDrop.ts`. While they were
private to ReportSidebar, only the sidebar BODY could recognise a dropped
PNG, which is the root of the bug. The `items` fallback in `hasImageFiles`
is load-bearing: during dragover the browser does not expose `files` yet, so
a target testing `files` alone never calls preventDefault and therefore
never RECEIVES the drop.
VERIFICATION. `present_layout.spec.ts` drives the real app: it seeds a deck
through the real backend verbs, then asserts a stacked split renders as ONE grid
column with the text genuinely above the figure, that a sparse slide's computed
font size is >1.25x a dense one's, and that the dense slide still fits its stage.
The fixture image is 480x300, not 1x1 — a single pixel satisfies every assertion
while showing nothing, so it would prove the slot was filled without proving
anything was DRAWN in it. Screenshots looked at. 14 new backend tests; 570
report tests pass; tsc clean.
talk_present.spec.ts no longer hard-codes the slide count or the theme hexes. It
asserts against a deck that is EDITED as well as generated — that is the point of
shipping a talk as a SpyDE file — so a fixed 21 failed with a different number
every run while the file was being worked on. The count is now discovered by
waiting for it to settle, and the theme is checked RELATIVELY (the painted
background must equal the theme variable the deck publishes), which is the
invariant that actually broke before and holds for any palette.
Note: doc/presentations/spyde-overview.spyde-report is deliberately NOT in this
commit. It currently holds hand-edits made in the app, not generator output.
A figure cell persisted as a RECIPE (figures/<id>.yaml) plus a baked still (assets/<id>.png). The recipe points at the SOURCE signal, so reopening a report without that data loaded gave you a picture: you could look at the figure but not pan it, zoom it, or touch a widget. The odd part is how close it already was — figure_builder.build_figure wants a spec AND a per-layer snapshot map, the spec already round-tripped, and only the pixels were missing. data/<id>.npz supplies them. On open, a cell whose sources don't resolve but whose pixels WERE saved is rebuilt into a real anyplotlib figure and marked DETACHED: every interaction works, and the one thing it cannot do is refresh from a signal that isn't there — so that is the one affordance the UI withholds (the ⟳ button becomes a non-interactive badge explaining why). `data_detached` is deliberately NOT `data_offline`; offline still means "there are no pixels, show the PNG", and that path is unchanged and still tested. Purely additive on disk. A reader that doesn't know about data/ ignores it, and a report written before this loads exactly as it did — there is a test for each direction. read_report keeps its 2-tuple (unpacked at ~20 call sites); the pixels come back through a sibling read_report_snapshots rather than churning all of them to carry something most don't want. np.load runs with allow_pickle=False and object arrays are dropped at pack time rather than pickled: a report is a file people email each other, and opening one must never be a code-execution path. A corrupt blob degrades to "static PNG", never to a failed open. Per-cell cap of 32 MB, over which the cell saves without data and reopens as the still — a multi-panel figure of 4096² float64 layers would otherwise quietly add hundreds of MB to a file meant to be shared. ALSO — the dead box in the screenshot that started this. A split cell whose figure side was offline fell through to "rendering…" and sat there forever: ReportSplitCell tested `cell.image` and `fig` but never `cell.png`, so the baked still the backend was already shipping went unread. Present mode has always checked both; only the editor didn't. AND — split slides now fill the stage like text slides do. The fill tiers were gated on `visualCells === 0`, so a split kept prose sizing; its text side is its own column, so scaling it cannot crowd the picture. `splitText` also moves from alignSelf:center to start — the row is stretched to the stage, so centring floated the heading at a height that depended on how many bullets followed and put it out of line with the top of the picture beside it. A slide with a full-width figure still does NOT qualify: there the text shares the vertical budget with the visual and bigger type would push it off the stage. Two existing tests changed contract rather than broke: a figure whose source is gone, and a scene3d cell whose orientation result is gone, now reopen DETACHED instead of offline. Both were asserting the fallback this change exists to replace, and a spinnable 3-D scene beats a picture of one. The offline-PNG path keeps explicit coverage via a new sibling test that strips data/ from the zip first, so nothing lost its guard. 16 new backend tests; 702 report/movie tests pass; tsc clean; both present specs pass against the real app.
…yle conflicts
THE DASHBOARD. Two different backend messages dispatch the renderer's READY
action: `ready` (the stdin loop is up — carries NO dashboard field) and
`dask_ready` (the cluster is up — carries the URL). The reducer wrote
`dashboardUrl: action.dashboardUrl ?? null`, so a `ready` arriving after a
`dask_ready` ERASED the URL that had just been delivered. With it null the File
menu item is `disabled: !state.dashboardUrl` and DaskMonitor renders no button
at all, so there is no way left to open the dashboard.
Ordering is not fixed and not something either side guarantees: `ready` is
emitted in app.py only after _prewarm_io() and prewarm_torch_cuda(), which cost
seconds, while the cluster comes up on a background thread and emits from there.
I did not capture a log proving which order this machine hits — the fix does not
depend on it. A READY that carries no URL carries no information ABOUT the URL,
so it must not overwrite one; that is correct under every interleaving.
THE STYLE WARNING. "Removing a style property during rerender (borderColor) when
a conflicting property is set (border)" — React then drops one of the two and
the element renders with the wrong border. Always the same base+modifier spread:
seg: { border: '1px solid #313244' } // base — SHORTHAND
segOpen: { borderColor: '#45475a' } // modifier — LONGHAND
style={{ ...S.seg, ...(open ? S.segOpen : {}) }}
DaskMonitor's own toggle was one of them, which is why the two turned up
together — but they are independent bugs and the warning was not what broke the
dashboard. There were FIFTEEN across the app (compose drop zones, the console
input, dropdowns, the movie editor, present-mode icon buttons, figure/movie drop
zones, sidebar cards, slide thumbnails, window overlays); each fires on every
toggle. All converted to the full shorthand.
A node:test guard now scans the renderer sources for the pattern, because it is
a property of the style objects and needs no DOM. It earned itself immediately:
it found a FIFTEENTH instance my own grep had missed, ReportMovieCell's
`dropZoneReady`, which uses borderStyle rather than borderColor. It lives in
src/main/ — it only READS the renderer tree, and the renderer tsconfig has no
node types.
Verified with a real cluster: dask_dashboard_link.spec.ts drives the app,
opens the monitor and asserts the button is there. 15 unit tests pass; both
tsconfigs clean.
…ive figure
A picture already in a report could not be swapped. An image cell carried only
its REORDER drag wiring, and a split cell's drop zone existed solely while the
figure side was EMPTY — so a file dragged onto an existing picture failed both
targets' tests, bubbled to the sidebar body, and was APPENDED as a new cell
below the one it was aimed at. Changing a picture meant deleting the cell and
re-adding it, which loses the caption, the display width, and — on a slide's
first cell — the slide break and the speaker notes.
Both sources now land on the same target:
* an image FILE from the OS -> report_set_cell_image
* a figure/window PILL -> report_add_figure {at_cell}
Both verbs address the EXISTING cell id, so the cell keeps its identity and
everything hanging off it. Two backend gaps had to close for that:
* report_set_cell_image refused an `image` cell ("no figure slot to fill").
It now swaps the bytes in place — same cell, same caption, same slide role.
* report_add_figure's at_cell knew `figure` and `split` targets but not
`image`, so it fell through to the APPEND branch: the photo stayed put and
an unrelated figure landed underneath it. An image target now converts IN
PLACE to a figure cell, which is the only way the slide attributes riding on
it can survive — there is no "image cell holding a figure" in the model.
The renderer side is one shared hook (useReplaceDrop) rather than two copies,
because the split cell and the image cell want identical behaviour and the
earlier duplication is what let the split's FILLED branch get missed. It
stopPropagations so a drop never also reaches the sidebar body — that
double-handling is the append bug itself.
VERIFIED IN THE APP, not just headlessly: report_replace_image.spec.ts
synthesizes a real DataTransfer carrying a real File and dispatches
dragenter/dragover/drop at the element, so the handlers under test are the ones
a user hits. It asserts the SAME cell shows a DIFFERENT image, that no second
cell appeared, and that the caption survived. Screenshots looked at.
Scope, stated plainly: the FILE path is driven end-to-end in the app for both an
image cell and a split's filled photo side. The PILL path is covered by backend
tests (report_add_figure at_cell on an image cell, twice) and shares the same
renderer handler, but is not driven with a synthesized pill drag. A filled
FIGURE cell still has no file-drop target either — its centre zone replaces with
another figure, and a dropped file there still falls through to the body. Both
are the same shape as what this fixes and are worth doing next.
4 new backend tests; 706 report/movie tests pass; 15 unit tests; tsc clean.
…it slide
Turning a text slide into a text-beside-picture slide meant rebuilding it: add a
split block, retype or paste the prose, delete the original. That loses whatever
was riding on the cell — and on a slide's FIRST cell that is the slide break, the
kind/style and the speaker notes, so the "rebuild" also quietly merged the slide
into the one above it.
Dropping a figure/window pill or an image file straight onto the text now
converts the cell IN PLACE: the prose it already had becomes the text side and
the picture fills the other. Same cell id, so the slide stays ONE slide and keeps
its notes — there is a test for exactly that, because it is the part a rebuild
gets wrong.
Both verbs learned the markdown target, mirroring each other:
* report_set_cell_image — a dropped FILE
* report_add_figure {at_cell} — a dropped WINDOW (a live figure side)
Each takes an optional `layout` (default text-left), so the caller can choose
which side the picture lands on without a follow-up round trip.
The renderer reuses useReplaceDrop — the same hook the image cell and the split's
photo side use, since the verbs already address the cell by id and the backend
decides what the conversion means. The markdown cell is the one place where two
kinds of drag land on the SAME element and must not fight: a pill/file converts
the cell, a cell-REORDER drag moves it. `replace` claims only the former (it
preventDefaults when it does), so anything it declines falls through to the
reorder wiring untouched.
The old "a slot-less cell must report why" test now uses a movie cell, since
markdown is a legitimate target — the guard it was written for is still there,
just not aimed at markdown any more.
Verified in the app: report_replace_image.spec.ts drops a real File on a real
text cell and asserts the markdown cell BECAME a split (by testid), that the
figure side draws, that the prose came with it, and that no cell was added.
Screenshot looked at — prose left, picture right, still Slide 1 with its notes
chip. 5 new backend tests; 711 report/movie tests pass; tsc clean.
Still open, same shape as before: a filled FIGURE cell has no file-drop target
(its centre zone replaces with another figure, but a dropped file falls through
to the body), and the PILL path is covered by backend tests rather than driven
with a synthesized pill drag.
The last of the drop gaps. A filled figure cell's centre zone replaced it with another FIGURE, but a dropped FILE fell through to the sidebar body and was appended as a new image cell below. The verb already handled it — report_set_cell_image accepts a `figure` target and converts it to a photo cell, tearing down the live window — so this was almost entirely a renderer problem, and a specific one: a report figure is an OUT-OF-PROCESS IFRAME that swallows drag events over itself. The cell only ever sees them through a transparent shield, and that shield was gated on `dragKind`, which is set at dragstart of an IN-APP pill. An OS file drag never sets it, so nothing covered the iframe, the iframe ate the dragover, and no drop event reached React at all. `useFileDragActive` supplies the missing signal. It watches window-level `dragover` and refreshes a short timer rather than waiting for `dragend`, because for a drag originating OUTSIDE the page dragend fires on the source — which is not in this document — and never arrives. The timer lapsing is the only reliable "the drag has gone". The docstring claimed report_set_cell_image took a figure PLACEHOLDER; it takes either, and now says so. VERIFICATION, honestly: the backend half is pinned by a new test — the figure cell converts to a photo cell, its spec is cleared, and its live window and snapshots are torn down rather than orphaned. The RENDERER half is NOT verified end-to-end. The spec I wrote for it resolved a window id of 0 from the first breadcrumb (the NAVIGATOR window, not the signal one) and report_add_figure then produced no figure cell with no backend error — so it was measuring nothing, and a test that passes for the wrong reason is worse than an absent one. It is removed, with a comment in the spec saying exactly what is uncovered and why. Getting a real painted signal window into that fixture is the remaining work. The other three cases in that spec were made self-sufficient about the sidebar being open — run alone, they depended on the first test having clicked it. 1 new backend test; 712 report/movie tests pass; 3 e2e cases pass; tsc clean.
DIAGNOSTICS ONLY — no behaviour change, nothing recovers yet.
Symptom: close and reopen a Mac laptop and every plot window is gone from the
workspace, but the app still works and you just re-load the data.
Two facts narrow that a long way, and both argue against the obvious suspects.
The Python backend is NEVER respawned — runner.ts sets `proc = null` on close and
every later sendAction no-ops — so if it had died, re-loading data afterwards
would be impossible. It isn't, so the backend is ALIVE. And the workspace's
window list lives ONLY in the renderer's React state (SpyDEContext `windows:
Map`); nothing persists it or rebuilds it from the backend. So the renderer lost
its state while Python kept running.
There are only two ways that happens, and these listeners tell them apart:
* the renderer PROCESS was recreated — render-process-gone, or a navigation the
app never asked for;
* or it survived and something reset the state, in which case none of these
fire and the answer is in the renderer.
child-process-gone is here for the Mac case specifically: the GPU / utility
processes are the usual casualty across a lid-close, and a dead GPU process takes
the WebGPU device with it — Chromium may then recreate the renderer to recover,
which is exactly how in-memory React state vanishes while Python lives on.
powerMonitor is logged purely to timestamp the sleep boundary so the log reads as
a story. Nothing acts on it. There is currently NO powerMonitor and NO
render-process-gone handling anywhere in the app, which is why this has been
invisible.
To reproduce: run from source, close the lid, reopen, and send the main-process
console output. The [spyde lifecycle ...] lines around power:resume are the
answer.
The workspace vanishing across a laptop sleep is not a crash: the Python backend survives (runner.ts never respawns it, and sendAction would no-op forever after a death — but re-loading data works), and the window list lives ONLY in the renderer's in-memory React state with nothing able to rebuild it. The plan is a consistency fix rather than new architecture. The Report Builder already does exactly the right thing — report_state is a full-document snapshot the renderer mirrors, 41 emit_state() call sites — and the window layer never got it: it accumulates from incremental messages with no snapshot and no way for the renderer to ask for one. Phased so the guesswork is isolated: a probe answers whether the figure HTML files and the awi_state replay stash survive a renderer loss, which is what decides whether Phase 2 is cheap or expensive. I deliberately did not estimate past that probe. Three decisions listed for review — geometry restore vs fresh layout, how far 'authoritative' should go, and automatic vs explicit trigger — plus what is out of scope (disk persistence, Dask reconnect, anything in the nav read path).
I asserted the cluster survived 'which is why re-loading data works'. That was wrong, and neither symptom supports it. Re-loading data proves nothing about the cluster: _await_dask gates on _dask_ready, a threading.Event LATCH set once at first startup and cleared in exactly one place (compute_config.py, on an explicit user-driven restart). Nothing clears it when a cluster dies, so a load sails through the gate and runs against a dead client indistinguishably from a live one. The dashboard disappearing proves nothing either: dashboardUrl lives in the SAME SpyDEContext React state as the window list, so whatever loses the workspace loses the link with it. That leaves two independent unknowns rather than one, and surfaces a latent bug worth more than the original: NOTHING in SpyDE detects cluster death — no liveness check, no re-arm of the gate — so after any cluster loss the app keeps accepting work against a dead client instead of waiting, restarting or saying so. A vanished window is obvious; a silently dead compute backend is not. Added discriminators that don't route through the state we know gets lost (look for the worker processes; probe from the backend; run a compute rather than a load), and kept the cluster workstream explicitly OUT of the resync scope rather than quietly merging them.
Requested follow-on to the section 8 correction. Kept SEPARATE from the resync work — different trigger, different code, different failure mode — and shippable on its own. The useful finding: the gap is DETECTION, not recovery. compute_config.py's restart path already does exactly the right three things (clear _dask_ready so loads wait rather than racing a dead client, dask_manager.restart(), let _on_dask_ready re-open the gate). It is proven code with one trigger: a user changing compute settings. Nothing calls it when a cluster dies on its own. So the work is notice-then-reuse, not build. Detection options compared; DaskStatsSampler flagged as the one worth checking first, since it is already talking to the cluster on a cadence for the StatusBar HUD and its failures may BE the signal for free. Phased so the smallest useful diff lands early: re-arming the gate alone converts 'silently computes against a dead client' into 'waits, then fails with a real message'. Traps recorded because they are what makes this risky rather than easy: a false positive restarts a healthy cluster and cancels in-flight work, a lost WORKER is not a lost CLUSTER (distributed replaces workers routinely), probes must stay off the asyncio main thread, auto-resubmit is refused outright, and threaded / SPYDE_NO_DASK mode has no cluster to be dead.
Two probes replace the plan's guesswork with numbers. resume_probe.spec.ts drives the real app. A lid-close can't be driven from CI, so it uses the faithful proxy — reload the renderer with data loaded, which destroys the in-memory React state exactly as a renderer-process death would while leaving Python and its cluster untouched. BEFORE windows=2 iframes=2 dashboardLink=true figureFiles=2 AFTER windows=0 iframes=0 dashboardLink=false figureFilesStillOnDisk=2/2 BACKEND exited=false logGrew=true RELOAD worked=true windows=2 The plan's premise is confirmed rather than assumed: windows and the dashboard link die together (same React state, which is why the link proves nothing about the cluster), Python survives, and re-loading data rebuilds everything. THE finding is the last column. The figure HTML files OUTLIVE the renderer, 2/2 — they are served over the spyde-fig:// protocol from the OS tmpdir and the MAIN process writes them. That answers the plan's open question 3.3-Q1 and puts Phase 2 on its cheap branch: a resync re-sends the same file_urls and the iframes remount from files already on disk. No figure re-render. test_dask_ready_latch.py characterises the load gate: _await_dask() is a pure Event read that consults no cluster; with the client torn out from under the session it STILL opens instantly, so a load runs against a dead client and cannot tell; and structurally there is exactly one _dask_ready.clear() (the user-driven restart) and no liveness path of any shape. Written so it FAILS when a detector is added — that is the point, not an accident. Assertions are deliberately narrow: only the backend surviving and the app staying usable are asserted. Window survival is RECORDED and loudly logged if it ever happens, because that would mean the plan is wrong, and a probe should be able to say so.
Observed directly on the failing machine, which upgrades the plan's load-bearing assumption from an inference to a measurement. Section 1 had reasoned it out from the code — runner.ts never respawns the backend, so a post-sleep load succeeding meant it could not have died. That is now corroborated on the real platform and the real failure, which also validates the Phase 1 proxy: reloading the renderer stands in soundly for this bug, since both agree on what lives and what dies. Recorded the distinction that survives this, because it is easy to lose: a RUNNING PYTHON PROCESS IS NOT A HEALTHY CLUSTER. The Dask workers are separate child processes and the scheduler its own endpoint, so 'the backend is alive' says nothing about them. Section 8's two unknowns collapse to one, not zero — and section 10.3 is precisely why the app cannot answer the survivor for us, since the latch makes a load succeed either way. The section 8.2 discriminators (look for the worker processes, probe from the backend rather than the UI, run a compute rather than a load) remain the only way to tell. Also fixed a duplicated 10.5 heading from the previous append.
…thesis is in doubt Real Mac lid-close: worker count unchanged, a genuine distributed compute worked, and neither render-process-gone nor child-process-gone fired. The cluster question is settled by the right test — a compute that round-trips through the scheduler rather than a load the section 10.3 latch waves through regardless. Section 9 drops in priority but does not go away: nothing detects cluster death from any cause, which is still a real latent bug, just not this one. The awkward part is the third result. Section 1 concluded the renderer lost its state and the natural mechanism was the process being recreated — and neither process-gone event fired, which argues against precisely that. If the renderer really did survive, something inside the app CLEARED the workspace, which is a different and probably much cheaper bug than this document scopes. So I have NOT marked this good to go. One control question decides it: did power:suspend / power:resume print at all? If they did, the finding is real and there is a root cause still to find. If nothing printed, the diagnostics were not live and the test is inconclusive — absence of evidence rather than evidence of absence, and those are not interchangeable. The resync work stands either way as the robustness fix (section 10.1 measured the workspace dying under a plain renderer reload). But if the renderer survives a sleep, resync would paper over a state-clearing bug that would still be there afterwards.
…chines NOT generator output — these are hand-edits made in the app, committed only so they survive a move to another computer. build_spyde_overview.py is therefore STALE relative to spyde-overview.spyde-report; regenerating would overwrite this work, which has already happened once. Decide which is the source of truth before running it again. spyde-overview.spyde-report 13 slides, "SpyDE — an overview2" 12.4 MB spyde-overview2.spyde-report 14 slides, "SpyDE — an overview2" 107 KB SpyDE_an_overview2.html an interactive HTML export 749 KB The size difference is not noise and is worth knowing: the 12.4 MB file was saved AFTER the detached-figures change (38bf08f) and carries data/<id>.npz — the figure pixels that make a reopened figure interactive instead of a flat PNG. The 107 KB file predates it. So the real-world cost of that feature on this deck is roughly 117x, well under the 32 MB per-cell cap but far more than I guessed when I set it. Two things follow. A deck meant for emailing may want the pixels dropped, and there is currently NO way to ask for that — the choice is all-or-nothing at save time. Worth a "save without figure data" option, or a smaller default cap.
power:resume printed, so the diagnostics were live and the absence of
render-process-gone / child-process-gone is a real negative. The renderer process
survived the sleep.
That matters more than it sounds. SpyDEProvider's state is a useReducer
initialised once and nothing in the reducer clears wholesale, so with
the process alive there are only two mechanisms left — and they are very
different jobs:
A. the page reloaded (would show as renderer-navigation /
renderer-did-finish-load near power:resume) — resync is the right fix;
B. the windows were never lost at all. MDIArea positions subwindows at
ABSOLUTE pixel coordinates derived from areaRef.clientWidth/clientHeight and
caches them in placedRef. A Mac lid-close routinely changes the display
configuration; if the area measures 0 or smaller during resume, cached
placements land outside the visible region. The windows would still be in
state, still streaming, just not where you can see them.
If B, this is a layout bug and the fix is re-clamping placements on an area-size
change — dramatically smaller than the resync, which would have papered over it.
The discriminator costs two seconds and needs no tooling: MDIArea keeps hidden
windows LISTED IN THE TOP BAR. If the bar still lists them, state is intact and
it is B. If the bar is empty too, it is A.
Phase 2 goes on hold. Not because resync is wrong — section 10.1 measured the
workspace dying under a plain renderer reload, so it stays a real gap — but
because building it now risks fixing the symptom while leaving the cause.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A ~12-minute conference talk about SpyDE, authored in SpyDE — a real
.spyde-reportpresentation document, not a PDF. It dogfoods the ReportBuilder's Present mode, and now its theme support.
To open it: launch SpyDE → Report sidebar (panel toggle, top right) →
Open → pick the file → Present.
→/←navigate,Stoggles thepresenter view,
Escexits.The deck uses only markdown / image / split cells — no live figure bindings — so
it opens standalone with no data loaded.
What the talk argues
The previous draft was a feature tour. It answered what SpyDE does well and
never answered the two questions a conference audience asks first — who is
this person and why should I trust the software with my data. The rewrite
puts both up front and keeps the engineering where it belongs, as evidence.
Where this comes from (slide 2) is the biographical argument: a PhD with
Paul Voyles at UW–Madison measuring disorder in glasses with 4D-STEM, then
HyperSpy and pyxem maintenance because the analysis did not exist yet, then
Direct Electron and the same complaint from the detector side. "SpyDE is what
those two jobs look like when you do them at the same time."
Open, or it isn't reproducible (slide 8) is the Apple-vs-Linux framing —
polished-and-closed against readable-and-still-running-in-ten-years. The point
it lands on: a number you cannot re-derive is an anecdote, and science needs the
property where you open the same file years later, run the same pipeline and get
the same answer. "SpyDE wants the polish of the first and the guarantees of the
second."
Direct Electron's bet on open source (slide 9) says the position plainly
rather than implying it: DE pays for this and gives it away, GPL-3.0, for any
detector, no paid tier and no licence server; the work goes upstream into
HyperSpy, pyxem and RosettaSciIO;
deapiandanyplotlibare MIT. "A camerais only as useful as what you can do with the data."
anyplotlib gets two slides where it had half of one. Slide 11 is what it is
— matplotlib's object-oriented API rendered in the browser, so pan/zoom/drag
never touch the kernel, and deliberately light (
anywidget,numpy,traitlets,colorcet, no matplotlib). Slide 12 is what it is for: abridge that runs in Jupyter Lab, in Electron, behind
deapi, and as aself-contained
save_html()export — with a HyperSpy interactive backendlisted as the goal, not a feature, because that is what it currently is.
Roadmap (slide 19) is quantem — the PyTorch-native toolkit for ptychographic
phase retrieval and HAADF tomography — framed as making the same bet SpyDE's own
GPU paths already make: put the whole problem on the device and batch it. The
integration goal is stated as a data-model question, not a rewrite.
"Inherit it, then give it back" — slide 6 now closes on the point that an
enormous amount of excellent work already exists and SpyDE's job is to put one
consistent interface on it and hand it back to the community free. Slide 20
ends on the same note.
The deck now carries a theme
Using the
theme:front-matter support that landed on main:#12121c, accent#89b4fa) — every screenshot inthis deck is a dark SpyDE window, so a light deck would frame each one in a
bright box
on every slide except the three title/section cards, which carry their own
attribution
data:URL, so the file still looks rightafter it is emailed to someone who has never had this repo on their disk
SPEAKERis now one table at the top of the generator — name, role, org, email,venue and date. A blank venue/date drops its line rather than leaving a
dangling separator, so the same file works for the next booking with one edit.
Slide-by-slide, with the time budget
Five more slides than the previous draft in about the same running time — the
data-model and signal-tree slides merged, and the engineering slides were
trimmed rather than dropped. The build now asserts the budget stays inside
the ~11–13.5 min slot instead of only printing it, so the talk cannot silently
drift out of its booking.
No placeholders left
The previous draft shipped
[TODO]markers on the slides and twodeliberately blank ones. Both are gone. Everything still to be decided moved
into the speaker notes, where a projector cannot show it to the room —
including which release to point people at, whether to mention the HyperSpy fork
upstreaming, and what to say if someone asks which machine the benchmarks ran
on.
Claims are still repo-checked rather than remembered:
SUPPORTED_EXTS(.csbincluded), the
eels/ebsd/atomsextras,requires-python = ">=3.10,<3.14",GPL-3.0-or-later, and the
spyde/{fitting,ebsd,atoms,drift,particles, spectroscopy}packages behind the Active-development slide.orixis stilldescribed as arriving transitively rather than as a declared dependency, and
lumispy is still absent because it is not a dependency.
One link correction: the deck previously pointed the audience at
directelectron.github.io/spydeandgithub.com/directelectron/spyde. Bothcurrently 404 — the live pair is
cssfrancis.github.io/spydeandgithub.com/CSSFrancis/spyde, which is what the deck now says. They areREPO_URL/DOCS_URLconstants at the top of the generator, so a move is oneedit.
How it was verified
electron/tests/talk_present.spec.tsruns against the real app: it opensthe committed deck through
report_open, enters Present mode, pages all 21slides, screenshots each, and fails a slide that renders no text or overflows
horizontally. It now also pins the theme, as computed styles — the failure
mode is a theme that round-trips through the front matter perfectly and then
loses to a hard-coded stylesheet rule:
--spyde-deck-accentcustom propertynon-title slides
h2inside slide markdown takes the themed colourPassing, and all 21 shots plus the presenter view were looked at. 76 report
tests still pass (
test_report_theme/_model/_present).Two defects came out of looking rather than out of the assertions:
dirnameshort, so the logo lookupsilently missed and the deck serialized with no logo. The
spydeimportworked anyway because the script is normally run from the repo root, which is
why it had never surfaced.
getByTestId('present-footer')matches every mounted slide — Presentmode keeps them all in the DOM — so the first version of the "title slide has
no footer" assertion passed for the wrong reason. It is now scoped to the
active stage, with a separate count check.
Housekeeping
The branch was 92 commits behind
mainand predated the deck-theme work, somainis merged in. No application source is touched — this PR is additive: thedeck, its generator, its screenshots, a README, and the two specs.