Operate: navigable steps, position-aware instruments, and the half-frame fix#100
Merged
Merged
Conversation
Six call sites carried `if width > height * 2: take the left half`, a guess at splitting a stitched dual-camera frame. This rig is single-camera and its native SPIM frames are 2048x512 (4:1), so the heuristic fired on every real frame and silently threw away half the image. An embryo right of centre produced an empty or garbage projection, one straddling the midline got sliced, and the same path feeds the perceiver. A genuine dual-view volume is 4D (Views, Z, Y, X) and every one of these sites already handles that case explicitly on the preceding line. So the aspect-ratio branch was never selecting a view — only truncating one. Removed at all six. tests/test_view_splitting_regression.py puts a marker in the right half of a synthetic 2048x512 volume and asserts it survives. Five of its six cases fail against the previous code and pass now; the images.py case passes either way and is kept as a guard rather than a proof. Claude-Session: https://claude.ai/code/session_01DNw6x8E3Kbd3yNXsgZxxrF
…tream
Ryan tests on the rig tomorrow. Three things stood between him and the workflow.
He never would have seen it. The devices tab opened on Map and the finished
guided flow sat behind an unlabelled fifth button. Operate is now the landing
view.
It could not be left or reversed. Progressive disclosure had been implemented as
an exclusive CSS selector list (one .op-group at a time) with every control
inside a group, so no control could outlive a step and setStep was only ever
called forward; the header stepper was styled as a breadcrumb but answered
clicks on one of its eight nodes. Every node is now a real button and the cursor
moves in both directions. This is safe because the gates that matter are
evaluated against live hardware state, not step position — revisiting Center
while the head is down still refuses to move. Progress (_states) stays monotonic
and is never rewound by navigation. Escape and an always-visible Cancel leave
any step.
Z was text, not an instrument. The device layer already returned {position, min,
max, distance_to_floor} on every axis call and the frontend dropped min/max on
the floor; a 1 Hz fdrive/bottom_z stream existed with a comment saying it was
for this view and no subscribers. Both axes now render as travel gauges — real
range, current position marked on it, hard-limit line on the F-drive — living on
a shelf outside the step system, so raising the head is one click from anywhere.
The F-drive gains a fine 1 um step.
Also:
- The head-down interlock survives a reload. It was a client-only flag starting
false, so refreshing mid-embryo re-enabled Center and would have commanded an
absolute XY move with the objective down while the strip read "up". Retract is
a relative +100, so there is no absolute safe height to derive it from — it is
persisted instead. A retract that fails no longer reports the head as up.
- One surface per camera stream. Deleted the Map's embedded bottom-cam panel and
guarded the frame handlers and renderStep on visibility, so a hidden Operate
stops decoding frames and racing the visible view for stream ownership.
- The Run chooser defaults to Manual. It defaulted to Adaptive, so the default
path launched a timelapse and never reached Center.
- The grid responds to its container, not the window. With the agent panel open
the container is ~560px on a 1792px display, which the fixed 264/248px side
columns overflowed — collapsing the viewport to a 24px strip and letting the
rail overlap it, while the stacking breakpoint never fired because it keyed off
viewport width.
Deliberately untouched: the stream encoder. Image quality was never reduced —
only frame rate (commit 6797e0a) — and the 15fps cap is what stands between a
lowered head and a repeat of the 2026-06-29 Video-TDR freeze.
Verified in-browser: landing view, forward and backward jumps, Escape from three
targets, Cancel, gauge geometry against known positions, the reload interlock,
and both layouts in light and dark.
Spec: docs/superpowers/specs/2026-07-18-operate-navigable-instruments-design.md
Claude-Session: https://claude.ai/code/session_01DNw6x8E3Kbd3yNXsgZxxrF
…t strip Follow-up to review on the rig-facing surface. Only the axis in play is shown. Offering SPIM-head controls during the bottom-camera steps invited moving the wrong drive; the bottom objective is equally irrelevant once the SPIM camera is live. Gauge visibility now follows the active camera. Awareness of the other axis is not lost — the safety strip carries the sample stage at every step. F-drive steps follow the height. The drive travels ~25000 µm down to a sample sitting around 50-60, and operators work it in bands: a jump to ~5000, then thousands, then hundreds, then tens. A fixed ±10/±100 was 2500 clicks at the top and a crash risk at the bottom. Steps are now generated per band and captioned with which band you are in, so the change is legible rather than surprising. Engagement is derived from telemetry, not just from clicks. The drive can be moved at the controller box, so a latched UI flag read "clear" while someone hand-drove the sample up to the objective — with XY centering still enabled. Either the latch or telemetry within 1000 µm of the limit now locks XY, and neither alone can report "clear". The threshold is a RIG-NOTE, unconfirmed against real geometry. Fixed a stale-telemetry bug this surfaced: the safety strip only repainted from renderStep, so under uncommanded motion TRAVEL LEFT sat frozen at its last commanded value while the sample closed in. It repaints with the gauges now. Also: - Removed the viewport floor bar. It hardcoded a 500 µm full scale and a literal "30 µm hard floor" caption, so it disagreed with the real axis limits and was showing 24800 while the drive was at 30. - The left spine says what it holds: "Where they are" over the map, "Marked embryos" over the worklist, "Mark more embryos" on the button, and an empty state that names the next action instead of "No embryos yet". - The safety strip labels what each reading means (SAMPLE STAGE / TRAVEL LEFT / LED / LASER / CAMERA) with a tooltip on each, rather than bare abbreviations. - Copy is direction-neutral about the hardware: the F-drive number falls as the sample closes on the objective, but which way it physically travels is not asserted here. "Lower the SPIM head" became "Approach the objective". Verified in-browser by replaying an approach as 1 Hz device-state ticks with no UI interaction: readout, marker, gauge foot and strip all follow, the band walks coarse→approach→close→fine, steps that would cross the limit grey out, and XY locks at 400 µm having never been clicked. Claude-Session: https://claude.ai/code/session_01DNw6x8E3Kbd3yNXsgZxxrF
The desktop shell spawns the backend with CREATE_NO_WINDOW in release, so when
something misbehaves mid-session there is no console to look at. The device
layer already had a captured-stdout tail buried in a card on the Devices tab;
the agent side had nothing reachable from the app at all.
A console button in the header opens a drawer with two tabs. Device layer reuses
the supervisor's live captured stdout and falls back to the layer's own log file
when it runs externally and the supervisor captured nothing. Agent tails the
newest {storage}/logs/gently_*.log via a new /api/logs/{source}, reading only
the last 512 KB so a multi-MB session log stays cheap.
Level filter, follow-tail, copy, Esc and Ctrl+` to toggle. Scrolling up turns
following off, because yanking the view back to the end while someone is reading
a traceback is the whole reason log viewers get abandoned. A dot on the header
button marks errors in the tail so a failure that happens while the drawer is
closed still gets noticed.
Polling runs ONLY while the drawer is open — a console that keeps fetching
behind a closed panel is the same class of hidden work that had the camera
streams fighting each other.
Two bugs found by driving it:
- The body used --img-bg, which stays dark under the LIGHT theme while --text is
also dark: black on black, every line invisible. --bg-dark/--text are a
matched pair in both themes.
- A refresh requested while a poll was in flight was dropped, so changing the
tab or the level filter appeared to do nothing for up to two seconds. Pending
refreshes are now remembered and re-run.
Also relabelled a misleading log line: "TUI client disconnected" fired on EVERY
agent-websocket disconnect, including ordinary browser tabs, which reads as the
TUI running when it is not. It now says what it is.
Claude-Session: https://claude.ai/code/session_01DNw6x8E3Kbd3yNXsgZxxrF
Delete the step/workflow layer rather than simplify it. Operate was an 8-node phase stepper (focus/mark/choose/center/approach/focus/calibrate/acquire) with per-embryo progress states, a five-mode run chooser and CSS-exclusive disclosure. It is now three independent surfaces: Bottom cam feed + bottom-Z + detect/mark, click an embryo to centre SPIM head feed + F-drive + sheet alignment + calibrate Acquisition embryo roster + what to run The invariant: no control reads the current pane. `_pane` is consulted only by render dispatch and camera-stream ownership. Gating comes from live hardware state — the XY interlock and the F-drive floor — never from position in a sequence. Selection is a cursor: it parameterises requests and disables nothing. Frontend only; every endpoint already existed. Safety: - One chokepoint for absolute XY moves. The server does not interlock XY against the F-drive, so this predicate is the whole guard. - The interlock is now visible, not merely enforced: a banner on both camera surfaces, markers drawn locked, the canvas cursor withdrawing its affordance, and an always-reachable back-off. Previously the latch could only be cleared on a step you might never reach, stranding XY locked until sessionStorage was cleared by hand. - Telemetry may now clear a stale latch (2x hysteresis), so raising the head at the controller box is noticed. It could previously only add lock. - Camera ownership keys on pane visibility, so at most one decoder ever runs — the condition the 15fps cap exists to protect against. Fixes found on the way: - replay-recorder blocked #op-cam-img by name; the split into two viewports would have silently started recording ~100KB/frame on the main thread. - The F-drive gauge was linear over 30-25000um, making the last 200um — the entire approach-and-crash region — sub-pixel. Now log-scaled with ticks. - The marking overlay sized its backing store once during layout and never re-synced, drawing every marker at the wrong place on a stretched canvas. Now tracks the CSS box, is dpr-aware, and observes resizes. - Markers live in stage coordinates and re-project onto each frame, so they track the sample instead of the viewport. The freeze-the-frame marking mode is gone. - Hidden elements could be defeated by a class setting `display`. Operate subtabs deliberately have no number-key shortcuts: 1-6 are global main-tab navigation, so binding them either steals a key or never fires. Extracted the risky pure logic to operate-math.js with 26 node --test cases (no JS toolchain added): the pixel<->stage transform, F-drive banding, the floor gate, gauge fractions, and the interlock truth table. Stories US-10..US-15 and US-18 rewritten for the new structure; US-19 (surfaces always live) and US-20 (interlock visible) added as the acceptance tests. US-16/17 untouched — they target the Operations tab. Claude-Session: https://claude.ai/code/session_01XMWRh8uasxiH9jKiDxYtrZ
It describes a design that no longer exists. Kept rather than deleted as the record of why the step model was abandoned instead of repaired. Claude-Session: https://claude.ai/code/session_01XMWRh8uasxiH9jKiDxYtrZ
A blockSelector that matches nothing does not error — it silently stops blocking. The failure is severe and invisible: rename the camera <img> and rrweb starts capturing a base64 data-URI src swap at frame rate on the main thread. That is exactly what the Operate rewrite would have caused, since BLOCK_SELECTOR still named the old single-viewport id. Audit the composed selector string actually handed to rrweb (BLOCK_SELECTOR, plus HIGH_CHURN in balanced fidelity) once at boot. Misses go to one console warning and ride along on the page-load action, so a postmortem explains an unexpectedly huge or janky recording instead of leaving it a mystery. Self-calibrating rather than gated on a page sentinel: this recorder also runs on launch/login/settings where none of these elements exist, so it only reports when SOME selector in the list matched — that is what identifies the page as the one the list belongs to. Blind spot: every selector rotting at once reads as "wrong page" and stays silent. They live in different subsystems, so that is far less likely than the false positives a sentinel would produce. Verified in-browser on all three paths: silent on launch (no matches), silent on a healthy index (6/6 match), and on a simulated rot it warns naming the selector and persists blockSelectorMisses into the session's ui-replay/actions.jsonl. Claude-Session: https://claude.ai/code/session_01XMWRh8uasxiH9jKiDxYtrZ
A trailing comment pushed the dom_count call over the line limit. Lifted the comment above the call rather than taking ruff's split-the-arguments fix, which reads worse. Claude-Session: https://claude.ai/code/session_01XMWRh8uasxiH9jKiDxYtrZ
pskeshu
added a commit
that referenced
this pull request
Jul 20, 2026
…ents CLAUDE.md is the file an agent reads before touching this repo, so it should carry what an agent cannot infer and defer everything else. The first draft duplicated most of CONTRIBUTING.md's toolchain section, which is how docs drift apart. Now it points at README.md for setup/run and CONTRIBUTING.md for the lint/type toolchain, and keeps only what neither says: - Where work lands: branch off `development`, PR against `development` in gently-project/gently, with and without write access to the org repo. - The pre-commit hook is not installed in a fresh clone, so commits silently bypass ruff and mypy — how an unformatted file reached CI on #100. - Non-obvious CI behaviour: the lint job stops at its first failing step (so green ruff is not evidence mypy ran), mypy-strict is continue-on-error and does not gate, no CI runs until a PR exists, and JS has no CI coverage at all. - Running off-Windows: the `D:/Gently3` default is not an absolute path on Linux/macOS and silently creates a junk `D:` directory (issue #56), so GENTLY_STORAGE_PATH must be set. Records the --no-api/--no-auth/--no-browser combination and the 8080 UI port. Also corrects the gently-perception reference to gently-project/gently-perception, matching what CI clones and what pyproject.toml expects beside this repo. Claude-Session: https://claude.ai/code/session_01XMWRh8uasxiH9jKiDxYtrZ
…st frame - Stop now freezes the last camera frame instead of clearing it, so the operator can keep reading the sample surface after ending live view; the "LIVE" badge dropping is the cue the frame is static. Falls back to the placeholder only when no frame was ever shown. - The viewport box (.op-cam-fit) is sized to the frame's own aspect ratio, set from the image, so the border hugs the image instead of letterboxing it inside an oversized box. - "Detect automatically" runs on the frame already on screen when one exists: the device layer caches the last full-res stream frame and the stage XY it was taken at, and detect_embryos reuses it (skipping capture and the LED/room-light dance). Falls back to a fresh capture when the viewport is empty. - A spinner overlay is shown over the image while detection runs. - Log the traceback in the detect_embryos capture/SAM except blocks (device layer + client) instead of only returning it in the response body, so environment failures (missing deps, checkpoint, CUDA) are diagnosable from the device log rather than leaving a clean-looking log. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
uv sync (--extra torch-gpu --extra sam) caught the lock up to the already committed pyproject: gently version bump to 1.0.0.dev0 and the playwright dev dependency (plus its transitive greenlet/pyee). No runtime dep changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Shared embryo rail (left of the camera, side-by-side across Bottom/SPIM):
- A persistent embryo list left of every instrument surface, rendering the
canonical set. Click a row to select (highlights everywhere); an × removes
a false positive. Bootstraps from /api/embryos/current on load so it
survives a browser refresh instead of vanishing until the next mutation.
- Hidden on Acquisition (its own roster is richer); stacks above the pane
only at the smallest width.
Count bootstrap:
- The global "N embryos" header/footer read /api/embryos (disk store), which
is empty until a volume is acquired, so they showed 0 on load. Bootstrap
from the in-memory /api/embryos/current (mapped to ids) and fan out an
EMBRYOS_UPDATE so the header strip refreshes too.
Detect UX:
- Return and display the frame SAM ran on, so a fresh-capture detect shows
the image the candidates came from instead of a blank viewport.
- Re-running Detect replaces the previous auto-detected set instead of
stacking onto it (kept: manual marks).
Persist embryos to session files:
- Register writes each embryo to embryos/{id}/embryo.yaml (best-effort,
never blocks registration); delete removes it from disk via a new
FileStore.delete_embryo so a removed false positive doesn't resurrect.
Reload on restart is via the existing --resume path (list_embryos).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Detect on an empty viewport was a single capture+SAM round-trip, so the
image only appeared at the very end, together with the overlays. Split it
into two phases so the operator sees the image first:
- device: detect_embryos now caches the frame it captures, and honours a
capture_only flag that returns the (illuminated) frame WITHOUT running SAM.
- client/route: thread capture_only through.
- operate.js: when the viewport is empty, runDetect first calls capture_only
and displays the frame ("Capturing…"), then calls use_last_frame to run SAM
on that exact image ("Detecting…") and overlays the results. When a frame is
already on screen it skips straight to detection on it.
Sequence is now: image → detecting spinner → overlaid candidates.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
pskeshu
added a commit
that referenced
this pull request
Jul 24, 2026
Review catch from @subindevs: `node --test tests/js/` fails on this branch — `tests/js/` exists only on the operate branch (#100), not on development. The old wording also implied JS tests already exist on development, which they don't. Reworded to the durable fact — CI runs no JavaScript, so verify UI changes by running the app — which holds whether or not an in-tree JS suite exists, so it does not go stale when #100 lands one. Claude-Session: https://claude.ai/code/session_01XMWRh8uasxiH9jKiDxYtrZ
Bind the capture error to a local before formatting it into the response so the line fits the 100-column limit, and let ruff format rejoin a log message that fits on one line. No behaviour change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prep for Ryan's first session on the rig. Three things stood between a first-time
user and the workflow, plus one real imaging bug found on the way.
He would never have seen the workflow
The devices tab opened on Map, and the finished guided flow sat behind an
unlabelled fifth button. Operate is now the landing view. Two lines, and the
single highest-leverage change here.
Operate could not be left or reversed
Progressive disclosure had been built as an exclusive CSS selector list — one
.op-groupat a time — with every control inside a group, so no controlcould outlive a step and
setStepwas only ever called forward. The headerstepper was styled as a navigable breadcrumb but answered clicks on one of its
eight nodes.
Every node is a real button now and the cursor moves in both directions. This is
safe because the gates that matter are evaluated against live hardware state,
not step position — revisiting Center while the sample is engaged still
refuses. Progress (
_states) stays monotonic and is never rewound bynavigation. Escape and an always-visible Cancel leave any step.
Z was text, not an instrument
The device layer already returned
{position, min, max, distance_to_floor}onevery axis call and the frontend dropped
min/maxon the floor; a 1 Hzfdrive/bottom_zstream existed with a comment saying it was for this viewand no subscribers. Both axes are travel gauges now — real range, current
position marked on it, hard limit drawn — and only the axis the current step can
act on is shown.
F-drive steps follow the height, because the drive travels ~25000 µm down to a
sample around 50–60 and operators work it in bands. A fixed ±10/±100 was 2500
clicks at the top and a crash risk at the bottom. Steps are generated per band
and captioned with the band you are in.
Safety
moved at the controller box, so a latched UI flag read "clear" while someone
hand-drove the sample to the objective — with XY centering still enabled.
Either the latch or telemetry within 1000 µm of the limit now locks XY.
false,so refreshing mid-embryo re-enabled Center and would have commanded an
absolute XY move with the sample engaged. Retract is a relative +100, so there
is no absolute safe height to derive it from; it is persisted instead.
renderStep, sounder uncommanded motion TRAVEL LEFT sat frozen while the sample closed in.
Half of every SPIM frame was being discarded
Six call sites carried
if width > height * 2: take the left half, a guess atsplitting a stitched dual-camera frame. This rig is single-camera and its native
frames are 2048×512, so it fired on every frame. An embryo right of centre
produced a garbage projection, and the same path feeds the perceiver. A genuine
dual-view volume is 4D and every site already handled that on the preceding
line, so the branch was never selecting a view — only truncating one.
Deliberately untouched
The stream encoder. Image quality was never reduced — only frame rate
(
6797e0a) — and the 15 fps cap is what stands between an engaged sample and arepeat of the 2026-06-29
nvlddmkmVideo-TDR freeze. Raising it needs a clientthrottle and a measurement on the rig first.
Also deferred: the Map's arbitrary 0,0 framing, and retiring SAM (needs a
precision/recall harness, and the labels captured so far are anchored to lossy
thumbnails).
Verification
pre-existing. Ruff clean.
tests/test_view_splitting_regression.py: puts a marker in the right half of asynthetic 2048×512 volume. Five of six cases fail against the previous code
and pass now; the
images.pycase passes either way and is kept as a guard,not a proof.
targets, Cancel, gauge geometry against known positions, the reload interlock,
and both layouts in light and dark.
readout, marker, gauge foot and strip all follow, the band walks
coarse→approach→close→fine, over-limit steps grey out, and XY locks at 400 µm
having never been clicked.
Needs confirming on the rig
objective, but I don't know whether that is physically up or down, so all copy
is direction-neutral. Worth making concrete once confirmed.
ENGAGED_WITHIN_UM = 1000, the threshold that locks XY. Marked RIG-NOTE.Spec:
docs/superpowers/specs/2026-07-18-operate-navigable-instruments-design.mdhttps://claude.ai/code/session_01DNw6x8E3Kbd3yNXsgZxxrF