Autoexposure/Autogain doesn't work in Protocol

[EtalumaSupport]

If AE/AG is used with AF in a Step, the image is overexposed. If AE/AG is used without AF, it skips the AE/AG process and takes an overexposed image.

A3_BF_2xOly_0000.tiff
A4_BF_2xOly_0000.tiff
AE_AFtest.tsv
api.log
autofocus.log
camera.log
gui_interactions.log
live_A2_BF_000001.tiff
lumaviewpro.log
lumaviewpro_errors.log
metrics.log
serial.log

[ericweiner]

Root cause located (code-side, no bench needed)

Auto-gain is dropped on the floor during protocol scans. The apply_layer_camera_settings call -- the only place that fires set_auto_gain(state=auto_gain, ...) -- is gated behind if not protocol_running_global.is_set(): at ui/layer_control.py:1008-1033. So during a protocol scan, the gate is FALSE and the entire block (gain + exposure + auto_gain apply) is skipped.

The gain/exposure are then set instead by protocol_image_writer.capture() at lines 203-227:

if not step['Auto_Gain']:
    self._scope.imaging.set_gain(step['Gain'])
    self._scope.imaging.set_exposure_time(step['Exposure'])
else:
    logger.warning(f"[CAPTURE DIAG] SKIPPING camera settings -- Auto_Gain is truthy: {_ag!r}")

When step['Auto_Gain'] is True, the static set_gain/set_exposure is SKIPPED. And nothing else calls set_auto_gain(True, ...) anywhere in the protocol execution path -- verified via grep -rn 'set_auto_gain' modules/ ui/. The only protocol-side call is at protocol_step_runner.py:265, which sets state=False at end-of-step (to disable AG between steps).

Result: when Auto_Gain=True for a protocol step, the camera fires with whatever gain/exposure it inherited from the previous context:

• AE/AG + AF: AF iterates gain/exposure during its sweep, leaves them at the last AF iteration's values (often high gain / long exposure for the focus metric). Capture uses those values + the channel's LED -> overexposed.
• AE/AG without AF: Camera holds whatever gain/exposure live mode / the previous step left -> overexposed (or wildly wrong) depending on context.

Evidence in the attached api.log

12 apply_layer_camera_settings ... auto_gain=True events total, ALL from manual UI tweaking before the protocol started. ZERO inside any of the 8+ save_camera_state tag=protocol -> restore_camera_state windows. Confirmed via awk '/save_camera_state.*protocol/,/restore_camera_state/' api.log | grep -c 'auto_gain=True' -> 0.

There's also a vestigial if step['Auto_Gain'] and time.monotonic() < p._auto_gain_deadline: return at protocol_step_runner.py:195 -- the runner waits for an autogain deadline, but no one actually fires the autogain procedure. The deadline expires (no convergence happens), capture fires the overexposed image.

Proposed fix (1 commit, well-scoped)

When step['Auto_Gain'] is True in protocol_image_writer.capture(), route through the same path as live-mode AE/AG:

if step['Auto_Gain']:
    self._scope.imaging.apply_layer_camera_settings(
        gain_db=step['Gain'],
        exposure_ms=step['Exposure'],
        auto_gain=True,
        auto_gain_settings=autogain_settings,  # already passed in
    )
    # Wait for convergence -- existing _auto_gain_deadline mechanism in
    # protocol_step_runner.py:195 covers this (loops scan_iterate until
    # the deadline expires or AG converges)
else:
    self._scope.imaging.set_gain(step['Gain'])
    self._scope.imaging.set_exposure_time(step['Exposure'])

Plus a corresponding regression test: protocol with Auto_Gain=True steps -> verify set_auto_gain(state=True, ...) fires per step.

Cluster context

Adjacent to #672 (Protocol-vs-Manual divergence on composite) -- likely same class of "protocol path bypasses a live-mode code path that owns important state." Worth investigating jointly.

Bench retest signal: none yet -- no fix has landed.

[ericweiner]

Fixed on 4.0.0-beta 191dfa0 fix(protocol): #673 Auto_Gain steps fire set_auto_gain in protocol scans -- shipping in beta14.

The else branch of protocol_image_writer.capture()'s if not step['Auto_Gain']: block now routes through apply_layer_camera_settings(auto_gain=True, auto_gain_settings=...) -- same path live-mode AE/AG uses. Convergence handled by the runner's existing _auto_gain_deadline loop in scan_iterate.

Regression test: tests/test_issue_673_protocol_auto_gain.py. Two cases:

• Auto_Gain=True step -> at least one apply_layer_camera_settings ... auto_gain=True inside the save_camera_state(protocol) -> restore_camera_state(protocol) window.
• Auto_Gain=False step -> no apply_layer_camera_settings ... auto_gain=True in the window (symmetry guard).

Both pass on current code.

Bench retest signal: run the attached AE_AFtest.tsv protocol on 4.0.0-beta @ 9b0c28b (beta14) or later. The captured images should NOT be saturated -- exposure should converge to a usable brightness. If they're still saturated, attach the new api.log and the contained apply_layer_camera_settings lines inside the protocol window.

#672 retest is gated on this -- once #673 is bench-confirmed, re-attach the protocol-vs-manual composite images. If they still diverge, the remaining cause is the H1/H2 in the prior triage (blend pipeline, not per-channel gain).

[ericweiner]

F1 of the API protocol-workflow audit closed at LVP 2fa67d1 (4.0.0-beta). Pairs with the prior #673 fix at 191dfa0; this is the ordering correction that makes Auto_Gain actually converge before the grab.

Static-read root cause (from the audit doc docs/AUDIT_PROTOCOL_WORKFLOW_2026-05-21.md F1, captured before the fix):

The Auto_Gain deadline-wait gate in scan_iterate (protocol_step_runner.py:195) ran for max_duration seconds BEFORE the apply_layer_camera_settings(...auto_gain=True) call inside capture() ever fired. So AG was OFF during the entire convergence window and got ONE frame of convergence between the apply call and the grab. Result: arbitrary or overexposed image every Auto_Gain step.

Fix: AG-arm hoisted into scan_iterate AHEAD of the deadline gate via a one-shot _auto_gain_armed_step keyed on _curr_step. Reset to -1 per scan in protocol_run_loop alongside _auto_gain_deadline. The arm fires once via io_executor.protocol_put (same serialization as other protocol-thread IO). capture()'s Auto_Gain branch now skips the apply (already armed; re-firing would restart AG mid-grab).

New ordering during a single Auto_Gain step:

1. scan_iterate arms AG (apply_layer_camera_settings ... auto_gain=True) -> camera starts converging.
2. Deadline-wait gate returns early for max_duration seconds. AG is converging during this entire window (not idling like before).
3. Deadline clears; capture() grabs a converged frame.
4. Post-grab set_auto_gain(state=False) disables AG for the next step.

Regression test: new class TestAutoGainArmedBeforeDeadlineWait (5 cases) static-asserts the arm block precedes the deadline gate, routes through io_executor, and the duplicate apply in capture() is removed. Existing tests/test_issue_673_protocol_auto_gain.py (2 cases) remains green.

To verify on bench:

1. Build a single-step Auto_Gain protocol with a contrasty sample.
2. Run it; the first captured image should be well-exposed rather than overexposed.
3. Check api.log: apply_layer_camera_settings ... auto_gain=True should appear ~max_duration seconds BEFORE the matching capture_and_wait (not adjacent to it).
4. Try with max_duration = 2 s to make the gap easy to see in the log.

[ericweiner]

Ready for bench verification

Fix shipped on 4.0.0-beta at 191dfa0 + 2fa67d1 -- Auto_Gain steps fire set_auto_gain in protocol scans + arm BEFORE deadline-wait. Current shipping tip is bf44db7 (beta14 build).

EtalumaSupport, please download the latest beta build and verify the fix. Then either:

• Close this issue if fully resolved, OR
• Add logs / a note describing what is still happening.

[EtalumaSupport]

SNlogs-2026-05-22-073254.zip

simple BF 96 well Protocol with AE/AG and AF produces overexposed images

[ericweiner]

Recurrence root cause found; fix shipped on 4.0.0-beta @ 4036059

Your 2026-05-22 bundle (SNlogs-2026-05-22-073254.zip) was decisive: 191dfa0 + 2fa67d1 hoisted the AG arm correctly but the deadline mechanism was mis-ordered. Camera.log shows AG enabled then disabled 66-120ms later across all 4 protocol steps -- nowhere near the intended 1000ms convergence window.

Root cause

• protocol_run_loop.py:158 set _auto_gain_deadline = scan_start + max_duration (1.0s) BEFORE AF.
• AF ran ~10s.
• First AG step's gate if time.monotonic() < deadline: saw deadline already ~9s in the past → fell through immediately → capture grabbed ~70ms after arm.
• protocol_step_runner.py:224 reset deadline AFTER the gate, so each step inherited the same broken pattern.

What changed

• modules/protocol_step_runner.py:198-224 -- set _auto_gain_deadline INSIDE the arm block (one-shot per step) and return so the next scan_iterate tick polls the gate properly. Subsequent ticks return-early until the deadline expires, then fall through to capture.
• modules/protocol_run_loop.py:158 -- removed the scan-start deadline init (it was the source of the mis-ordering).
• tests/test_audit_fixes.py::TestAutoGainArmedBeforeDeadlineWait -- 3 new tests: deadline set inside arm block (not after gate), arm block returns after setting deadline, run-loop does NOT set deadline at scan start.

Bench retest signal

Run your simple BF 96-well protocol with AE/AG + AF. Check:

• api.log: apply_layer_camera_settings ... auto_gain=True event should appear ~1000ms BEFORE the next capture/LED event
• camera.log: GainAuto.SetValue(Continuous) → GainAuto.SetValue(Off) timestamps should span the configured max_duration (default 1.0s), not 70-120ms
• Resulting images should not be overexposed

If overexposure persists after this fix and the convergence window is confirmed correct, the dominant cause shifts to LED brightness / AG target — a separate config concern.

Cluster note

Cluster sibling of #655 (Red AG histogram dim). Both share the AG subsystem but are different mechanisms. Your retest should benefit from #655's fix (e9e0bd4) too -- AG now has both convergence TIME (this fix) and convergence ROOM on the exposure axis (#655).

[EtalumaSupport]

First images of an AG/AE BF Protocol are overexposed

SN12064-2026-05-25-082651.zip

A1_BF_2xOly_0000.tiff
A2_BF_2xOly_0000.tiff
A3_BF_2xOly_0000.tiff
A4_BF_2xOly_0000.tiff
A5_BF_2xOly_0000.tiff
A6_BF_2xOly_0000.tiff
A7_BF_2xOly_0000.tiff
A8_BF_2xOly_0000.tiff
protocol_record.tsv
unsaved_protocol.tsv