DATA-8: /process/acquisition/start_date bug fixed upstream (tomoscan + dxfile + meta-cli); 2-BM end_date workaround can be relaxed for new files

[decarlof]

Summary

The /process/acquisition/start_date bug in 2-BM scan HDF5 files that CORA surfaced in the 2026-08-11 briefing (every non-trivial test_00N.h5 carrying the previous scan's last-frame time as start_date) is now fixed in production. Three repositories cooperated on the fix, each addressing a different layer of the pipeline. end_date was correct throughout and remains untouched. Canonical bug ticket: tomography/tomoscan#180 (closed).

This issue exists to give CORA a single place to record the fix provenance and to specify how CORA's own end_date-based workaround at 2-BM should evolve now.

What the bug was

Live reproduction (2026-08-12)

An operator started a new scan at approximately 15:30:05 CDT on 2026-08-12. The resulting file at /local2/2BM/2026-08-DeCarlo-1015116/test_006.h5 contained:

• /process/acquisition/start_date = 2026-08-12T06:21:17-0500 (off by -9 h 8 min 48 s)
• /process/acquisition/end_date = 2026-08-12T15:33:38-0500 (correct)
• array_counter = 7824, acquire_period = 0.019528192 s (approximately 2 min 33 s of raw acquisition, approximately 3 min 33 s wall-clock including dark and flat overhead)

The stale timestamp 06:21:17 sat within 5 seconds of the previous scan's (test_005) closing time (06:21:22). That interval is the gap between the last NDArray of test_005 being tagged by the areaDetector NDAttributes plugin and the HDF5 plugin closing that file.

The prior sweep by the CORA maintainer over test_000..test_005 on 2026-08-11 showed the same pattern in every non-trivial file. Only test_002 looked plausible, and only by coincidence (its file open happened close in time to the predecessor's close).

Root cause

Two configuration files in the 2-BM detector-IOC deployment cooperated to produce the bug.

The NDAttribute definitions in TomoScanDetectorAttributes.xml (lines 5-6) declared both timestamps as EPICS-PV monitors on the same APS time-of-day PV:

<Attribute name="DateTimeStart" type="EPICS_PV" source="S:IOC:timeOfDayISO8601" dbrtype="DBR_STRING" description="Date and time at start"/>
<Attribute name="DateTimeEnd"   type="EPICS_PV" source="S:IOC:timeOfDayISO8601" dbrtype="DBR_STRING" description="Date and time at end"/>

The HDF5 layout in TomoScanLayout.xml wired them into the two datasets that ended up in every scan file:

<dataset name="start_date" source="ndattribute" ndattribute="DateTimeStart" when="OnFileOpen"  />
<dataset name="end_date"   source="ndattribute" ndattribute="DateTimeEnd"   when="OnFileClose" />

Mechanism: EPICS_PV NDAttributes in areaDetector are snapshotted onto each NDArray as it passes through the NDAttributes plugin, at frame-creation time. Between scans, no NDArrays flow, so the plugin's cached value stays frozen at the moment the last frame of the previous scan was tagged. When the HDF5 plugin evaluates when="OnFileOpen" for start_date, no fresh NDArray of the current scan has arrived yet, so it writes the stale cached value. end_dat e avoids the problem because by when="OnFileClose" the current scan's final frame has already refreshed the snapshot.

Put more compactly: when="OnFileOpen" is essentially incompatible with EPICS_PV NDAttribute sources for values that need to reflect "now at file open". The IOC-side pipeline cannot observe scan boundaries; only the tomoscan client can.

Impact prior to the fix

• Every scan HDF5 file written by tomoscan with this NDAttribute + layout pattern is affected (not just 2-BM; any beamline sharing the template inherits the bug).
• Skew ranges from seconds (short inter-scan gap) to days (long idle between scans). The 2026-08-12 test observed a 9-hour offset.
• No warning is emitted anywhere; the file looks internally consistent.
• Any consumer trusting start_date (time-range filtering, per-run duration math, audit-trail time provenance, cross-referencing beamline logs) silently uses a wrong value. This is precisely the class of silent data-integrity bug CORA's accountability layer is designed to surface.

What was fixed, where

1. Client-side write of the correct value (tomoscan)

Commit: decarlof/tomoscan@d0025a2 (tomoscan_2bm: write scan-start time via h5py in end_scan()).
File touched: tomoscan/tomoscan_2bm.py (+29 lines; one new import from datetime import datetime, timezone).

The client now captures self.scan_start_iso at the top of begin_scan() and, once the areaDetector HDF5 plugin has closed the file, end_scan() calls a new add_start_date() method that reopens the file via h5py and writes the Python-captured ISO timestamp to /process/acquisition/start_date. Called right after add_theta(). Dataset created if missing; first element overwritten otherwise.

Effect: every new scan HDF5 file written by tomoscan at 2-BM carries the correct start_date.

Note: the recommended fix in tomography/tomoscan#180 suggested preserving the stale IOC-written value under a companion attribute start_date_from_ioc for provenance. The committed fix does not add that attribute; the wrong value is simply overwritten. See "What this means for CORA at 2-BM" below for the consequence.

2. IOC-side removal of the source of the stale value (dxfile / areaDetector XML)

Commit: data-exchange/dxfile@de33f3a (fix(2-BM): remove stale DateTimeStart/start_date from XML layout).
Files touched (7 total under doc/demo/areadetector/2-BM/):

• 2bmaTomoScanDetectorAttributes.xml
• 2bmbTomoScanDetectorAttributes.xml
• adimec2bmaTomoScanDetectorAttributes.xml
• adimec2bmbTomoScanDetectorAttributes.xml
• 2bmaTomoScanLayout.xml
• adimec2bmaTomoScanLayout.xml
• adimec2bmbTomoScanLayout.xml

In each *DetectorAttributes.xml: the DateTimeStart EPICS_PV NDAttribute on S:IOC:timeOfDayISO8601 is deleted. DateTimeEnd (same PV, when="OnFileClose" in the layout) is retained unchanged.

In each *Layout.xml: the <dataset name="start_date" source="ndattribute" ndattribute="DateTimeStart" when="OnFileOpen" /> line is deleted. The end_date dataset (when="OnFileClose") is kept.

Net effect: the IOC no longer writes the bad start_date; the tomoscan client (fix 1) is now the sole writer. This removes the OnFileOpen + EPICS_PV race pattern entirely from the pipeline.

3. Handling for old files on disk (meta-cli)

Commit: xray-imaging/meta-cli@29a56bc (show: mark stale /process/acquisition/start_date as [derived]).
Files touched: src/meta_cli/__main__.py (+2), src/meta_cli/utils.py (+50). No new subcommand or flag. Hooks into the existing show command via a new internal helper utils._mark_stale_start_date(meta_dict).

For files written before fix 1 was deployed, the wrong start_date is still on disk. meta show now detects the stale value, computes a corrected value in-memory as

start_date_derived = end_date - (num_angles * acquire_period
                                 + num_dark * exposure
                                 + num_flat * flat_exposure_time
                                 + 30 s overhead)

and prints it with a " [derived]" suffix so operators and audit tooling can tell it apart from a native-correct value. No files on disk are modified by this change; the correction is display-only at read time.

What this means for CORA at 2-BM

For new scans (after the fix deployment)

• /process/acquisition/start_date is now correct and trustworthy at 2-BM.
• CORA can relax the "read end_date at 2-BM and record which timestamp it used" workaround for files written after fix 1 rolled out. Recommendation: keep the accountability-layer record of which timestamp field was read in the CORA entry (that pattern generalises well to other beamlines), but the value read from start_date for a new-file scan can be trusted at face value.
• Because fix 1 does not write a start_date_from_ioc companion attribute, a new-file start_date cannot be compared against the previously-wrong on-disk value. CORA's accountability record should note the fix-deployment date and treat every scan file after that date as having a client-written start_date with no shadow of the earlier IOC value.

For scans written before deployment (backfill)

• The stale start_date remains on disk. Nothing rewrites it.
• The authoritative read for these files is: run meta show on them and use the [derived] value, or reproduce the derivation directly from end_date, array_counter / num_angles, acquire_period, num_dark_fields, exposure_time, num_flat_fields, and flat_exposure_time.
• If CORA wants a batch-computed corrected value stored alongside its per-scan entries, the meta-cli formula above is the reference; a small script wrapping meta_cli.utils._mark_stale_start_date over /local*/2BM/ and /gdata/dm/2BM/*/ would produce it without touching the source HDF5 files.
• CORA's existing end_date-based workaround stays correct for these older files and can be left in place until the pre-fix files fall out of the active audit window.

Deployment status at 2-BM

• Fix 1 (tomoscan@d0025a2) is in the production tomoscan client running on tomdet. All scans launched after this deployment produce correct start_date.
• Fix 2 (dxfile@de33f3a) is deployed via the working tree at /home/beams/2BMB/conda/dxfile/ (remote decarlof/dxfile, HEAD at the fix commit). The 2bmbSP2 IOC boot dir at /net/s2dserv/xorApps/epics/PreBuilts/2bmbSP2/iocBoot/ioc2bmbSP2/TomoScanDetectorAttributes.xml and TomoScanLayout.xml symlinks into that working tree, so the fixed files are already on the paths the IOC opens. An IOC restart (to drop the CA monitor for the removed DateTimeStart
  NDAttribute and to re-parse the layout XML) is the remaining step to make the IOC stop writing the stale start_date at file-open time; until that restart happens, fix 1's post-close h5py overwrite is what keeps new files correct.
• Fix 3 (meta-cli@29a56bc) is a client tool; picks up the fix at next meta install / update.

Provenance / paper trail for CORA's accountability record

For the CORA audit layer, the following facts about this bug and its resolution are worth capturing in one place (this issue can serve as the anchor):

• Bug surface date: 2026-08-11 (CORA maintainer's briefing sweep of test_000..005.h5).
• Root-cause diagnosis: EPICS_PV NDAttribute snapshot is only refreshed when an NDArray flows through the areaDetector NDAttributes plugin; when="OnFileOpen" fires before any NDArray of the new scan flows, so the plugin cache still holds the last-frame value of the previous scan.
• Live reproduction: test_006.h5 started 2026-08-12 15:30:05 CDT recorded start_date = 2026-08-12T06:21:17-0500 (9 h 8 min 48 s stale), matching test_005.h5's closing time.
• Fix strategy: move the write from the IOC (which cannot observe scan boundaries) to the client (which can). No IOC-side change to the areaDetector core plugin was needed.
• Fix commits: three (linked above), spanning tomoscan client, areaDetector XML template, and meta-cli reader.
• Canonical bug ticket: tomography/tomoscan#180 (closed).

Doc-side follow-up

A note in 2bm-docs (docs/source/ops/item_019.rst Data Management, or the MCTOptics / detector block in docs/source/manual/item_020.rst) will flag: (a) that pre-fix scan files have a wrong on-disk start_date and should be read via meta show, (b) that the tomoscan client is now the authoritative writer of start_date at 2-BM. Will land as a separate small 2bm-docs PR.

Related

• tomography/tomoscan#180 (closed) - canonical bug ticket with root-cause and reproduction detail.