Skip to content

Support multi-pulse ev44 messages #708

@SimonHeybrock

Description

@SimonHeybrock

Problem

_require_single_pulse in src/ess/livedata/handlers/to_nxevent_data.py raises NotImplementedError when an ev44 message contains events from multiple pulses. The ev44 schema supports packing events from several pulses into a single message via reference_time (array of pulse timestamps) and reference_time_index (array of starting indices for each pulse's events within the flat event arrays).

This appears to be a leftover simplification from early development, when all producers sent single-pulse messages. It will break if an upstream source starts batching pulses.

Inconsistency with fast-path adapter

KafkaToMonitorEventsAdapter (the optimized direct-from-flatbuffer path) already silently ignores multi-pulse structure: it takes the flat TimeOfFlightAsNumpy() array and uses reference_time[-1] as the message timestamp, discarding per-pulse boundaries entirely. So the slow path (via MonitorEvents.from_ev44 / DetectorEvents.from_ev44) raises an error, while the fast path silently drops pulse information. Both behaviors are wrong.

What needs to change

The ev44 multi-pulse layout is:

  • reference_time[i] — timestamp for pulse i
  • reference_time_index[i] — starting index in the event arrays for pulse i
  • Events for pulse i span indices reference_time_index[i] to reference_time_index[i+1] (or end-of-array for the last pulse)

Each pulse should produce a separate events object with its own reference_time used as the event_time_zero timestamp in the downstream accumulator.

Implementation sketch

  • from_ev44 returns a list of events objects (one per pulse), slicing the flat arrays using reference_time_index as boundaries
  • The MessageAdapter protocol and AdaptingMessageSource need to support 1:N message expansion (one ev44 → N messages), or the expansion can be handled at the adapter level
  • KafkaToMonitorEventsAdapter needs the same treatment
  • The change is mechanically straightforward (NumPy index slicing), but touches several layers: dataclasses, adapters, and the adapting source

Alternatives

1->N message translation might not be what we want. Explore changing the internal detector/monitor message schema, adding reference_time (etc.) fields to them.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions