Refactor package for unified data and device API

[bruno-f-cruz]

Refactor package for unified data and device API

Objective

Establish a clean, typed Python API for the Harp protocol that:

• works identically for live device communication and bulk file reads
• provides full static type support so IDEs and type-checkers give correct completions and error highlighting for register payloads
• exposes a declarative DSL (Field, BitFlag, GroupMask) that code generators for downstream device packages can emit directly from device.yml (the codegen is NOT part of this PR and likely will not belong in this repository)
• keeps each concern in its own installable package: wire protocol (harp-protocol), device/transport (harp-device + harp-serial), and tabular data (harp-data)

---

Packages

The repo is a uv workspace; every member lives under src/packages/* with a uniform src/-layout, and the top-level project is named harp. There are four packages:

harp-protocol   wire protocol, message parsing, payload/register DSL   (numpy only)
harp-device     transport-agnostic Device + ITransport, core registers, REGISTER_MAP
harp-serial     serial transport: SerialTransport + open_serial_device  (pulls in pyserial)
harp-data       load register data into pandas DataFrames               (pulls in pandas)

Separation of concerns

Each package owns exactly one concern, depends only on the layers below it, and isolates its heavy third-party dependency so consumers pay only for what they import:

Package	Owns (single responsibility)	Explicitly does NOT	Depends on	Heavy dep
harp-protocol	Wire protocol: message framing/parsing, the payload + register DSL (Field/BitFlag/GroupMask), and the pandas-free Column/to_columns() tabular view	No transport, no device lifecycle, no DataFrames	—	numpy only
harp-device	Transport-agnostic device: the Device base (framing, request/reply, reader thread, read/write), the ITransport protocol, core register definitions, REGISTER_MAP	No concrete transport (no serial/zmq), no DataFrames	harp-protocol	—
harp-serial	The serial transport: SerialTransport + the open_serial_device factory	No protocol/device logic — only moves bytes	harp-device, harp-protocol	pyserial
harp-data	Turning parsed register data into pandas DataFrames: read_dataframe / to_dataframe	No I/O, no protocol logic — consumes Columns	harp-protocol	pandas

Dependency edges:

• harp-device → harp-protocol
• harp-serial → harp-device, harp-protocol
• harp-data → harp-protocol

A downstream device package (e.g. harp-device-behavior) depends only on harp-protocol; it imports harp-device only if it also wants the live transport, harp-serial only if it wants serial, and harp-data only if it wants DataFrames. The heavy third-party dependencies (pandas, pyserial) live only in the leaf packages that need them, so harp-protocol and harp-device stay lightweight.

---

Major features/architecture

1. Register DSL (harp.protocol)

RegisterBase[U] is the schema object for a Harp register. Every register class carries address, payload_type, and payload_class as ClassVars (plus length for array registers) and exposes three uniform class-methods:

Method	Purpose
parse(value)	Decode a single HarpMessage (or raw bytes/bytearray/memoryview) → typed U
parse_bulk(source, *, parse_timestamp=True)	Decode a bulk binary buffer → (data, timestamps, message_types, payload) (numpy, zero-copy strided views)
format(value?, ...)	Build a Harp wire frame (Read if no value, Write otherwise)

Note: registers are numpy-only — they don't produce pd.DataFrames. Tabular output is a harp-data concern: harp.data.read_dataframe(register, source) (see §5).

The same register class works regardless of whether the source is a serial port reply, an in-memory buffer, or a .bin file on disk. No subclassing or reconfiguration is required.

Concrete register base-classes for every Harp scalar type are provided (RegisterU8, RegisterU16, …, RegisterFloat) as well as array variants (RegisterU16Array, …). These use a metaclass so that one-off anonymous registers can be created inline:

Reg = RegisterU16(address=0x20)
frame = Reg.format()

2. Payload DSL — anonymous (scalar / array) payloads

For scalar and array registers (no internal structure), PayloadU8, PayloadU16, …, PayloadFloatArray are provided. RegisterBase.parse unwraps these directly to a numpy scalar or ndarray — i.e. no .value accessor needed.

# ── Anonymous scalar register ─────────────────────────────────────────────
class WhoAmI(RegisterU16):
    address: ClassVar[int] = 0

who = WhoAmI.parse(msg)                # np.uint16 — no wrapper
frame = WhoAmI.format(np.uint16(1216)) # encode a write frame

NOTE I went with numpy types rather than native python types since these preserve bitdepth and add additional typing information. We should discuss. In general, while the type hinting may break, native python types will still get implicitly cast by numpy.

# ── Anonymous array register (e.g. AnalogData: 3 × S16) ──────────────────
# Array registers are sized by *calling* the base class with a length, which
# returns a concrete register type.
AnalogData = RegisterS16Array(44, length=3)

samples = AnalogData.parse(msg)        # NDArray[np.int16], shape (3,)

# Bulk parse — zero-copy strided views into the raw buffer
data, timestamps, msg_types, payload = AnalogData.parse_bulk(buf)

# DataFrame assembly lives in harp.data (see §5)
from harp.data import read_dataframe
df = read_dataframe(AnalogData, buf)   # columns: "timestamp", "0", "1", "2"

3. Payload DSL — structured payloads

StructPayload + Field / BitFlag / GroupMask descriptors declare the layout of a structured register payload. Under @dataclass_transform the class gets a fully type-checked keyword-only __init__ for free.

Field

Each member declares its byte position with offset= (in base-element units). offset defaults to 0, which suits a single-member payload; a payload with several members must give each an explicit offset= or the layout overlap-check rejects it.

from harp.protocol import StructPayload, Field, StringConverter, IdentityConverter

class FileSettings0Payload(StructPayload[np.uint8]):
    cycles:              np.uint32 = Field(converter=IdentityConverter(np.uint32), offset=0)
    duration_us:         np.uint32 = Field(converter=IdentityConverter(np.uint32), offset=4)
    update_frequency_hz: np.uint32 = Field(converter=IdentityConverter(np.uint32), offset=8)
    path:                str       = Field(converter=StringConverter(33),
                                           offset=12, default="220khzwaveform")

BitFlag + GroupMask — bit-packed payloads

Descriptor types for registers whose payload packs multiple boolean flags or enum values into a single byte (the OperationControl pattern common across all Harp devices). The right-shift for a GroupMask is derived automatically from the mask's trailing-zero count — there is no separate shift= argument.

Enum members are emitted in canonical UPPER_CASE (e.g. OperationMode.ACTIVE, EnableFlag.ENABLED), and the core registers use canonical names (HardwareVersionHigh, TimestampSeconds, ClockConfiguration, …) — both come straight from device.yml via the generator.

from harp.protocol import HarpMessage, MessageType
from harp.device import (
    OperationControl, OperationControlPayload, OperationMode, EnableFlag,
)

# ── Construct from kwargs — type-checked by IDE/type-checker ─────────────
p = OperationControlPayload(
    operation_mode=OperationMode.ACTIVE,
    heartbeat=EnableFlag.ENABLED,
)
print(p.operation_mode)   # OperationMode.ACTIVE   ← Python enum, no [0]
print(p.dump_registers)   # False                  ← Python bool

# ── Encode / decode a single frame ───────────────────────────────────────
frame  = OperationControl.format(p, message_type=MessageType.Write)
parsed = OperationControl.parse(HarpMessage.parse(frame))
assert parsed.heartbeat == EnableFlag.ENABLED

# ── Bulk read — same descriptors, array-shaped results ───────────────────
data, timestamps, _, payload = OperationControl.parse_bulk(buf)
print(payload.operation_mode)   # NDArray of raw enum codes, shape (N,)

from harp.data import read_dataframe
df = read_dataframe(OperationControl, buf, decode_enums=True)
# columns: "timestamp", "operation_mode" (Categorical), "dump_registers", …

The descriptors are shape-polymorphic: a 0-D (single) parse returns a Python scalar; a 1-D (bulk) parse returns a numpy array. The descriptor code is identical in both cases.

4. Device & transport layer (harp-device + harp-serial)

The device side uses composition over inheritance, split into two independent axes so they multiply without a diamond:

• what the device is — registers, __whoami__, device logic → the Device axis
• how bytes move — serial, zmq, file → the ITransport axis

Device  (protocol logic)  ──has-a──▶  ITransport  (byte channel)

Device (in harp.device) owns all protocol logic — framing, the request/reply correlation, the reader thread, and read/write — and drives an ITransport. ITransport is a typing.Protocol (structural — any object with open/write/read/close qualifies; failures surface as TransportError). A device must be opened (with / .open()); opening starts the reader thread and validates __whoami__ (a class attribute; 0x0 skips the check).

harp-serial provides SerialTransport (conforms structurally to ITransport) and an open_serial_device factory that — like the builtin open — returns an already-connected device:

from harp.device import Device
from harp.serial import open_serial_device

with open_serial_device(Device, port="COM3", baudrate=1_000_000) as dev:
    print(dev.read(dev.WhoAmI).parsed)   # core registers are exposed as Device attrs

A device definition is a pure subclass with no transport concern — exactly what a generator emits:

class Behavior(Device):
    __whoami__ = 1216
    # + generated device-specific register class-attrs

Because the transport is just an ITransport, new ones drop in with one implementation + a factory and zero changes to device classes:

• zeromq — talk to a device (or broker) over a ZMQ socket: open_zmq_device(MyDevice, endpoint=...).
• http — drive a device behind an HTTP/REST bridge.
• a file-system "device" — a transport (or a sibling factory) that "knows" how to open register datastreams from a standard harp folder structure (similar to what harp-python implements currently)

harp.device also exposes a REGISTER_MAP: dict[int, type[RegisterBase[Any]]] of the core registers, which a downstream (generated) package spreads into its own map:

from harp.device import REGISTER_MAP as _CORE_REGISTER_MAP

REGISTER_MAP = {**_CORE_REGISTER_MAP, 32: DigitalInputState, 34: OutputSet, ...}

5. Data layer (harp-data)

Tabular/DataFrame concerns live in harp-data. The payload layer (harp-protocol) exposes a pandas-free, numpy-only tabular view via to_columns(), returning list[Column]:

@dataclass(frozen=True, slots=True, eq=False)
class Column:
    name: str
    data: NDArray[Any]
    categories: Any | None = None   # labels if enum-backed (data holds codes), else None

harp-data turns those columns into a DataFrame — building pd.Categorical.from_codes for enum columns, which keeps the integer codes and copies nothing:

from harp.data import read_dataframe, to_dataframe

# register + source (path / bytes / open binary file) → DataFrame
df = read_dataframe(OperationControl, "OperationControl.bin", timestamp=True)

# or, from an already-parsed payload
_data, ts, _msg, payload = OperationControl.parse_bulk(buf)
df = to_dataframe(payload)

This keeps the heavy/optional pandas dependency out of the protocol and device layers, and gives harp-data room to grow into a home for further analysis utilities.

Also, this opens the door to future implementations (e.g: arrow), using the same protocol exposed interface.

6. Code-generation affordances

A downstream package for a specific device only needs to emit register/payload class declarations — the entire declaration is mechanical and driven directly by device.yml:

# Generated from device.yml — no boilerplate beyond this
class AnalogDataPayload(StructPayload[np.int16]):
    analog_input0 = Field(converter=IdentityConverter(np.int16), offset=0)
    encoder       = Field(converter=IdentityConverter(np.int16), offset=1)
    analog_input1 = Field(converter=IdentityConverter(np.int16), offset=2)

class AnalogData(RegisterBase[AnalogDataPayload]):
    address:       ClassVar[int]                     = 44
    payload_type:  ClassVar[PayloadType]             = PayloadType.S16
    payload_class: ClassVar[type[AnalogDataPayload]] = AnalogDataPayload

The generator emits one Field/BitFlag/GroupMask per payload member, one RegisterBase subclass per register, and (for a device package) a Device subclass with __whoami__ plus a spread REGISTER_MAP. No per-device framework extension is required; the output is pure application code that composes the DSL, and the constructor is type-hinted for free.

---

[MicBoucinha]

Hi @bruno-f-cruz,

Regarding what we spoke last time on the meeting, here his the explanation on how things are working right now.

Everything is based on PayloadType. We use this as the source of truth as defined in the protocol.

Single-message parsing

Single-message parsing occurs in 4 steps:

• Step 1 - get PayloadTypefrom the message bytes, extracts its size, skipping timestamp if it exists and calculates the element count.

• Step 2 - Uses struct_charto get the structformat char directly from the PayloadType's bit flags

_UNSIGNED_CHARS = {1: "B", 2: "H", 4: "I", 8: "Q"}

def struct_char(self) -> str:
    if self.is_float():
        return "f"
    c = _UNSIGNED_CHARS[self.type_size()]
    return c.lower() if self.is_signed() else c

• Step 3 - Creates a struct.Struct(f"<{count}{char}") and calls .unpack(raw). From there it returns plain Python int, float or list.

• Step 4 - RegisterBase.parse(message) calls cls.decode(message.payload) to convert the raw scalar/list into the register's typed representation (an IntEnum, a StructPayload dataclass, etc.).

Bulk dump loading (harp-data)

Everything also occurs in 4 steps:

• Step 1 - load_register_dump()receives a file path and a RegisterBase subclass.

• Step 2 - Calls numpy_scalar_dtype(payload_type) where it will pick up the PayloadTypeand convert it to a corresponding np.dtype. This is done by reading the type_size(), is_float() and is_signed()methods from PayloadType.

def numpy_scalar_dtype(payload_type):
    itemsize = int(payload_type.type_size())
    if payload_type.is_float():
        return np.dtype("<f4")
    kind = "i" if payload_type.is_signed() else "u"
    return np.dtype(f"<{kind}{itemsize}")

• Step 3 - Constructs the complete structured frame's dtype that describes the entire frame layout, using build_frame_dtype()

• Step 4 - Reads the file's raw bytes with np.fromfile / np.frombuffer, then creates a zero-copy view: raw.view(frame_dtype).

Notes

It is important to note, that Complex registers also go through this. The PayloadTypeis there, and it is well defined.

Both paths derive from the same three PayloadType properties: type_size(), is_float(), is_signed(). One path produces struct format chars for single-message parsing, the other produces numpy dtype strings for bulk analysis. They cannot drift because neither maintains a separate mapping since both read the PayloadType's bit flags.

Sure, we have two parsing paths but the cost of maintaining this seems small since we have two small functions that base themselves on the same source. It is not like we have two parallel lookup tables or something more complex to maintain here. The big advantage is that we keep numpywhere it belongs, on the data analysis side of things, while keeping the protocol and transport numpyfree since it might not exactly be available in certain scenarios as discussed.

[bruno-f-cruz]

Can you point me to Step 4 for both code paths? This is the thing that I am struggling with being "low cost". I am probably missing something very obvious, but at some point you gotta have a signature:

Let T be the payload type for the register; you will need:

decode_single(struct_or_native_type: bytes) -> T
decode_from_numpy(massive_blob: bytes) -> nd.array[T]

I still don't see how these are 0-cost. But I may be missing a common data representation between the python native types and np.dtypes. For instance, if T is a datastruct, how do you vectorize its parsing?

--- edit

Ok I found it: https://github.com/fchampalimaud/generated-pyharp/blob/5f8577f2ebdba044f88401c5d322593e0a42c8c1/src/harp-data/harp/data/harp_data.py#L220

This is the code fork I was trying to avoid maintaining. I think your argument is that it is small enough that it won't be an issue (?). Still, you are pushing complexity into this RegisterDump class to know how to parse EVERY single type (string, interface_type is bool, masks, struct field, etc...). Maybe I should have made this point a bit more clear, but the architecture of the current PR is that this conversion should be instead handled by the underlying Converter (

pyharp/src/packages/harp-protocol/src/harp/protocol/_payload_converters.py

Lines 25 to 35 in d7ed4b7

	@abstractmethod
	def decode_scalar(self, view: np.generic) -> T:
	"""Decode a 0-D structured-array element into a Python value."""
	@abstractmethod
	def decode_batch(self, view: "NDArray[np.generic]") -> Any:
	"""Decode a 1-D structured-array column into an array-like."""
	@abstractmethod
	def encode_into(self, view: NDArray[np.generic], value: T) -> None:
	"""Write a Python value back into a structured-array element."""

) or Field-like attribute instead (

pyharp/src/packages/harp-protocol/src/harp/protocol/_payload.py

Lines 349 to 356 in d7ed4b7

	def _to_batch(self) -> "_BitMaskBatch[F]":
	"""Returns the metadata for the corresponding batch type"""
	return _BitMaskBatch(
	self._mask,
	self._enum,
	slot=self._slot,
	dtype=self._dtype,
	)

). This way adding a new type does not require changing an internal function that must be aware of all other types.

So I think there are a few issues here:

1. Where does the parsing logic live (in the converter vs in the "reader-like" object)
2. You are still duplicating struct/np.dtype logic for the attributes. If we consider this to be a finite number, sure, I think what you are saying may make sense: we pay the cost once, and we don't need to "maintain it". It kinda worries me how the same code base is already forked in some places.
3. What happens if people want to add custom types on their side? How can they change this parser directly? In the current case, the converters own the conversion logic; people can override it as they wish. Here, they are stuck with what we decide to implement.

Anyway, for the sake of moving this forward, I think we should just make the decision of what we are willing to support with the current team/effort and just go with it. As @glopesdev mentioned, we can hopefully always go back and remake it if necessary in a backwards compatible way with the public DSL.

[MicBoucinha]

Hi @bruno-f-cruz,

I continue to think that having numpy as a dependency at all levels where it will be mostly used on the data analysis component is not the best architectural design solution and limits the potential usage of the library.

However, and since it seems that there is a more pressing necessity to have it released, keep your version, but I will kindly ask you to consider in the near future to try to remove the numpy dependency as an active goal. When you do, feel free to use as a base what is already available on the generated-pyharp repo I linked before if you think it will help.

For the devices themselves, including the generic one, I would ask you to consider adding convenience methods that would simply wrap your read and write methods, as an alternative access method. These are easily generated. For example for the Behavior, but would work similarly to any device:
def read_analog_data() -> AnalogDataPayload,
or a write method as:
def write_output_set(DigitalOutputs.DO2)).

These methods, in my opinion, will be extremely helpful for newcomers to the library and it eases the interaction with the device, where the full API scope of the available registers is explicit. I believe that they will also assist in the documentation.

Please let me know how I can further assist in the next steps.

Thank you for all the time dedicated to this. It is really appreciated.

[glopesdev]

@MicBoucinha Glad we're aligned on moving forward with the current approach for the release. Thanks also for all the feedback you provided. I feel we now have both a stronger implementation and benchmarks we can use to improve it going forward.

I continue to think that having numpy as a dependency at all levels where it will be mostly used on the data analysis component is not the best architectural design solution and limits the potential usage of the library.

Do you have a concrete case we could look at where the numpy dependency would be problematic? It would help to bound the exact constraints by grounding it in a real example, since the tradeoffs are hard to get right without one. The risk otherwise is that we end up designing for an idealized general case, and then find the real constraints do not fit our solution anyway.

As for the convenience methods, I think we should move that discussion to the generators repo over at harp-tech/generators#119, since it isn't critical to this PR, which is only about the core architecture.

I would suggest the next steps as:

1. merge this PR
2. update the release CI/CD to publish the packages to PyPI
3. review and merge Add Python device interface generator generators#119
4. generate the Python interface for device.behavior
5. update CI/CD so the Python interface is published automatically on a device.behavior release

After that we should be able to extend the pyharp interface generation to all device packages.

What do you think?

[glopesdev]

A pass over the branch focused on documentation, metadata, and leftover infrastructure. None of it touches the core protocol or the register and payload DSL, which look solid; the comments group into a few themes.

• Organization references and attribution, now that the repo lives under harp-tech. The fchampalimaud URLs across pyproject.toml, mkdocs.yml, and README.md move to harp-tech, the authors fields use the org identity, and the copyright notices switch to an inclusive, undated harp-tech and Contributors while keeping the historical lines.
• Documentation. Several mkdocstrings references in api/protocol.md and api/serial.md no longer resolve against the refactored packages, harp.device and harp.data have no API page yet, an example and its markdown still use the pre-rename read_dataframe, and the theme logo is pyharp-branded while the favicon already uses the standard harp logo.
• Leftover infrastructure. The docs build still aggregates device docs by cloning harp.devices, which can come out until the per-device structure is settled, and the harp-benchmarks package together with the scripts/ benchmark and exploration files sit alongside the shippable packages, so they would be clearer in a dedicated internal location.
• One item is a flag rather than a change. The distribution is now harp with harp-* components, while the repo, docs, and logo still brand as pyharp, so the project name probably wants a single deliberate decision rather than piecemeal edits here.

Happy to iterate on any of these.

[ZegCricket]

@glopesdev, just wanted to let you know that MkDocs is no longer maintained (it hasn't had a commit in 9 months; I'll try to find a blog post about it). As an alternative, the team behind the Material for MkDocs theme started developing Zensical. They are trying to reuse part of the MkDocs ecosystem, so migrating from MkDocs to Zensical is not difficult. It might be something to consider.

[glopesdev]

Approving. I went back through the full branch against every point raised in review, and it's all addressed.

• Organization references and attribution are consistent under harp-tech, with the additive LICENSE keeping the historical credits alongside harp-tech and Contributors.
• The device-docs aggregation is gone, the API pages for harp.protocol, harp.serial, harp.device, and harp.data all resolve, and the README reads cleanly for the four-package layout.
• Versioning is lockstep via setuptools-scm across every package, type checking moved to pyright, and the release pipeline publishes to PyPI through trusted publishing, with the version pinned from the release tag and a guard against an unversioned build.
• CI is green across the full matrix, all three operating systems and Python 3.11 through 3.13.

Really nice outcome, and thanks for working through all of it.

Two non-blocking reminders for the first release: tag at 0.5.0 or higher, since harp-protocol and harp-serial are already published at 0.4.0 and every package now releases at the tag version; and configure PyPI trusted publishing for each project beforehand, including pending publishers for the not-yet-created harp-device and harp-data, so uv publish can authenticate.