Feature/merging robs work on offline data logging

devdocs/devbin-backwards-compatibility.md

@@ -0,0 +1,105 @@
+# Devbin Backwards Compatibility
+
+## Context
+
+RaftJS is used by both Axiom and Cog. Axiom was already using the current devbin payload layout, but Cog firmware v1.9.5 is already in production and publishes device data using an older RaftCore record body layout. The app therefore has to keep supporting the newer Axiom/Cog format while remaining compatible with Cog v1.9.5 devices in the field.
+
+The failure mode seen in Axiom Experiment App was:
+
+- Cog connected successfully over BLE.
+- The live preview received binary device messages.
+- Accelerometer data did not appear.
+- The browser logged malformed sample warnings and previously could surface errors such as `RangeError: Offset is outside the bounds of the DataView`.
+
+The root cause was a format mismatch. RaftJS assumed that devbin records contained a device sequence byte followed by length-prefixed samples. Cog v1.9.5 sends fixed-size raw samples without the device sequence byte. When parsed as the newer format, the first timestamp byte was interpreted as a sample length, so the parser read the wrong boundaries and eventually tried to decode attributes past the end of a sample.
+
+## Supported Record Layouts
+
+### Current Format
+
+Current devbin frames use:
+
+```text
+[msgType:2]
+[devbin envelope: magic/version, topicIndex, envelopeSeq]
+[recordLen:2]
+[statusBus:1]
+[address:4]
+[devTypeIdx:2]
+[deviceSeq:1]
+[sampleLen:1][sampleData:sampleLen]...
+```
+
+`sampleData` contains the poll-result timestamp followed by the device payload. This is the format used by current Axiom builds and newer Raft firmware.
+
+### Cog v1.9.5 Legacy Format
+
+Cog v1.9.5 uses the older record body:
+
+```text
+[msgType:2]
+optional [devbin envelope]
+[recordLen:2]
+[statusBus:1]
+[address:4]
+[devTypeIdx:2]
+[timestamp:2][payload:fixedSize]...
+```
+
+There is no per-device sequence byte and no per-sample length byte. Samples are decoded using the fixed payload size derived from the device type metadata.
+
+In testing, Cog v1.9.5 was observed sending a hybrid shape: the newer `DB` devbin envelope was present, but each record still used the legacy raw sample body. The parser must therefore not infer the record body format from the envelope alone.
+
+## RaftJS Parser Behavior
+
+`DeviceManager.handleClientMsgBinary` now supports two record payload modes:
+
+- `lengthPrefixed`: current records with `deviceSeq` and `[sampleLen][sampleData]`.
+- `legacyRaw`: Cog v1.9.5 records with fixed-size `[timestamp][payload]` samples and no `deviceSeq`.
+
+The parser first locates the record stream using the message prefix and optional devbin envelope. After it has the `devTypeIdx`, it fetches the device type info and validates the actual sample layout against the metadata. This lets it correctly identify the Cog v1.9.5 hybrid case where the frame has the current envelope but the record body is legacy raw.
+
+Malformed samples are bounded to their record/sample range before decoding. If a sample cannot be decoded, RaftJS skips that sample and emits a throttled warning rather than throwing from `DataView`.
+
+## Legacy Sample Size
+
+For legacy raw records, the fixed sample size is:
+
+```text
+2-byte timestamp + payload size
+```
+
+The payload size is normally derived from the sum of the attribute struct sizes in `resp.a`. If a custom response handler is used, or if the schema cannot be sized safely, RaftJS falls back to `resp.b`.
+
+This schema-derived sizing is required for Cog v1.9.5 light sensor records because that firmware reports a doubled light payload size in metadata while the actual raw record contains one fixed payload matching the attribute schema.
+
+## Direct Device Key Compatibility
+
+Cog v1.9.5 publishes multiple direct-connected devices on bus `0`, address `0`. In the current key scheme this collapses to a single `0_0` device and causes metadata collisions, for example LightSensors and Power sharing the same key.
+
+For legacy raw records only, RaftJS appends the device type index to direct bus/address zero records:
+
+```text
+0_0_<devTypeIdx>
+```
+
+This keeps legacy Cog direct devices distinct while preserving the existing key behavior for Axiom and newer length-prefixed records.
+
+Command paths should use the stored `DeviceState.busName` and `DeviceState.deviceAddress`, not only parse the displayed device key. This avoids sending commands to an address such as `0_2` when the compatibility key is `0_0_2`.
+
+## Verification
+
+The compatibility behavior is covered by `src/RaftDeviceManager.test.ts`:
+
+- current length-prefixed records decode correctly
+- Cog v1.9.5 raw accelerometer records decode correctly
+- Cog v1.9.5 raw records inside a devbin envelope decode correctly
+- legacy direct devices with bus/address `0_0` stay distinct by device type index
+
+The real-device validation used Axiom Experiment App:
+
+- Cog v1.9.5 connected over BLE as `Robotical Cog`
+- live `MXC400xXC` `ax`, `ay`, and `az` samples appeared in simple mode
+- no page errors were observed during the live-preview watch
+- an Axiom real-device connection still decoded live LSM6DS data, confirming the current length-prefixed path remained intact
+

notes/web-ble-reconnect-retry.md

@@ -0,0 +1,69 @@
+# Web BLE Reconnect Recovery
+
+Date: 2026-04-29
+
+## Issue
+
+The Axiom offline data logging UI can be refreshed while Web Bluetooth is still
+connected. In that path the browser may report `gatt.connect()` success and then
+disconnect during `getPrimaryService()`, producing an error like:
+
+```text
+GATT Server is disconnected. Cannot retrieve services. (Re)connect first with `device.gatt.connect`.
+```
+
+Before this change, `RaftChannelBLE.web.ts` treated a failed primary-service
+lookup as a terminal failure and returned `false` immediately. That meant a
+transient reconnect race could fail the whole app-level connection attempt even
+though the device was still running and still advertising.
+
+The harsher path is a browser hard refresh while BLE is still connected. During
+page unload, the normal app disconnect path cannot reliably complete because it
+sends a graceful BLE command and waits asynchronously. If the page disappears
+before that finishes, the browser can keep the GATT connection in a half-closed
+state long enough for the next immediate reconnect to fail.
+
+## Solution
+
+`RaftChannelBLE.connect()` now keeps the existing connection retry loop active
+for primary-service lookup failures:
+
+- If no supported primary service is found and more connection attempts remain,
+  it disconnects any still-open GATT connection.
+- It waits briefly before retrying.
+- It only returns `false` after the final service lookup attempt fails.
+
+This preserves the previous final failure behavior while allowing transient
+Web Bluetooth/GATT cleanup races to recover inside the raftjs channel.
+
+`RaftConnector` also now exposes `disconnectForPageUnload()`. It is deliberately
+smaller than the normal `disconnect()` path:
+
+- It disables automatic lost-connection retry.
+- It detaches the current channel from the connector immediately.
+- It starts the channel-level GATT disconnect without waiting for the normal
+  graceful BLE command sequence.
+
+The Axiom app uses this from `beforeunload`/`pagehide` so a browser hard refresh
+still starts Web Bluetooth cleanup before the page is replaced. Normal user
+disconnects continue to use the existing graceful path.
+
+## Validation
+
+Validation was done from `Axiom-Experiment-App` against real Axiom hardware
+(`Axiom009_adcf1e`) with firmware serial logs open on `/dev/cu.usbmodem2101`.
+
+The diagnostic flow was:
+
+1. Connect over Web Bluetooth.
+2. Start Axiom offline data logging for LSM6DS at 1 Hz.
+3. Hard-refresh the browser without issuing the app-level Disconnect command.
+4. Immediately reconnect over Web Bluetooth.
+5. Confirm the offline logger is still active and points to the same log file.
+6. Stop the session and delete the generated e2e log.
+
+The firmware logs showed no reboot or panic during the refresh/reconnect flow.
+The logger stayed active across the browser refresh. The page-unload run showed
+the Axiom app calling the immediate GATT disconnect path, the firmware observing
+`BLE connection change isConn NO`, and reconnect reporting the same active log
+with additional samples.

src/RaftAttributeHandler.ts

@@ -23,12 +23,13 @@ export default class AttributeHandler {
     private POLL_RESULT_RESOLUTION_US = 100;
 
     public processMsgAttrGroup(msgBuffer: Uint8Array, msgBufIdx: number, deviceTimeline: DeviceTimeline, pollRespMetadata: DeviceTypePollRespMetadata,
-                        devAttrsState: DeviceAttributesState, maxDataPoints: number): number {
+                        devAttrsState: DeviceAttributesState, maxDataPoints: number, msgEndIdx = msgBuffer.length): number {
 
         // console.log(`processMsgAttrGroup msg ${msgHexStr} timestamp ${timestamp} origTimestamp ${origTimestamp} msgBufIdx ${msgBufIdx}`)
+        const boundedMsgEndIdx = Math.min(Math.max(msgEndIdx, msgBufIdx), msgBuffer.length);
 
         // Extract msg timestamp
-        const { newBufIdx, timestampUs } = this.extractTimestampAndAdvanceIdx(msgBuffer, msgBufIdx, deviceTimeline);
+        const { newBufIdx, timestampUs } = this.extractTimestampAndAdvanceIdx(msgBuffer, msgBufIdx, deviceTimeline, boundedMsgEndIdx);
         if (newBufIdx < 0)
             return -1;
         msgBufIdx = newBufIdx;
@@ -41,7 +42,7 @@ export default class AttributeHandler {
         if ("c" in pollRespMetadata) {
 
             // Extract attribute values using custom handler
-            newAttrValues = this._customAttrHandler.handleAttr(pollRespMetadata, msgBuffer, msgBufIdx);
+            newAttrValues = this._customAttrHandler.handleAttr(pollRespMetadata, msgBuffer, msgBufIdx, boundedMsgEndIdx);
 
             // Apply per-attribute transforms that the custom handler doesn't handle
             for (let attrIdx = 0; attrIdx < pollRespMetadata.a.length && attrIdx < newAttrValues.length; attrIdx++) {
@@ -88,10 +89,9 @@ export default class AttributeHandler {
                 // console.log(`RaftAttrHdlr.processMsgAttrGroup attr ${attrDef.n} msgBufIdx ${msgBufIdx} timestampUs ${timestampUs} attrDef ${JSON.stringify(attrDef)}`);
 
                 // Process the attribute
-                const { values, newMsgBufIdx } = this.processMsgAttribute(attrDef, msgBuffer, msgBufIdx, msgDataStartIdx);
+                const { values, newMsgBufIdx } = this.processMsgAttribute(attrDef, msgBuffer, msgBufIdx, msgDataStartIdx, boundedMsgEndIdx);
                 if (newMsgBufIdx < 0) {
-                    newAttrValues.push([]);
-                    continue;
+                    return -1;
                 }
                 msgBufIdx = newMsgBufIdx;
                 newAttrValues.push(values);
@@ -107,7 +107,7 @@ export default class AttributeHandler {
         // Check if any attributes were added (in addition to timestamp)
         if (newAttrValues.length === 0) {
             console.warn(`DeviceManager msg attrGroup ${JSON.stringify(pollRespMetadata)} newAttrValues ${newAttrValues} is empty`);
-            return msgDataStartIdx+pollRespSizeBytes;
+            return -1;
         }
 
         // All attributes must have the same number of new values
@@ -175,6 +175,11 @@ export default class AttributeHandler {
                 const alpha = deviceTimeline.emaCalibrationPolls < 20 ? 0.3 : 0.05;
                 deviceTimeline.emaIntervalUs = alpha * instantIntervalUs
                                               + (1.0 - alpha) * deviceTimeline.emaIntervalUs;
+            } else if (numNewDataPoints === 1) {
+                const instantIntervalUs = timestampUs - deviceTimeline.emaPrevPollTimeUs;
+                if (Number.isFinite(instantIntervalUs) && instantIntervalUs > 0) {
+                    deviceTimeline.emaIntervalUs = instantIntervalUs;
+                }
             }
             deviceTimeline.emaPrevPollTimeUs = timestampUs;
             deviceTimeline.emaCalibrationPolls++;
@@ -186,7 +191,9 @@ export default class AttributeHandler {
             ? deviceTimeline.timestampsUs[deviceTimeline.timestampsUs.length - 1]
             : -Infinity;
         for (let i = 0; i < numNewDataPoints; i++) {
-            timestampsUs[i] = deviceTimeline.emaLastSampleTimeUs + (i + 1) * deviceTimeline.emaIntervalUs;
+            timestampsUs[i] = numNewDataPoints === 1
+                ? timestampUs
+                : deviceTimeline.emaLastSampleTimeUs + (i + 1) * deviceTimeline.emaIntervalUs;
             // Ensure monotonically increasing timestamps
             if (i === 0 && timestampsUs[0] <= lastTimeUs) {
                 timestampsUs[0] = lastTimeUs + 1;
@@ -196,7 +203,7 @@ export default class AttributeHandler {
         }
         // Advance the piecewise model cursor past all samples in this batch
         if (deviceTimeline.emaCalibrated && numNewDataPoints > 0) {
-            deviceTimeline.emaLastSampleTimeUs += numNewDataPoints * deviceTimeline.emaIntervalUs;
+            deviceTimeline.emaLastSampleTimeUs = timestampsUs[timestampsUs.length - 1];
         }
 
         // Check if timeline points need to be discarded
@@ -265,18 +272,21 @@ export default class AttributeHandler {
         }
     }
 
-    private processMsgAttribute(attrDef: DeviceTypeAttribute, msgBuffer: Uint8Array, msgBufIdx: number, msgDataStartIdx: number): { values: (number | string)[], newMsgBufIdx: number} {
+    private processMsgAttribute(attrDef: DeviceTypeAttribute, msgBuffer: Uint8Array, msgBufIdx: number, msgDataStartIdx: number, msgEndIdx: number): { values: (number | string)[], newMsgBufIdx: number} {
 
         // Current field message string index
         let curFieldBufIdx = msgBufIdx;
         let attrUsesAbsPos = false;
+        const boundedMsgEndIdx = Math.min(Math.max(msgEndIdx, msgDataStartIdx), msgBuffer.length);
+        const attrTypesOnly = attrDef.t;
+        const numBytesConsumed = structSizeOf(attrTypesOnly);
 
         // Check for "at" field which means absolute position in the buffer
         if (attrDef.at !== undefined) {
             // Handle both single value and array of byte positions
             if (Array.isArray(attrDef.at)) {
                 // Create a new buffer for non-contiguous data extraction
-                const elemSize = structSizeOf(attrDef.t);
+                const elemSize = numBytesConsumed;
                 const bytesForType = new Uint8Array(elemSize);
 
                 // Zero out the buffer
@@ -285,14 +295,16 @@ export default class AttributeHandler {
                 // Copy bytes from the specified positions
                 for (let i = 0; i < attrDef.at.length && i < elemSize; i++) {
                     const sourceIdx = msgDataStartIdx + attrDef.at[i];
-                    if (sourceIdx < msgBuffer.length) {
-                        bytesForType[i] = msgBuffer[sourceIdx];
+                    if (sourceIdx >= boundedMsgEndIdx) {
+                        return { values: [], newMsgBufIdx: -1 };
                     }
+                    bytesForType[i] = msgBuffer[sourceIdx];
                 }
 
                 // Use this buffer for attribute extraction
                 msgBuffer = bytesForType;
                 curFieldBufIdx = 0;
+                msgEndIdx = bytesForType.length;
             } else {
                 // Standard absolute position in the buffer
                 curFieldBufIdx = msgDataStartIdx + attrDef.at;
@@ -301,16 +313,15 @@ export default class AttributeHandler {
         }
 
         // Check if outside bounds of message
-        if (curFieldBufIdx >= msgBuffer.length) {
+        const attrEndIdx = curFieldBufIdx + numBytesConsumed;
+        const effectiveMsgEndIdx = Math.min(Math.max(msgEndIdx, curFieldBufIdx), msgBuffer.length);
+        if (curFieldBufIdx >= effectiveMsgEndIdx || attrEndIdx > effectiveMsgEndIdx) {
             // console.warn(`DeviceManager msg outside bounds msgBuffer ${msgBuffer} attrName ${attrDef.n}`);
             return { values: [], newMsgBufIdx: -1 };
         }
 
-        // Attribute type
-        const attrTypesOnly = attrDef.t;
-
         // Slice into buffer
-        const attrBuf = msgBuffer.slice(curFieldBufIdx);
+        const attrBuf = msgBuffer.slice(curFieldBufIdx, effectiveMsgEndIdx);
 
         // Check if a mask is used and the value is signed
         const maskOnSignedValue = "m" in attrDef && isAttrTypeSigned(attrTypesOnly);
@@ -319,9 +330,6 @@ export default class AttributeHandler {
         const unpackValues = structUnpack(maskOnSignedValue ? attrTypesOnly.toUpperCase() : attrTypesOnly, attrBuf);
         let attrValues = unpackValues as (number | string)[];
 
-        // Get number of bytes consumed
-        const numBytesConsumed = structSizeOf(attrTypesOnly);
-
         // Check if any values are strings (from 's' format) — skip numeric transforms for those
         const hasStringValues = attrValues.some(v => typeof v === 'string');
 
@@ -443,11 +451,12 @@ export default class AttributeHandler {
         return value;
     }
 
-    private extractTimestampAndAdvanceIdx(msgBuffer: Uint8Array, msgBufIdx: number, timestampWrapHandler: DeviceTimeline):
+    private extractTimestampAndAdvanceIdx(msgBuffer: Uint8Array, msgBufIdx: number, timestampWrapHandler: DeviceTimeline, msgEndIdx = msgBuffer.length):
                     { newBufIdx: number, timestampUs: number } {
+        const boundedMsgEndIdx = Math.min(Math.max(msgEndIdx, msgBufIdx), msgBuffer.length);
 
         // Check there are enough bytes for the timestamp
-        if (msgBufIdx + this.POLL_RESULT_TIMESTAMP_SIZE > msgBuffer.length) {
+        if (msgBufIdx + this.POLL_RESULT_TIMESTAMP_SIZE > boundedMsgEndIdx) {
             return { newBufIdx: -1, timestampUs: 0 };
         }
 
@@ -514,4 +523,4 @@ export default class AttributeHandler {
         return false;
     }
 
-}
+}