DataLogger

DataLogger

The DataLogger is a SysMod that records device data to on-board flash storage (LittleFS) in either CSV or JSONL format. It is designed for long-duration autonomous logging — sessions can be time-limited or run until storage is full.

Overview

• Records decoded or raw poll data from I2C bus devices
• Supports CSV (decoded, human-readable) and JSONL (raw hex, machine-readable) formats
• Per-device logging rate control (from max poll rate down to 1/day)
• Per-device attribute selection (CSV format — log a subset of fields)
• Configurable session duration with automatic stop
• Double-buffered writes to minimise latency on the bus polling task
• REST API for start, stop, status, and simulation
• Log files stored in /local/logs/ on the device filesystem

REST API

All endpoints are under /api/datalog:

Action	URL	Description
Start	datalog?action=start&config=<json>	Start a logging session with the given configuration
Stop	datalog?action=stop	Stop the current session
Status	datalog?action=status	Returns session state, file name, bytes written, sample count, statistics
Simulate	datalog?action=simulate&rate=<ms>&devices=<n>&recsize=<bytes>	Start a simulated session for testing write throughput

Start Configuration JSON

The config parameter is a JSON object:

{
  "format": "csv",
  "csvHeader": true,
  "durationMs": 600000,
  "devices": [
    {
      "bus": "I2CA",
      "addr": "76a",
      "rateMs": 10000,
      "attrs": ["gx", "gy", "gz"]
    },
    {
      "bus": "I2CA",
      "addr": "16a",
      "rateMs": 0
    }
  ]
}

Field	Type	Description
format	string	"csv" or "jsonl"
csvHeader	boolean	Include metadata comment block at top of CSV file (default: true)
durationMs	number	Auto-stop after this many milliseconds (0 = unlimited)
devices	array	Per-device configuration (see below)

Per-device fields:

Field	Type	Description
bus	string	Bus name, e.g. "I2CA"
addr	string	Device address as lowercase hex, no 0x prefix (e.g. "76a" for slot 7, I2C addr 0x6a)
rateMs	number	Minimum interval between logged records in milliseconds. 0 = log every poll result
attrs	string[]	(CSV only) Subset of attribute names to log. Omit or empty to log all attributes

Status Response

{
  "rslt": "ok",
  "state": "active",
  "file": "/logs/log_20260406_170251.csv",
  "format": "csv",
  "durMs": 35000,
  "maxDurMs": 600000,
  "bytes": 12480,
  "samples": 1560,
  "flushes": 7,
  "overflows": 0,
  "flushLatMinUs": 1200,
  "flushLatMaxUs": 8500,
  "bytesPerSec": 356
}

CSV Format

CSV files begin with an optional metadata comment block (lines prefixed with #), followed by a column header row and data rows.

Metadata Header

# Axiom Data Log
# Start (UTC): 2026-04-06 17:02:51
# Start (local): 2026-04-06 17:02:51
# Label: log
# Bus I2CA, Addr 76a: LSM6DS - gx, gy, gz, ax, ay, az
# Bus I2CA, Addr 16a: LSM6DS - gx, gy, gz, ax, ay, az

Column Layout

The first column is always time — elapsed seconds from session start, with millisecond precision (e.g. 10.050).

Remaining columns are named <busNum>_<addr>_<attrName>, e.g. 1_76a_gx.

Each data row contains values for one device only. Columns belonging to other devices are left empty (sparse rows). This avoids timestamp alignment issues when devices produce data at different rates.

time,1_76a_gx,1_76a_gy,1_76a_gz,1_76a_ax,1_76a_ay,1_76a_az,1_16a_gx,1_16a_gy,1_16a_gz,1_16a_ax,1_16a_ay,1_16a_az
0.000,,,,,,,0.00,0.00,0.00,0.34,0.00,0.00
0.010,-0.12,0.00,0.00,0.02,-0.03,-0.97,,,,,,
0.010,,,,,,,0.00,0.00,0.00,-0.03,0.00,0.00

Field values are decoded using the device type's decode function (generated from DeviceTypeRecords.json) and have divisors and addends applied. The f format specifier in the attribute definition controls the decimal precision.

JSONL Format

Each line is a self-contained JSON object. The file begins with a header record, followed by device-info records for each configured device, then data records.

Header Record

{"_t":"header","fmt":"jsonl","label":"log","startTime":"2026-04-06T17:03:48Z","startMs":128019,"config":{...}}

Device Info Record

{"_t":"devinfo","a":"76a","b":1,"ti":11,"info":{...},"pollSize":244}

Contains the full device type info JSON and poll data size, enabling offline decoding without access to the device.

Data Record

{"_t":"d","t":26,"a":"76a","b":1,"ti":11,"x":"f42c24000000feff..."}

Field	Description
_t	Record type: "d" for data
t	Timestamp — milliseconds since session start
a	Device address (lowercase hex)
b	Bus number
ti	Device type index
x	Raw poll data as hex string

The x field contains the raw poll response bytes in hex. This can be decoded offline using the attribute definitions or custom decode function from the corresponding devinfo record.

Implementation Details

Data Flow

1. At session start, registerBusCallbacks() scans all bus devices and registers a data-change callback for each device matching the session configuration.
2. The bus polling task calls deviceDataCallback() when new data is available.
3. The callback pads the raw data to the expected record size, calls the generated decode function, formats the output (CSV row or JSONL record), and appends it to the active write buffer.
4. The main loop() periodically flushes the write buffer to the file using a double-buffer swap pattern to avoid blocking the bus task.

Rate Limiting

Each device callback context tracks lastOutputTimeMs. If targetRateMs > 0, samples arriving faster than the configured rate are dropped. For FIFO devices producing multiple samples per poll, only the last sample within each rate window is emitted.

Device Discovery at Session Start

For CSV format, the logger waits up to 5 seconds at session start for all configured devices to be identified on the bus (polling every 500ms). This handles the case where a device has not yet been identified shortly after boot.

For JSONL format, retryUnregisteredDevices() is called periodically from loop() to register devices that appear after the session has started.

Sign Extension

When using custom decode functions (e.g. LSM6DS FIFO), the pseudocode assembles values as unsigned integers. The code generator automatically inserts sign-extension code for attributes with signed underlying types ("t":"<h") stored as float ("o":"float"). See Device Type Record Format — Automatic Sign Extension.

Timestamp Handling

The decode state is pre-seeded with the current timestamp wrap offset when callbacks are registered. This ensures that 16-bit timestamps from FIFO data are correctly unwrapped even if wraps occurred before the logging session started.

SysType Configuration

The DataLogger is added as a SysMod in the SysType JSON:

{
  "DataLogger": {
    "enable": true,
    "logDir": "/logs"
  }
}

Field	Type	Default	Description
enable	boolean	true	Enable the DataLogger SysMod
logDir	string	"/logs"	Directory on the filesystem for log files

Log File Management

Log files are stored in /local/logs/ and named <label>_<YYYYMMDD>_<HHMMSS>.<format> (e.g. log_20260406_170251.csv).

Files can be listed, downloaded and deleted via the standard Raft file API (/api/filelist/local/logs, /api/fileread/local/logs/<name>, /api/filedelete/local/logs/<name>).

See Also

• Device Type Record Format — device definitions used for decoding
• Device Data Publishing — real-time data publishing (complementary to logging)
• Device Manager — bus and device management