feat: add data_load_mf4 — ASAM MF4/MDF4 file loader (mdflib + DBC CAN decode)#180
Open
facontidavide wants to merge 16 commits into
Open
feat: add data_load_mf4 — ASAM MF4/MDF4 file loader (mdflib + DBC CAN decode)#180facontidavide wants to merge 16 commits into
facontidavide wants to merge 16 commits into
Conversation
Net-new ASAM MF4/MDF4 file DataSource plugin. Scope: numeric+string+enum channels, DBC-based CAN decode, channel-selection dialog, synthetic+real test fixtures. Backed by ihedvall/mdflib (MIT) and dbcppp (MIT). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Multi-model fusion-plan council (winner: claude risk-first, 31/31) + a
dedicated Codex read-only critique (council's Codex seat had stalled).
Codex caught a Critical mdflib-CPM build breakage (${CMAKE_SOURCE_DIR}
resolves to the parent project), CAN timestamp/ID grounding fixes, the
per-data-group memory reality, field-name collisions, no-master groups,
and that dbcppp is a static lib pinning boost/1.80 -> prefer dbc_parser_cpp.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Proven against ASAP2_Demo (finalized, measurement) + canedge (UnFinMF, 139k CAN frames): mdflib reads both, unfinalized transparently (risk retired), CanBusObserver+CanId/Timestamp work. New finding: mdflib needs <algorithm>/<cstdint> patch to compile on gcc 15 (public header idatawriter.h). Both patches now mandatory in the CPM integration. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New data_load_mf4 plugin (FileSourceBase stub) with mdflib vendored via CPM
(DOWNLOAD_ONLY + self-built mdf target from globbed sources, sidestepping
mdflib's ${CMAKE_SOURCE_DIR} breakage). gcc-15 needs SHELL:-include
algorithm/cstdint (CMake de-dups plain -include). Conan expat is expat::expat
-> alias shim. Builds -Werror clean to libmf4_source_plugin.so (3.5MB); test
passes; self-contained (no external DT_NEEDED, own symbols hidden,
-Bsymbolic-functions). Wired into both root CMake blocks + root conanfile
(zlib/1.3.1 matches arrow, expat/2.6.4).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pure ChannelDataType -> PJ::PrimitiveType mapping: numeric (int/float, any endian) -> kFloat64 (CC-applied engineering doubles); text -> kString; byte-array/MIME/CANopen-date-time/complex -> kUnspecified (v1-unsupported, reader skips with a counted warning). 3 hermetic tests, all green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codex review of the committed Phase 1-2 code: - force-include gcc-15 workaround now covers GNU|Clang|AppleClang (mdflib's missing <algorithm> breaks libc++/macOS CI too, not just gcc 15). - pin the mdflib CPM tarball with URL_HASH SHA256 (reproducible/verified fetch). Deferred (documented): CANopen date/time -> kUint64, real reader tests (Phase 3), full importData (Phase 5). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mf4Reader: open()+ReadEverythingButData() -> flat group/channel index (name/unit/type/master/bus_type); readGroup() streams a group's records as rows via CreateChannelObserverForChannelGroup + ReadData + per-sample GetEngValue/GetChannelValue, then ClearData (bounded per-data-group). Absolute ts = GetStartTime() + master*1e9. Field names de-duped within a group (name/name#1). No-master groups rejected. New mf4_core static lib shared by plugin + test. 5 synthetic-fixture tests (MdfWriter Mdf4Basic), all green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codex review of Phase 3: - CRASH FIX: mdflib returns a null observer for undecodable types (Complex); readGroup dereferenced it before the kUnspecified skip. Null-check added (stays aligned with valueChannelNames since null<->unsupported). New test UnsupportedChannelIsSkippedWithoutCrash proves it with a real ComplexLe chan. - Assert absolute epoch (rows[0].ts == kStartNs), not just spacing. - Document known v1 limitation: appended multi-measurement files with distinct per-DG start times anchor later groups to the first epoch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CanDecoder decodes raw CAN frames -> named physical signals via dbc_parser_cpp (LinuxDevon/dbc_parser_cpp, MIT, no Boost, big+little endian), vendored via CPM (DOWNLOAD_ONLY + self-built 'dbc' lib + header-only fast_float, both pinned by commit+SHA256). dbc_parser_cpp fully hidden behind a pImpl -> can_decoder.hpp pulls in no DBC headers. Frames match on the masked 11/29-bit id (extended flag ignored); multiple DBCs accumulate. 4 hermetic inline-DBC tests green; plugin .so stays self-contained. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codex review of Phase 4: - HIGH: replace lossy 0x1FFFFFFF masking with proper standard/extended disambiguation. decode() now takes an extended flag; extended frames prefer the Vector-flagged (bit31) message then fall back to raw, standard frames use raw only, so a standard and extended message sharing a numeric id no longer collide. - HIGH: reject frames shorter than the message (dbc_parser_cpp zero-fills and returns success -> silent garbage). - Document upstream limitations: multiplexed signals skipped, >32-bit imprecise. Tests: +signed, +extended-vs-standard (both directions), +truncation. 7 green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Full importData(): opens the reader, warns on unfinalized files, progress via filesystem file_size, honors isStopRequested. Measurement groups -> one topic per group (collision-free names) with per-channel fields; string values use a row-scoped string_view (no dangling); invalid samples -> NullValue. CAN groups with a loaded DBC -> readCanGroup + CanDecoder -> one topic per message (CAN/<name or hex id>) with signal fields, counting unmatched frames. Groups with no master / empty / CAN-without-DBC are skipped and summarized. Added Mf4Reader::readCanGroup (CanBusObserver, ts from CanMessage Timestamp) and CanDecoder::messageName. Config persists filepath + dbc_paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Drive Mf4Reader directly on the real fixtures (no plugin_host harness). ASAP2_Demo_V171.mf4: 7 groups, finalized, reads rows+series with real CC conversions. canedge_00000001.MF4: unfinalized (UnFinMF) reads fine, CAN frames extracted incl. 29-bit extended (J1939). Fixtures not committed; tests GTEST_SKIP unless MF4_TEST_DATA_DIR points at them. Verified locally: both pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mf4Dialog (PJ::DialogPluginTyped) parses the file metadata and shows a table of channel groups (name, samples, channels, bus type) plus a summary line. .ui embedded via pj_embed_ui (no Qt build dep). Wired into mf4_source (kCapabilityHasDialog + getDialog + setFilePath in loadConfig; PJ_DIALOG_PLUGIN). Plugin .so now exports both data_source and dialog vtables, -Werror clean, still self-contained. v1 display-only; interactive selection + DBC picker are documented follow-ups. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
All 8 phases done on feat/mf4-plugin. 19 tests -Werror clean on gcc 15; ASAN-clean with leak detection across all tests incl. real ASAP2 + unfinalized CANedge fixtures; self-contained .so (data_source + dialog vtables). Every phase Codex-reviewed; final orchestrator/dialog review found no high-confidence bugs. Documented follow-ups: aggregate build, pj_proto_app E2E smoke, interactive dialog, and the enumerated v1 scope limitations. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Documents the plugin, timestamp handling, DBC CAN decoding, and — per the integration tests being skip-if-absent — where to download the two real sample files (ASAP2_Demo_V171.mf4 from asammdf, canedge_00000001.MF4 from CSS Electronics) and how to run the tests against them via MF4_TEST_DATA_DIR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds ReadsMdf3File (mdflib Mdf3Basic writer -> Mf4Reader): confirms the version-agnostic path handles older MDF v3 files, not just v4. Backs the README claim "Reads MDF v3 and v4". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a new
data_load_mf4plugin — an ASAM MF4/MDF4 file DataSource(
.mf4,.mdf,.dat). This is net-new (MF4 was not among the original portedplugins), so the committed design spec is its baseline.
Parsing uses
ihedvall/mdflib(MIT);optional CAN decoding uses
dbc_parser_cpp(MIT). Both are vendored via CPM and statically linked, so the plugin
.sopulls in no extra runtime deps.
Features
##DZ-compressed and unfinalized (UnFinMF,e.g. CANedge) files
as engineering (CC-applied) doubles, text channels as strings
.dbcdatabases are supplied (one topic per message, standard/extended 29‑bitframes disambiguated)
Modules
mf4_value(type map) ·mf4_reader(mdflib wrapper: measurement rows + CANframes) ·
can_decoder(DBC behind a pImpl) ·mf4_source(orchestrator) ·mf4_dialog(metadata preview).Verification
1 scaffold) — green
-Werrorclean on gcc 15 (-Wconversion -Wold-style-cast -Wshadow…)demo and the unfinalized CANedge log
.so:DT_NEEDED= system libs only; own symbols hidden;exports both the data_source and dialog vtables
Design + implementation plan are committed under
docs/(superpowers/specs/and
plans/). Each phase was reviewed by an independent model and the findingsfolded in (gcc‑15/Clang include fix,
URL_HASHpins, a null‑observer crash,absolute‑epoch tests, CAN id disambiguation + frame‑truncation rejection).
Test fixtures
The two integration tests are skip‑if‑absent — they run only when
MF4_TEST_DATA_DIRpoints at the sample files. Those files are not committed(third‑party assets, above the repo's binary‑fixture budget); download links and
the exact commands are in
data_load_mf4/README.md.Before merge — please watch on first CI run
root
conanfile.py(zlib/1.3.1— matches Arrow — andexpat/2.6.4), but onlythe single‑plugin
PJ_BUILD_PLUGIN=data_load_mf4path was verified locally.pj_proto_appE2E smoke was not run (binary not available in the dev env).MF4_TEST_DATA_DIRso the integration tests execute in CI.Known v1 limitations (documented in the README)
Numeric channels import as
kFloat64(native integer types not preserved);enum→text channels, channel arrays, multiplexed CAN signals, and appended
multi‑measurement epochs are out of scope; the dialog is a read‑only preview
(interactive channel selection + DBC picker are follow‑ups).
🤖 Generated with Claude Code