feat: migrate away from Arduino#438
Open
hhvrc wants to merge 102 commits into
Open
Conversation
I'd prefer to be able to distinguish between the different types easily in support chat.
…o feat/arduino-3.0
Merge develop into feat/arduino-3.0. Replace remaining tcb::span with std::span, add missing WiFi.h include. Build not yet clean — remaining issues: Serial namespace rename (SerialCmds), USBSerial API, and CaptivePortalConfig struct changes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove the committed .claude/ settings and hook, .mcp.json, and the BEFORE_MERGE_WORK migration notes. Stop tracking .vscode/settings.json now that .gitignore covers .vscode/ (the local file stays on disk).
Restore the proven parallel-then-link pipeline shape. build-firmware now compiles only the code partitions (app, bootloader, partition-table) with no frontend/static0 coupling, so it runs concurrently with build-staticfs, which packs the board-independent static0 image once. A new merge-partitions job links each board's partitions with the shared staticfs into the single flashable binary via scripts/merge_image.py (esptool merge-bin at the ota_4mb.csv offsets; bootloader 0x0/0x1000 by chip) - no full ESP-IDF needed. Restores the firmware_staticfs artifact and the CDN's per-board staticfs copy.
install-esp-idf-action only sets IDF_PATH; scripts/build.py then invoked idf.py
with the system Python, which idf.py rejects ("not spawned within an ESP-IDF
shell environment"), failing every board build. Source export.sh in setup-esp-idf
and persist the venv-activated PATH/IDF_PYTHON_ENV_PATH to GITHUB_ENV so python
and idf.py resolve to the ESP-IDF venv in later steps. Install littlefs-python
into that venv for gen_staticfs.py (full compiledb build); leave esptool to
idf.py's bundled copy. Keep the ~/.espressif tools cache (it also captures the
venv, so the littlefs-python install is a no-op on a hit).
Sourcing export.sh failed under the action's EIM-managed install ("tool
esp-rom-elfs has no installed versions"): EIM manages tools in a way the
classic export.sh does not see. The action's intended entry point is the
idf.py wrapper it puts on PATH, which activates the EIM tool + venv
environment itself. Make scripts/build.py prefer that wrapper on POSIX and
fall back to `python tools/idf.py` for an already-active shell (Windows local).
setup-esp-idf drops the export.sh/venv-activation dance (just installs +
caches, broadened to /tmp/esp where the action actually installs).
build-compilationdb builds only `app` (still emits compile_commands.json and
a CodeQL compile pass) so it needs no staticfs/frontend/littlefs. Build steps
call python3 since setup-python no longer provides `python`.
The cpp-linter job crashed with KeyError: 'patch'. GitHub returns 406 for this PR's ~27k-line diff, dropping cpp-linter into a paginated fallback that dereferences file['patch'] on files GitHub omits a patch for. Scan the source tree (files-changed-only: false) instead of the diff API to sidestep the diff-size limit. Disable clang-tidy (tidy-checks: '-*'): the native ESP-IDF compile DB is generated by xtensa-esp32s3-elf-gcc, whose flags LLVM clang-tidy rejects, and standalone headers have no compile command so their ESP-IDF includes don't resolve - both surface as spurious clang-diagnostic-errors. clang-format still gates style and CodeQL covers static analysis; the compile DB build stays so tidy can be re-enabled once it is clang-toolchain compatible. Ignore build/ so the tree walk skips generated sources (CMake compiler probes, sdkconfig.h), and wrap the hand-aligned DNS answer byte grid in clang-format off/on so it passes clang-format.
Move the application out of the leftover PlatformIO src/ + include/ split into auto-discovered components: - main/ holds only main.cpp (the app_main entry). - components/core/ holds the interconnected subsystem tangle (config, serialization, wifi, captive portal, gateway, ota, http/JsonAPI, message handlers, radio, visual, estop, command handler). These form a single dependency cycle (an SCC), so they cannot be separate components. - components/serial/ holds the serial console + command handlers, the one cleanly separable piece (nothing in core depends on it). Drop the now-dead EXTRA_COMPONENT_DIRS/PlatformIO scaffolding from the root CMakeLists, move the esp_websocket_client managed dep and Kconfig.projbuild into core, and repoint generated flatbuffer headers to components/core/include/serialization/_fbs. Builds clean on Wemos-D1-Mini-ESP32 and Seeed-Xiao-ESP32C3 (IDF 6.0.2).
Remove src/LanguageVersionCheck.cpp (a translation unit of #error directives asserting a C++23 compiler) and its registration from the common component.
Phase 0 encapsulation cleanups (no behavior change): - Delete dead includes: estop/EStopManager.h from VisualStateManager.h (header uses no EStop symbol), config/Config.h from WSGateway.cpp (no Config:: use), and esp_wifi_types.h from WSLocal.h (wifi_auth_mode_t reaches consumers via wifi/WiFiNetwork.h). - Demote .cpp-only deps to PRIV_REQUIRES so they stop propagating to consumers: concurrency (logging), dns_server (util/logging/lwip; declare freertos), and rfc8908 (logging/osjson; drop unused common). Public REQUIRES now list only the deps that actually appear in each component's public headers.
Split the serial subsystem into a foundation-layer transport and the app console, and route logging through raw serial writes instead of C stdio: - components/serial (NEW, foundation): OpenShock::Serial owns the raw console byte I/O (UART0 / USB-Serial-JTAG). Init/Read as before, plus Write(). Depends only on the IDF console drivers, so it must not use OS_LOG (Init reports failure via its return value). Write() falls back to the ROM serial output before the driver is installed, so early-boot logs survive. - components/serial_console (renamed from the old app-level `serial`; `console` is a reserved IDF component name): SerialInputHandler + command handlers, now PRIV_REQUIRES core + serial. - logging: openshock_log_printf formats the line and writes the bytes via Serial::Write instead of vprintf/stdout. Logging.h drops its unused <esp_log.h>, so it carries no hardware include (matching its own doc comment). Removing <esp_log.h> from Logging.h dropped a transitive <string.h>; add the <cstring> that HTTPRequestManager, PartitionUtils, CommandHandler, WiFiCredentials and RFTransmitter actually rely on. Builds on Wemos-D1-Mini-ESP32 (UART) and Seeed-Xiao-ESP32C3 (USB-Serial-JTAG).
Add a linux-target Unity host_test for the RF protocol encoders — the highest-value pure-logic pocket in the firmware, previously with zero off-hardware coverage. It compiles the encoder sources directly with a stub <driver/rmt_types.h> (just rmt_symbol_word_t) and no-op Logging.h, so it needs neither esp_driver_rmt nor the serial transport. Coverage (11 cases): EncodeBits<N> MSB-first bit packing; CaiXianlin frame layout decoded back from the RMT symbols (id/channel/type/intensity/checksum), intensity clamping, and the Sound-mutes-intensity quirk; Sequence model dispatch across all five shocker models. Run: cd components/protocols/host_test && idf.py --preview set-target linux && idf.py build then execute build/protocols_host_test.elf (all 11 pass).
Phase 0.5: stop core's captive-portal and gateway internals from leaking heavy IDF headers into its public surface. - Move CaptivePortalInstance.h (pulled esp_http_server / dns_server / fs) and GatewayClient.h (pulled esp_websocket_client / esp_event) out of include/ into src/, alongside their .cpp — both are consumed only within core/src. Same for the message_handlers internals (ShockerCommandList.h, impl/WSGateway.h, impl/WSLocal.h); only WebSocket.h stays public. - Add PRIV_INCLUDE_DIRS "src" so those internal headers still resolve. - Demote esp_http_server, fs, dns_server, rfc8908 and esp_websocket_client to PRIV_REQUIRES now that no public header pulls them, so they no longer propagate to core's consumers (main, serial_console). Public core headers now carry only lightweight includes. Builds on Wemos-D1-Mini-ESP32 and Seeed-Xiao-ESP32C3.
- Annotate set_failed as NoReturn.
- Collapse repeated separators in sanitize() so names don't yield doubled dashes.
- Parse the branch from a non-PR ref with split('/', 2) so refs containing extra
slashes keep their full tail.
hhvrc
commented
Jul 10, 2026
Contributor
Author
There was a problem hiding this comment.
-DDARDUINO_USB_CDC_ON_BOOT=1 was dropped here, need to investigate this.
hhvrc
commented
Jul 10, 2026
Contributor
Author
There was a problem hiding this comment.
-DDARDUINO_USB_CDC_ON_BOOT=1 was dropped here, need to investigate this.
Add coverage for three untested pure-logic primitives (P1.2): Checksum::Sum8 over spans/integrals/structs plus the ReverseNibble lookup tables, TinyVec growth/ zero-fill/assign/move/span semantics, and the FnProxy member-function trampoline. Also unbreak the util host_test on IDF 6.0: hal/gpio_types.h moved from `hal` to `esp_hal_gpio`, so Convert.h no longer resolved (REQUIRES fixed). And drop the OS_PANIC/OS_PANIC_OTA defines from the stub Logging.h — they also live in the stub Panic.h, and TinyVec (the new test) is the first source to pull both, which double-defined them under -Werror. 42 tests pass (was 30) via idf.py --preview set-target linux.
Hoist the OtaUpdateStep -> FirmwareBootType mapping out of OtaUpdateManager's boot path into a pure constexpr OpenShock::InferFirmwareBootType(OtaUpdateStep) in the common enum header, and host-test every step's mapping (P1.3). OtaUpdateManager now calls it instead of the inline switch. This does not remove GatewayClient's dependency on OtaUpdateManager (it still needs the boot-time cached value from GetFirmwareBootType()); the win here is host-testable boot-type logic. common host_test: 14 pass. Firmware builds on Wemos-D1-Mini-ESP32.
…component Move core/include/serialization/_fbs/*_generated.h into a new low-level `serialization` component (REQUIRES only flatbuffers) and point scripts/generate_schemas.py at its include tree. The include prefix (serialization/_fbs/) is unchanged, so no consumer #include edits — only core's REQUIRES gains `serialization`. These generated wire types are shared across config, wifi, the message codecs and the handlers; pulling them out of core into a single flatbuffers-only layer that everything depends on one-way is the prerequisite for extracting config (and later the codecs themselves) from core without a schema dependency cycle. The component is named `serialization` because it is the start of that layer — the codecs (JsonAPI/JsonSerial/WSGateway/WSLocal) move here in a later step. Builds on Wemos-D1-Mini-ESP32.
Move config/ out of core into components/config. Now that the generated schemas are their own `serialization` component, config depends only on low-layer components — common, osjson, util, serialization (HubConfig fbs), flatbuffers, esp_wifi (wifi_auth_mode_t) and esp_hal_gpio in its public headers; chipset, fs, logging, panic and concurrency privately — and nothing in core. core REQUIRES config one-way. No forward-declare/invert was needed: extracting the schemas already broke the config<->serialization cycle, so this is a clean move + CMakeLists. Public config headers still expose the HubConfig flatbuffer (tightening that to a forward-declare is a separate encapsulation step). This unblocks the device_control extraction, whose only upstream-core edge was config. Builds on Wemos-D1-Mini-ESP32 and Seeed-Xiao-ESP32C3.
…rom core Move the hardware-facing actuators and inputs — EStopManager, RFTransmitter, VisualStateManager and the shocker CommandHandler — out of core into a new components/device_control. Their only upstream-core dependency was config, now its own low-layer component, so device_control depends only on config + the foundation/ driver layer and nothing else in core. Wiring: core PRIV_REQUIRES device_control (estop/CommandHandler/visual used only in core .cpp — captiveportal/message_handlers/gateway); main and serial_console REQUIRE it directly (they include those headers). No core public header referenced them, so it's a one-way edge with no cycle. Public headers need common/esp_hal_gpio/esp_driver_rmt/freertos; everything else (config, chipset, events, led_drivers, protocols, esp_wifi/event/netif, ...) is private. Builds on Wemos-D1-Mini-ESP32 and Seeed-Xiao-ESP32C3.
FnProxy: replace the four hand-written cv/noexcept partial specializations with a single shared Body::call trampoline plus four thin Deduce specializations that only pattern-match the qualifier combination. Same capability (every non-ref-qualified member function), body written once, no macros. FnProxy<&C::m> is still a plain R(*)(void*, Args...) function pointer for C callback APIs. DigitCounter: constrain Digits10Count with std::integral and tidy the magnitude/ sign handling. Kept pure arithmetic - no buffer, no allocation. util host_test 42/42 (incl. a const-noexcept case); builds on Wemos-D1-Mini-ESP32 and Seeed-Xiao-ESP32C3.
Body gains a bool NX parameter and declares call() noexcept(NX), so a noexcept member function now yields a noexcept function pointer (it still converts to a plain one for C callback APIs). The const specializations bind C as `const C` so the cast matches the method's constness rather than casting const away. host_test adds static_asserts pinning the deduced pointer types (a plain method stays plain, a const-noexcept method gains noexcept). 42/42; builds on Wemos-D1-Mini-ESP32 and Seeed-Xiao-ESP32C3.
Each ConfigBase::ToJSON now emits its own JSON object rather than writing properties into the parent's already-open object, mirroring the source firmware where every config creates its own cJSON object and the parent attaches it under a key. ToJSON gains a const char* name parameter: non-null adds the object as a named member (push/pop), null emits an anonymous object (start/end) for the document root and array elements. Adds JSON::objBegin/objEnd helpers to pick the right pair. Output JSON is unchanged.
Reorganize the low-level component layer and fix the config JSON layer that no longer compiled on this branch. - config: rewrite each *Config::FromJSON to use the new JsonView tryGetX API instead of the removed Internal::Utils::FromJson* helpers; fix the broken FromJsonGpioNum (typo + missing key lookup). - osjson: JsonView integer getters now parse via OpenShock::Convert (strict - rejects leading zeros, '+' prefixes, and overflow) instead of from_chars. - merge StringUtils into common as StringHelpers. - merge the panic component into logging (OS_PANIC* now live in Logging.h). - merge util and concurrency into common; repoint every REQUIRES accordingly. - split the mbedTLS-dependent helpers (Hashing, Base64) into a new crypto component so common stays mbedTLS-free. - add scripts/gen_dep_graph.py to emit a Mermaid component-dependency graph.
hhvrc
marked this pull request as ready for review
July 13, 2026 22:18
Contributor
Change file check✅ OK — 1 valid change file(s) added in this PR. |
Replace the Arduino-style one-byte getter (nextSerialByte) with an index-based scanner over the staging chunk. Serial::Read already batches the transport read; only the parser was still consuming a byte at a time. tryReadSerialLine now walks the chunk with a local index, handling control bytes individually and bulk-appending runs of printable characters via a new SerialBuffer::append. skipSerialWhitespaces scans the chunk directly as well. Behavior is byte-identical to the previous implementation (verified against the old parser across backspace/tab/control-byte/overflow cases).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.