RaftLibraryDeveloperGuide

Raft Library Developer Guide

Building applications with Raft is documented in Quick Start and Build Process. This page covers the other side — building, testing and debugging Raft libraries themselves — and the conventions that let a library ship its own examples and tests without forking the build system.

The Raft libraries (RaftCore, RaftSysMods, RaftI2C, RaftWebServer, …) are normally consumed via FetchContent inside an application project. To make them practical to develop on, each library follows a small set of conventions:

• A raftdevlibs/ override mechanism in application projects, so a library can be edited in place without being clobbered by the build.
• An in-library unit_tests/ folder that builds the library as a Raft project, with the library itself injected as a component.
• An in-library examples/ folder containing one or more standalone Raft applications.
• An in-library linux_unit_tests/ folder for host-side tests that compile with plain g++ — no ESP-IDF needed.
• An in-library devdocs/ folder for design notes that may be promoted into this wiki.

raftdevlibs — editing libraries in place

The most important developer-assistance feature is the raftdevlibs/ override folder. It is the recommended way to develop Raft libraries against a real application.

When RaftBootstrap.cmake is about to fetch a component listed in RAFT_COMPONENTS, it first checks for:

<project-root>/raftdevlibs/<ComponentName>

If that directory exists, it is used in place of the GitHub fetch. The build prints:

============================================================
  LOCAL (DEBUG) LIBRARY: RaftCore
  Using: /path/to/project/raftdevlibs/RaftCore
  (Skipping fetch from https://github.com/robdobsn/RaftCore.git)
============================================================

Why this matters for library developers:

• No risk of overwrite. raftdevlibs/ lives at the project root, alongside main/ and systypes/. The Raft build never writes into it — fetched libraries land in build/_deps/<component>-src/, which gets blown away on idf.py fullclean. Anything you put in raftdevlibs/ survives every kind of clean.
• Real-world testing. You are running the library exactly as a downstream user would, including SysType layering, config merging, GenWebUI and partition handling.
• Per-library, not per-project. Override only the library you are working on; everything else is still fetched and pinned, so you do not accidentally drift the rest of the system.
• No code changes required. The override is detected purely from directory existence — the application's CMakeLists.txt, features.cmake and RAFT_COMPONENTS all stay untouched.

Typical setup:

my-app/
├── raftdevlibs/
│   └── RaftI2C/             ← `git clone https://github.com/robdobsn/RaftI2C` here
├── main/
├── systypes/
└── CMakeLists.txt

The folder name must exactly match the component name in RAFT_COMPONENTS (case-sensitive). Add raftdevlibs/ to .gitignore so it does not leak into the application repo. Full reference: Local Raft Development Libraries.

Tip. When iterating on multiple libraries together (e.g. a RaftCore change consumed by a new RaftSysMods feature), put both clones in raftdevlibs/. The build will use both local copies and resolve find_package/REQUIRES between them transparently.

In-library unit tests

Most Raft libraries ship a unit_tests/ folder at the repo root that builds the library as a Raft project against itself:

RaftCore/
├── components/        ← the library being tested
├── unit_tests/
│   ├── CMakeLists.txt ← bootstraps a Raft project
│   ├── main/
│   │   ├── CMakeLists.txt   ← test sources + REQUIRES RaftCore
│   │   └── *.cpp            ← Unity tests
│   └── systypes/
│       └── unittest/
│           ├── features.cmake
│           ├── partitions.csv
│           └── sdkconfig.defaults

The trick that makes this work is in unit_tests/CMakeLists.txt:

cmake_minimum_required(VERSION 3.16)

# Use the local RaftCore (parent directory) instead of fetching from GitHub
set(FETCHCONTENT_SOURCE_DIR_RAFTCORE "${CMAKE_SOURCE_DIR}/.." CACHE PATH "" FORCE)

# Bootstrap as a normal Raft project
set(BOOTSTRAP_URL "https://github.com/robdobsn/RaftCore/releases/download/v1.37.1/RaftBootstrap.cmake")
file(DOWNLOAD ${BOOTSTRAP_URL} "${CMAKE_BINARY_DIR}/RaftBootstrap.cmake")
include("${CMAKE_BINARY_DIR}/RaftBootstrap.cmake")

# Add parent directory to find components
list(APPEND EXTRA_COMPONENT_DIRS "..")

project(${_systype_name} DEPENDS ${ADDED_PROJECT_DEPENDENCIES})

Two CMake mechanisms do the heavy lifting:

1. FETCHCONTENT_SOURCE_DIR_<NAME> — a built-in FetchContent override. Setting FETCHCONTENT_SOURCE_DIR_RAFTCORE tells FetchContent_Declare(raftcore …) to use that directory verbatim instead of cloning. Here ${CMAKE_SOURCE_DIR}/.. is the library checkout itself, so the library tests its own working tree. This is the unit_tests/-folder equivalent of raftdevlibs/ — necessary because unit_tests/ lives one level below the library, and the library cannot put itself in its own raftdevlibs/ folder.
2. list(APPEND EXTRA_COMPONENT_DIRS "..") — points ESP-IDF's component scanner at the library's components/ tree, so test targets can REQUIRES RaftCore directly without involving the bootstrap registration.

The matching unit_tests/main/CMakeLists.txt then registers the test runner against Unity:

idf_component_register(
    SRCS
        "test_app_main.cpp"
        "RaftJson_test_values.cpp"
        # … other test files …
    INCLUDE_DIRS "."
    REQUIRES esp_system unity RaftCore nvs_flash
    WHOLE_ARCHIVE
)

Building and running:

cd RaftCore/unit_tests
raft run -s unittest         # build, flash, monitor on a connected device
# or, with raw idf.py:
idf.py -DSYSTYPE=unittest build flash monitor

WHOLE_ARCHIVE is important — without it the linker discards Unity test functions that are only referenced by name from the runner.

Adding a unit test to a library

1. Create the test source under <lib>/unit_tests/main/MyFeature_test.cpp using TEST_CASE macros.
2. Add the file to SRCS in unit_tests/main/CMakeLists.txt.
3. If the test needs new dependencies, append them to REQUIRES.
4. cd unit_tests && raft run -s unittest — runs on hardware via Unity.

Why not just CTest?

The Raft tests run on real ESP32 hardware (or QEMU) so they can exercise NVS, the file system, BLE/WiFi stacks and the comms channels in the same shape end-users see. Pure host-side checks live in linux_unit_tests/ (below).

In-library examples

Examples follow the same EXTRA_COMPONENT_DIRS ".." pattern as the unit tests, but are organised as full demonstrator applications under an examples/ folder:

RaftWebServer/
├── components/
└── examples/
    ├── basic/
    │   ├── CMakeLists.txt
    │   ├── main/
    │   ├── systypes/
    │   │   └── BasicWebServer/
    │   ├── Makefile
    │   ├── test.py
    │   └── dependencies.lock
    └── perftest/
        └── …

The example's top-level CMakeLists.txt is identical to a normal Raft application except it pulls in the parent library:

cmake_minimum_required(VERSION 3.16)

set(BOOTSTRAP_URL "https://github.com/robdobsn/RaftCore/releases/download/v1.37.1/RaftBootstrap.cmake")
file(DOWNLOAD ${BOOTSTRAP_URL} "${CMAKE_BINARY_DIR}/RaftBootstrap.cmake")
include("${CMAKE_BINARY_DIR}/RaftBootstrap.cmake")

project(${_systype_name} DEPENDS ${ADDED_PROJECT_DEPENDENCIES})

Each example has its own systypes/<Name>/, partitions.csv and features.cmake. To use the parent library directly (rather than re-fetching it from GitHub), append EXTRA_COMPONENT_DIRS ".." and/or FETCHCONTENT_SOURCE_DIR_<NAME> as in the unit_tests/ pattern. Some libraries (e.g. RaftI2C's TestWebUI/) also call FetchContent_Populate(raftcore) and include(${raftcore_SOURCE_DIR}/scripts/RaftProject.cmake) directly, which is useful when an example needs a WebUI build and an FSImage.

Adding an example to a library

1. Create examples/<MyExample>/ with the standard Raft layout (main/, systypes/<MyExample>/, top-level CMakeLists.txt).
2. In the CMakeLists.txt, add list(APPEND EXTRA_COMPONENT_DIRS "../..") so the parent library is available as a component.
3. List the parent library in your example main/CMakeLists.txt REQUIRES.
4. Optionally add a Makefile shortcut for raft build, raft flash etc. and a test.py for CI smoke tests.

Linux unit tests (host-side)

For tests that should not require a flashed device, several libraries ship a linux_unit_tests/ folder built with plain g++ and a Makefile. Host-side tests are useful for parsers, expression evaluation, JSON handling, device-record code generation and other CPU-only logic.

RaftI2C/linux_unit_tests/
├── Makefile               ← build with `make`
├── main.cpp               ← test driver
├── utils.cpp / utils.h    ← thin host-side stubs
├── Logger.h, esp_attr.h   ← shims for ESP-IDF symbols
├── TestDevTypeRecs.json
└── DeviceTypeRecords_generated.h  ← produced by ConvertJsonToC.py

The library's Makefile typically:

• Adds -I paths into the real component source directories (-I../components/...), so the same .cpp files compile against host stubs.
• Optionally clones RaftCore into ./RaftCore/ via python3 ../scripts/FetchGitRepo.py if the test needs RaftCore sources too.
• Runs any code-generation scripts (e.g. ProcessDevTypeJsonToC.py) before compiling.
• Builds a single executable that runs all tests when invoked.

Running:

cd RaftI2C/linux_unit_tests
make
./linux_unit_tests

Both VS Code workspace folders for RaftCore and RaftI2C ship a default make build task targeting linux_unit_tests/, so Ctrl+Shift+B runs them directly.

Use linux_unit_tests/ for fast iteration. Use unit_tests/ (Unity on hardware) when the code under test depends on FreeRTOS, NVS, network stacks or any other ESP-IDF runtime.

devdocs/ folder

Each library ships a devdocs/ folder at its repo root containing design notes, plans and investigations that are too rough or too implementation-specific for the public wiki. Examples:

• RaftCore/devdocs/bus-device-tracking-refactor-plan.md
• RaftI2C/devdocs/i2c-adaptive-yield-plan.md
• RaftSysMods/devdocs/BLE-publish-throughput-analysis.md

These are intended to be living documents — write them as you investigate or refactor, and promote the relevant parts into a wiki page once the work has stabilised. The documentation improvement plan tracks promotion candidates.

Conventions:

• Markdown only.
• One file per topic. Date-prefix or version-prefix the filename if the doc captures a specific point-in-time investigation.
• Cross-link to wiki pages with [Title](../WikiPageName) when you reference public docs from a devdoc.
• Do not duplicate API details that belong in the wiki — link to them.

Other developer-assistance features

A few smaller conveniences that come up while developing libraries:

• Pinned bootstrap URL. The BOOTSTRAP_URL in every CMakeLists.txt pins to a specific RaftCore release. To test with an in-development bootstrap, override RAFTCORE_GIT_TAG/RAFTCORE_GIT_REPO_URL or — simpler — drop a fresh RaftCore checkout into raftdevlibs/. Full mechanics: RaftBuildProcess.
• raft CLI. raft build, raft flash, raft monitor, raft run, raft debug and raft ota are wrappers around idf.py that understand SysType selection. See Raft CLI.
• dependencies.lock. Each top-level project (and each example) has a dependencies.lock at its root. Commit it to pin transitive ESP component versions; delete it to allow upgrades on the next idf.py reconfigure.
• Generated headers. Several libraries generate C/C++ headers from JSON inputs (e.g. RaftI2C/scripts/ConvertJsonToC.py, RaftCore/scripts/GenDeviceTypes.py). When editing the JSON, run the generator script — or trigger a CMake reconfigure — before rebuilding.
• Docker images. Several libraries ship a Dockerfile next to unit_tests/ that pre-bakes ESP-IDF and the Python tooling. Useful for CI and reproducible builds. compose.yaml (where present) wires up the dev container.
• VS Code tasks. RaftCore/linux_unit_tests/ and RaftI2C/linux_unit_tests/ define default build tasks so Ctrl+Shift+B builds the host-side tests of whichever workspace folder is active.
• raft debug. Streams the device's log buffer over the network without taking the serial port, including across BLE and WiFi. See Remote Logging.

See also

• Local Raft Development Libraries — the raftdevlibs/ override mechanism in detail.
• Raft Build Process — bootstrap stages, RAFT_COMPONENTS, FetchContent flow.
• Raft CLI — raft build / flash / run / debug / ota.
• SysTypes — variant configuration consumed by every example, test and application.
• Writing Your First SysMod — the application-side equivalent of this guide.