From 712bde8532eb5cfaf56ad162dcd1c235cb9c12cc Mon Sep 17 00:00:00 2001 From: David Cozens Date: Mon, 11 May 2026 19:15:40 +0000 Subject: [PATCH 1/3] docs: update DEVLOG for S08.09 SolidSyslogFreeRtosTcpStream + TCP BDD --- DEVLOG.md | 93 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 93 insertions(+) diff --git a/DEVLOG.md b/DEVLOG.md index da464795..ea0c2611 100644 --- a/DEVLOG.md +++ b/DEVLOG.md @@ -1,5 +1,98 @@ # Dev Log +## 2026-05-11 — S08.09 SolidSyslogFreeRtosTcpStream + tcp_transport BDD on QEMU (#341) + +First slice of S08.06 (TCP transport on FreeRTOS-Plus-TCP). Lands the +TCP stream adapter, extends the sockets fake to cover the TCP-side +API, flips Plus-TCP into TCP mode in the BDD target, and wraps UDP+TCP +under SolidSyslogSwitchingSender so behave's `--transport tcp` flips +the active transport over the UART. `tcp_transport.feature` green +against QEMU on the first CI run; no slirp-NAT flakiness on the +bounded-connect cap. + +### Decisions + +- **Bounded blocking connect via SO_SNDTIMEO=200ms, not non-blocking + + select.** FreeRTOS-Plus-TCP does not expose non-blocking connect + with a `select` companion (no analogue of POSIX `EINPROGRESS` → + `select` → `SO_ERROR`). The choreography is therefore: socket → + setsockopt(SNDTIMEO=200ms) → connect → setsockopt(SNDTIMEO=0, + RCVTIMEO=0) on success, or closesocket on failure. 200ms is short + enough that the Service task keeps draining predictably during an + outage, long enough for a healthy peer to ACK over slirp/LAN. + Clearing both timeouts post-connect restores the non-blocking + single-call contract `SolidSyslogStream` requires of Send/Read. +- **SwitchingSender with two inners on FreeRTOS, not three.** The + BDD_TARGET_SWITCH_TLS enum slot stays in `BddTargetSwitchConfig` for + cross-platform parity, but the FreeRTOS wiring sets `senderCount = 2` + with only UDP+TCP inners. SwitchingSender's existing bound-check + falls a `set transport tls` through to its NilSender (returns false, + no crash) rather than a NULL deref — graceful degradation until + S08.07 lands real TLS on FreeRTOS. +- **Pin every argument to every FreeRTOS call in the unit tests, not + just the obvious ones.** First-pass tests covered socket/connect/ + send/recv args by value but only checked setsockopt's *option name* + via the side-effect on `LastSndTimeoSet`/`LastRcvTimeoSet`. Added + two tests pinning the setsockopt socket handle, level=0, and + optlen=sizeof(TickType_t) too, so a regression that changed the + level or optlen would fail at the bar instead of slipping through + the option-name proxy. MISRA-style argument coverage. +- **Single level of abstraction in Open/Send/Read, mirrored across all + three.** Each vtable function now reads as a three-line sentence: + guard on stream state → call the same-level helper + (`OpenSocket` / `SendOrCloseOnFailure` / `ReceiveOrCloseOnFailure`) + → return. Native FreeRTOS calls are pushed into the second layer + (`TryConnect`, `TrySend`) where they belong. Pulled the + `Try*`/`*OrCloseOnFailure` pair pattern across Send and Read for + symmetry even though Receive has different return semantics than + Send — the names carry the intent at each layer. `IsClosed` removes + the negative guard in Open. Every static now wears the + `FreeRtosTcpStream_` prefix for MISRA C 5.9 file-scope identifier + uniqueness, matching FreeRtosDatagram. +- **InteractiveTask stack *40 → *48.** StreamSender.Connect allocates + a SolidSyslogAddressStorage plus a SolidSyslogFormatter sized for + SOLIDSYSLOG_MAX_HOST_SIZE on stack; TransmitFramed adds a small + octet-prefix formatter. Empirically tipped a *40 budget into a + Cortex-M lockup (PC=0, fault-during-fault — classic + stack-overflow-corrupting-the-exception-return signature) when + SwitchingSender flipped UDP→TCP→UDP across repeated sends. *48 + restored ~4 KB headroom and the same scenario completed cleanly. +- **Test fixture: openStream() / readIntoBuffer() helpers and a + CHECK_SOCKET_CLOSED_ONCE macro.** The arrange step in 25+ tests + collapses to `openStream();`. The macro pins both the closesocket + call count *and* the identity of the closed socket — applying it + uniformly across the close-on-failure tests caught three sites + (SendClosesSocketOnError, ReadReturnsNegativeOneOnErrorAndClosesSocket, + DestroyClosesOpenSocket) that previously only checked the count. + The check count went 56 → 59 from that tightening alone. + +### Deferred + +- **CMake-driven memory scaling** for both static-`_Create` storage + sizes and FreeRTOS task stack depths. The recurring stack bumps + (*16 → *32 → *40 → *48) and the per-class `SOLIDSYSLOG_*_SIZE` + macros point to this as the right next intervention: expose them as + CMake-tunable variables so integrators size memory once, + declaratively, instead of editing `main.c` each time a new feature + lands. Captured in memory `project_freertos_stack_budget` (renamed + from *40 to *48 ceiling). +- **tcp_reconnect.feature** → S08.10 (#339, next session). +- **switching_transport.feature** (and the `switch` runtime command in + `OnSet`) → S08.11 (#340). +- **TLS on FreeRTOS** → S08.07 (#272). The + `BDD_TARGET_SWITCH_TLS` slot graceful-fallback decision above is + the bridge until then. +- **End-to-end BDD verification locally.** Docker isn't available + inside the freertos-target devcontainer, so the + `bdd-freertos-qemu` job was the real arbiter; QEMU smoke (UDP+TCP + switching) was the strongest local signal. Captured in memory + `project_behave_container_gaps`. + +### Open questions + +- None. The merged CI run was all-green first try — including the + bounded-connect over slirp NAT, which was the most-suspect path. + ## 2026-05-11 — S24.03 drop AtomicOps vtable; select atomics at link time (#326) E24 close-out story. The vtable through which AtomicCounter reached From f169c90e6f7f7b2e15b6add737547e8aefde586f Mon Sep 17 00:00:00 2001 From: David Cozens Date: Mon, 11 May 2026 19:47:39 +0000 Subject: [PATCH 2/3] feat: S08.10 tcp_reconnect BDD scenario green on FreeRTOS QEMU Split @buffered into @buffered (interactive process) + @store (file- backed store) so the FreeRTOS compose filter admits tcp_reconnect without dragging in store_and_forward / capacity_threshold / store_capacity / block_lifecycle / power_cycle_replay. Route the buffered-interactive spawn through target_driver so QEMU plus `set transport tcp` over the UART falls out for free on FreeRTOS; Linux/Windows argv path unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) --- Bdd/features/block_lifecycle.feature | 2 +- Bdd/features/capacity_threshold.feature | 2 +- Bdd/features/power_cycle_replay.feature | 2 +- Bdd/features/steps/syslog_steps.py | 79 +++++++++++++------------ Bdd/features/store_and_forward.feature | 2 +- Bdd/features/store_capacity.feature | 2 +- DEVLOG.md | 77 ++++++++++++++++++++++++ ci/docker-compose.bdd.yml | 8 ++- 8 files changed, 129 insertions(+), 45 deletions(-) diff --git a/Bdd/features/block_lifecycle.feature b/Bdd/features/block_lifecycle.feature index 352fb236..04d176b9 100644 --- a/Bdd/features/block_lifecycle.feature +++ b/Bdd/features/block_lifecycle.feature @@ -1,4 +1,4 @@ -@tcp @buffered +@tcp @buffered @store Feature: Block lifecycle The block store rotates blocks across files of fixed size. Once every record in a non-active block is sent and acknowledged, that block is disposed diff --git a/Bdd/features/capacity_threshold.feature b/Bdd/features/capacity_threshold.feature index 29a8fada..0253aaec 100644 --- a/Bdd/features/capacity_threshold.feature +++ b/Bdd/features/capacity_threshold.feature @@ -1,4 +1,4 @@ -@tcp @buffered +@tcp @buffered @store Feature: Capacity threshold alert an early-warning callback fires when the block store crosses a configured capacity threshold, before the terminal full-store callback engages. diff --git a/Bdd/features/power_cycle_replay.feature b/Bdd/features/power_cycle_replay.feature index 08ec0bc0..ed911b10 100644 --- a/Bdd/features/power_cycle_replay.feature +++ b/Bdd/features/power_cycle_replay.feature @@ -1,4 +1,4 @@ -@tcp @buffered +@tcp @buffered @store Feature: Power cycle replay from block store After a power cycle, unsent messages persisted in the block store are replayed before new messages. The collector sees old-session diff --git a/Bdd/features/steps/syslog_steps.py b/Bdd/features/steps/syslog_steps.py index 9d239fa3..08d5c758 100644 --- a/Bdd/features/steps/syslog_steps.py +++ b/Bdd/features/steps/syslog_steps.py @@ -5,7 +5,6 @@ import re import shutil import socket -import subprocess import threading import time from datetime import datetime, timezone @@ -489,65 +488,72 @@ def step_syslog_ng_is_running(context): ) -def build_buffered_command(context, transport, no_sd=False): - """Build the command line for the buffered BDD target with all options. +def build_buffered_extra_args(context, transport, no_sd=False): + """Build the extra-args list for the buffered BDD target. + + Returned list excludes the binary itself — start_bdd_target_process + routes the spawn through target_driver.spawn_example_process, which + appends the args to argv on Linux/Windows and translates them to + interactive `set NAME VALUE` lines after the prompt on FreeRTOS. Linux runner (syslog-ng oracle) and Windows runner (OTel oracle) both ship SolidSyslogBddTarget — same basename, same flags, same wire-record sizes byte-for-byte (S24.05 unified the basename so capacity-feature block packing is identical across runners). """ - binary = context.example_binary - assert os.path.exists(binary), ( - f"BDD target binary not found at {binary} — build with cmake first" - ) - - cmd = [ - os.path.join(".", binary), - "--transport", transport, - ] + args = ["--transport", transport] if getattr(context, "store_type", None): - cmd.extend(["--store", context.store_type]) + args.extend(["--store", context.store_type]) if getattr(context, "store_max_blocks", None): - cmd.extend(["--max-blocks", str(context.store_max_blocks)]) + args.extend(["--max-blocks", str(context.store_max_blocks)]) if getattr(context, "store_max_block_size", None): - cmd.extend(["--max-block-size", str(context.store_max_block_size)]) + args.extend(["--max-block-size", str(context.store_max_block_size)]) if getattr(context, "store_discard_policy", None): - cmd.extend(["--discard-policy", context.store_discard_policy]) + args.extend(["--discard-policy", context.store_discard_policy]) if getattr(context, "capacity_threshold", None): - cmd.extend(["--capacity-threshold", str(context.capacity_threshold)]) + args.extend(["--capacity-threshold", str(context.capacity_threshold)]) if getattr(context, "message_body", None): - cmd.extend(["--message", context.message_body]) + args.extend(["--message", context.message_body]) if no_sd: - cmd.append("--no-sd") + args.append("--no-sd") if getattr(context, "halt_exit", False): - cmd.append("--halt-exit") - return cmd - - -def start_bdd_target_process(context, cmd): - """Start the BDD target process and wait for the initial prompt.""" - context.interactive_process = subprocess.Popen( - cmd, - stdin=subprocess.PIPE, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - text=True, + args.append("--halt-exit") + return args + + +def start_bdd_target_process(context, extra_args): + """Start the buffered BDD target and wait for the initial prompt. + + Single Popen path across native and QEMU backends — target_driver + picks the spawn (subprocess on Linux/Windows, qemu-system-arm with + -kernel on FreeRTOS) and apply_extra_args delivers the same + --flag value pairs either as argv (native) or `set NAME VALUE` over + the UART (FreeRTOS) after the prompt is up. Adding a new embedded + platform reuses both helpers without touching this function. + """ + binary = context.example_binary + assert os.path.exists(binary), ( + f"BDD target binary not found at {binary} — build with cmake first" + ) + + context.interactive_process = spawn_example_process( + context, extra_args=extra_args ) context.example_pid = context.interactive_process.pid wait_for_prompt(context.interactive_process) + apply_extra_args(context, context.interactive_process, extra_args) @given("the BDD target is running with transport {transport:w}") def step_bdd_target_running_with_transport(context, transport): - cmd = build_buffered_command(context, transport) - start_bdd_target_process(context, cmd) + start_bdd_target_process(context, build_buffered_extra_args(context, transport)) @given("the BDD target is running with transport {transport:w} and no structured data") def step_bdd_target_running_with_transport_no_sd(context, transport): - cmd = build_buffered_command(context, transport, no_sd=True) - start_bdd_target_process(context, cmd) + start_bdd_target_process( + context, build_buffered_extra_args(context, transport, no_sd=True) + ) @given("the block store is enabled") @@ -1269,8 +1275,7 @@ def step_check_last_sequence_id(context, value): @given("the BDD target is running with default transport {transport:w}") def step_bdd_target_running_with_default_transport(context, transport): - cmd = build_buffered_command(context, transport) - start_bdd_target_process(context, cmd) + start_bdd_target_process(context, build_buffered_extra_args(context, transport)) @when("the client switches to transport {transport:w}") diff --git a/Bdd/features/store_and_forward.feature b/Bdd/features/store_and_forward.feature index 0833bb1a..05a5bda3 100644 --- a/Bdd/features/store_and_forward.feature +++ b/Bdd/features/store_and_forward.feature @@ -1,4 +1,4 @@ -@tcp @buffered +@tcp @buffered @store Feature: Store and forward during sender outage When the syslog oracle goes down, messages accumulate in the file-based store. Once the oracle recovers, the service loop diff --git a/Bdd/features/store_capacity.feature b/Bdd/features/store_capacity.feature index 025dbf6d..a9d8d739 100644 --- a/Bdd/features/store_capacity.feature +++ b/Bdd/features/store_capacity.feature @@ -1,4 +1,4 @@ -@tcp @buffered +@tcp @buffered @store Feature: Store capacity limit and discard policy The block store uses rotating files with a configurable capacity. When the store is full, the discard policy determines whether diff --git a/DEVLOG.md b/DEVLOG.md index ea0c2611..ffa69d6e 100644 --- a/DEVLOG.md +++ b/DEVLOG.md @@ -1,5 +1,82 @@ # Dev Log +## 2026-05-11 — S08.10 tcp_reconnect BDD scenario on QEMU (#339) + +Second slice of S08.06. The reconnect-on-the-wire is already a property +of `SolidSyslogStreamSender` plus the close-on-Send-failure contract +the FreeRTOS adapter inherited in S08.09 (test +`SendClosesSocketOnError` pins it). So this story is the verification +slice: re-tag the BDD features so the FreeRTOS runner can pick up +`tcp_reconnect.feature` without dragging in the file-store-dependent +ones, and unify the interactive-process spawn through `target_driver` +so the buffered scenarios run on QEMU. + +### Decisions + +- **Split `@buffered` into `@buffered` + `@store`.** The earlier tag + was overloaded: it meant *both* "uses the long-lived interactive + process" *and* "needs a file-backed store". The freertos compose + filter `not @buffered` then excluded scenarios that need only the + interactive process (tcp_reconnect, switching_transport) alongside + the genuinely file-store-dependent ones (store_and_forward, + capacity_threshold, store_capacity, block_lifecycle, + power_cycle_replay). New convention: `@buffered` keeps the + interactive-process meaning, `@store` marks the file-store + dependency. FreeRTOS filter now excludes `@store` only; + `tcp_reconnect.feature` (`@tcp @buffered`, no `@store`) is admitted. + `switching_transport.feature` would also be admitted but lacks + `@udp`/`@tcp` at the feature level, so the existing + `(@udp or @tcp)` clause keeps it out until S08.11 lights it up. +- **Route `start_bdd_target_process` through `target_driver`, not + `subprocess.Popen` directly.** The one-shot path + (`run_example` → `spawn_example_process` → `apply_extra_args`) + already abstracted native-vs-QEMU spawn and argv-vs-UART + translation; the interactive path bypassed it. Unifying both + through `spawn_example_process` + `apply_extra_args` removes the + duplicated Popen shape and means future embedded platforms + (semihosting, OpenOCD/picocom against real hardware, …) extend + one branch point in `target_driver` instead of two parallel + orchestration helpers. Linux/Windows behaviour is byte-for-byte + unchanged: flags still arrive as argv. FreeRTOS gets QEMU plus + `set transport tcp` over the UART after `wait_for_prompt`. +- **`SendClosesSocketOnFailure` already covered.** The story's + optional unit-test deliverable was satisfied by S08.09's + `SendClosesSocketOnError` (and the matching + `SendClosesSocketOnShortWrite` / + `ReadReturnsNegativeOneOnErrorAndClosesSocket`) — the close-on- + failure contract is the *only* thing that makes reconnect possible + and it's already pinned by three tests using + `CHECK_SOCKET_CLOSED_ONCE`. No additional test added here. +- **`SO_SNDTIMEO=200ms` Service-starvation question deferred to the + CI scenario.** Local docker isn't available in the freertos-target + devcontainer and the BDD step that swaps the syslog-ng config to + trigger the outage can break docker networking if run outside the + compose environment — `bdd-freertos-qemu` is the safe arbiter. + QEMU smoke proved the interactive task survives `set transport tcp` + plus repeated `send` cycles, so the producer side is unblocked; + the Service-task drain behaviour during the actual outage is what + CI will surface. + +### Deferred + +- **`switching_transport.feature` on FreeRTOS** → S08.11 (#340). The + tagging cleanup now leaves the door open (no `@store`), but the + feature is tagged `@buffered` only — adding `@udp`/`@tcp` is part + of S08.11. +- **CMake-driven memory scaling** still open from S08.09 — captured + in memory `project_freertos_stack_budget`. Not touched here. +- **End-to-end BDD verification locally.** Docker still unavailable + in the freertos-target devcontainer; CI remains the real arbiter + for outage scenarios. + +### Open questions + +- Whether the 200ms blocking-connect cap during the outage window + starves the Service task enough to delay the post-resume delivery + past `wait_for_messages`' 10s deadline. CI will tell us; if it + flakes, the fix is to bump the timeout (per the + `feedback_no_flaky_ci` rule) rather than chase a real bug. + ## 2026-05-11 — S08.09 SolidSyslogFreeRtosTcpStream + tcp_transport BDD on QEMU (#341) First slice of S08.06 (TCP transport on FreeRTOS-Plus-TCP). Lands the diff --git a/ci/docker-compose.bdd.yml b/ci/docker-compose.bdd.yml index 3bd5e3f8..e600fb1c 100644 --- a/ci/docker-compose.bdd.yml +++ b/ci/docker-compose.bdd.yml @@ -102,10 +102,12 @@ services: - BDD_TARGET=freertos - EXAMPLE_BINARY=build/freertos-cross/Bdd/Targets/FreeRtos/SolidSyslogBddTarget.elf # S08.09 admits @tcp scenarios so tcp_transport.feature runs alongside the - # existing @udp ones; @buffered excludes store-and-forward features that - # need a file-backed store (no FreeRTOS file abstraction yet), and + # existing @udp ones. S08.10 splits the earlier @buffered exclusion: @store + # marks features that need a file-backed store (no FreeRTOS file + # abstraction yet), while plain @buffered just means "uses the long-lived + # interactive process" and is supported on FreeRTOS via QEMU's UART. # @windows_wip excludes the Windows-only TCP singletask variant. - command: behave --junit --junit-directory Bdd/junit --tags='not @wip and not @freertoswip and not @rtc and not @buffered and not @windows_wip and (@udp or @tcp)' Bdd/features/ + command: behave --junit --junit-directory Bdd/junit --tags='not @wip and not @freertoswip and not @rtc and not @store and not @windows_wip and (@udp or @tcp)' Bdd/features/ volumes: bdd-output-linux: From 6c46c0ad587435c18768966fd4906e4318c7235d Mon Sep 17 00:00:00 2001 From: David Cozens Date: Mon, 11 May 2026 19:54:18 +0000 Subject: [PATCH 3/3] fix: preserve explicit zero in build_buffered_extra_args `--capacity-threshold 0` is the documented "disable" sentinel for SolidSyslogStoreThresholdFunction; truthy-coerce would silently drop it. No scenario passes 0 today but the bug is latent in the helper. CodeRabbit catch on #342. Co-Authored-By: Claude Opus 4.7 (1M context) --- Bdd/features/steps/syslog_steps.py | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/Bdd/features/steps/syslog_steps.py b/Bdd/features/steps/syslog_steps.py index 08d5c758..bc00ff5b 100644 --- a/Bdd/features/steps/syslog_steps.py +++ b/Bdd/features/steps/syslog_steps.py @@ -504,13 +504,16 @@ def build_buffered_extra_args(context, transport, no_sd=False): args = ["--transport", transport] if getattr(context, "store_type", None): args.extend(["--store", context.store_type]) - if getattr(context, "store_max_blocks", None): + # Numeric option guards use `is not None` so an explicit 0 survives: + # `--capacity-threshold 0` is the documented "disable" sentinel for + # SolidSyslogStoreThresholdFunction (Core/Interface/SolidSyslogBlockStore.h). + if getattr(context, "store_max_blocks", None) is not None: args.extend(["--max-blocks", str(context.store_max_blocks)]) - if getattr(context, "store_max_block_size", None): + if getattr(context, "store_max_block_size", None) is not None: args.extend(["--max-block-size", str(context.store_max_block_size)]) if getattr(context, "store_discard_policy", None): args.extend(["--discard-policy", context.store_discard_policy]) - if getattr(context, "capacity_threshold", None): + if getattr(context, "capacity_threshold", None) is not None: args.extend(["--capacity-threshold", str(context.capacity_threshold)]) if getattr(context, "message_body", None): args.extend(["--message", context.message_body])