Skip to content

pbr-1.2.3: fix deferred boot race condition by passing state via arguments#129

Merged
stangri merged 1 commit into
mossdef-org:1.2.3from
Kasoo:fix_deferred_boot_1.2.3
Jul 16, 2026
Merged

pbr-1.2.3: fix deferred boot race condition by passing state via arguments#129
stangri merged 1 commit into
mossdef-org:1.2.3from
Kasoo:fix_deferred_boot_1.2.3

Conversation

@Kasoo

@Kasoo Kasoo commented Jul 13, 2026

Copy link
Copy Markdown

When the uplink is down at startup, start_service defers to boot mode. The wrapper writes /var/run/pbr.boot so that service_triggers can register a boot-retry trigger. However, service_started spawns a new ucode process that does not inherit the in-memory _deferred_to_boot flag, so it incorrectly treats the deferral as a startup failure and emits FAILED TO START.

Previously, the .boot marker file was used to bridge this gap, however service_triggers deletes it before service_started runs (both are called sequentially by rc.common in the same shell process).

Fix this by passing $_pbr_deferred_to_boot directly as the second argument to $_ucode service_started, into pbr.service_started(), managing this state in the shell process instead.

Changed test 05_interfaces/04_uplink_down_no_failed_to_start to use a separate create_pbr() instance for service_started, matching the real two-process lifecycle, and update it to pass state via arguments.

Also added a test case simulating the rc.common shell-script boot lifecycle 06_shell_wrapper.

Also added several other tests covering different parts of the boot process that I looked at while trying to diagnose this problem. None of these tests failed but are probably useful to have in future.

@Kasoo Kasoo changed the title pbr: fix deferred boot race condition by passing state via arguments pbr-1.2.3: fix deferred boot race condition by passing state via arguments Jul 13, 2026
@stangri
stangri requested a review from Copilot July 13, 2026 22:00

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a startup lifecycle race where service_started runs in a fresh ucode process and can’t see the in-memory “deferred to boot” flag, causing an incorrect “FAILED TO START” path when uplink is down at boot.

Changes:

  • Pass deferred-boot state from the init.d wrapper into ucode service_started() via an explicit argument.
  • Update and add tests to model the real two-process rc.common lifecycle (including new shell-wrapper tests).
  • Extend the test runner to execute .sh shell-wrapper tests.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tests/run_tests.sh Runs .sh tests (bash) in addition to existing testcase format; fixes selected-test iteration.
tests/06_shell_wrapper/01_deferred_boot_lifecycle.sh New shell-level lifecycle test for deferred boot state propagation.
tests/06_shell_wrapper/02_boot_fast_path.sh New test asserting boot() bypasses ucode and only registers triggers.
tests/06_shell_wrapper/03_service_data_unset.sh New test ensuring degraded starts unset service_data hook to avoid wiping procd data.
tests/06_shell_wrapper/04_on_interface_reload_gatekeeper.sh New test for on_interface_reload early-return gatekeeping behavior.
tests/05_interfaces/04_uplink_down_no_failed_to_start Adjusts test to reflect two-process lifecycle and pass deferred state explicitly.
tests/05_interfaces/09_deferred_boot_no_lock New test verifying no lockfile and no failure message on deferred boot when state is passed.
tests/05_interfaces/10_normal_boot_creates_lock New test verifying lockfile creation on normal (uplink-up) start.
tests/05_interfaces/11_reload_degraded_upgrades_to_start New test verifying reload behavior upgrades to full interface processing when service data is empty.
files/lib/pbr/pbr.uc Changes service_started() signature to accept deferred state via argument.
files/lib/pbr/cli.uc Forwards the second CLI argument into pbr.service_started().
files/etc/init.d/pbr Captures _pbr_deferred_to_boot from ucode and passes it into service_started.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread files/lib/pbr/pbr.uc Outdated
Comment thread tests/06_shell_wrapper/01_deferred_boot_lifecycle.sh

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

Comment thread tests/run_tests.sh Outdated
Comment thread tests/06_shell_wrapper/01_deferred_boot_lifecycle.sh Outdated
Comment thread tests/06_shell_wrapper/02_boot_fast_path.sh
When the uplink is down at startup, start_service defers to boot mode.
The wrapper writes /var/run/pbr.boot so that service_triggers can
register a boot-retry trigger. However, service_started spawns a new
ucode process that does not inherit the in-memory _deferred_to_boot
flag, so it incorrectly treats the deferral as a startup failure and
emits FAILED TO START.

Historically, pbr relied on the .boot marker file to bridge this gap,
but service_triggers deletes it before service_started runs (both are
called sequentially by rc.common in the same shell process), making
any stat check unreliable.

Fix this by passing $_pbr_deferred_to_boot directly as the second
argument to $_ucode service_started, through cli.uc into
pbr.service_started(). This uses the shell process memory to bridge
the stateless gap between ucode invocations, and allows us to remove
the dead .boot stat fallback entirely.

Fix test 04 to use a separate create_pbr() instance for
service_started, matching the real two-process lifecycle, and update
it to pass state via arguments rather than a fake .boot file.
Add regression tests for the deferred-boot lock file invariants
and the degraded-reload-to-full-start upgrade path.
Add shell-level integration tests (tests/06_shell_wrapper/) that
mock rc.common and exercise the boot lifecycle, the boot() fast-path,
the service_data unset path, and the on_interface_reload lock gate.
Update run_tests.sh to execute .sh test files natively and fix
multi-argument test selection.
@Kasoo
Kasoo force-pushed the fix_deferred_boot_1.2.3 branch from 27d458c to 825271d Compare July 14, 2026 19:00
@stangri

stangri commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

Thank you for the fix and prompt attention to copilot suggestions @Kasoo , merged with gratitude.

PS. I'd appreciate if you could test mwan4 if you have time.

@stangri
stangri merged commit c5ad27d into mossdef-org:1.2.3 Jul 16, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants