zebra: defer RIB sweep until metaqueue is drained by deepak-singhal0408 · Pull Request #27093 · sonic-net/sonic-buildimage

deepak-singhal0408 · 2026-04-30T09:26:49Z

What I did

SONiC-only carry patch for FRR zebra — defer RIB sweep until the metaqueue drains. Adds bounded retry logic (50 retries × 20ms = ~1s cap).

To be dropped when the upstream startup-ordering rework (FRRouting/frr#21550) lands in a future FRR tag bump.

How I did it

When zebra starts without -K (graceful_restart=0), the sweep timer fires with 0-second delay but the metaqueue work_queue has a 10ms batching hold (ZEBRA_RIB_PROCESS_HOLD_TIME). This causes the sweep to walk an empty RIB and miss stale routes still queued in the metaqueue.

The patch checks zrouter.mq->size > 0 before sweeping. If the metaqueue is non-empty, it reschedules the sweep at 2× the hold time (20ms), up to 50 retries (~1 second). After max retries, it sweeps anyway with a warning log.

How to verify it

Boot a SONiC VS image with this patch
Verify stale kernel routes are properly swept after startup
Check syslog for RIB sweep deferred debug messages (with debug rib enabled)

Which release branch to backport

202511, msft-202412

Cherry-pick tracking:

Branch	PR	Patch #	Status
`master`	#27093	0108	✅ Merged
`202511`	#27210	0107	⏳ CI pending
`msft-202412`	Azure/sonic-buildimage-msft#2260	0147	⏳ CI pending

Upstream references

FRRouting/frr#21550 — structural fix (pending merge upstream, will supersede this patch)
FRRouting/frr#21826 — this targeted fix (closed in favor of #21550)
Fixes: Bug: FRR failed to clean up static blackhole route when recovering from ungraceful exits. #27012

mssonicbld · 2026-04-30T09:26:57Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-04-30T09:27:12Z

Azure Pipelines successfully started running 1 pipeline(s).

Copilot

Pull request overview

This PR updates the SONiC FRR patchset to address a zebra startup timing issue where rib_sweep_route() can run before queued kernel routes have been processed into the RIB, potentially leaving stale/orphaned kernel routes after an ungraceful restart.

Changes:

Add a zebra patch that defers rib_sweep_route() execution while the zebra metaqueue still has pending work.
Update the FRR patch series to include the new patch in the applied patchset.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
src/sonic-frr/patch/series	Adds the new zebra patch to the ordered FRR patch series.
src/sonic-frr/patch/0107-zebra-defer-rib-sweep-until-metaqueue-drained.patch	Defers RIB sweep until the metaqueue is drained by rescheduling the sweep timer.

mssonicbld · 2026-04-30T15:58:40Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-04-30T15:58:55Z

Azure Pipelines successfully started running 1 pipeline(s).

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

mssonicbld · 2026-05-05T05:42:45Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-05-05T05:43:00Z

Azure Pipelines successfully started running 1 pipeline(s).

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

mssonicbld · 2026-05-06T04:36:54Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-05-06T04:37:08Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-05-06T04:47:05Z

/azp run Azure.sonic-buildimage

mssonicbld · 2026-05-06T04:47:05Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-05-06T04:47:15Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2026-05-06T04:47:15Z

Azure Pipelines successfully started running 1 pipeline(s).

deepak-singhal0408 · 2026-05-06T05:03:06Z

/azpw run Azure.sonic-buildimage

mssonicbld · 2026-05-06T05:06:51Z

⚠️ Notice: /azpw run only runs failed jobs now. If you want to trigger a whole pipline run, please rebase your branch or close and reopen the PR.
💡 Tip: You can also use /azpw retry to retry failed jobs directly.

Retrying failed(or canceled) jobs...

mssonicbld · 2026-05-06T22:13:46Z

The change is not in msft-202412 yet. @deepak-singhal0408, please manually create the cherry pick PR for branch msft-202412.
You can ping the release branch owner(github account: r12f) to approve your cherry pick PR.
If this change is already in msft-202412, please comment "already in msft-202412". Thanks!

_{---Powered by SONiC BuildBot}

mssonicbld · 2026-05-06T22:13:47Z

The change is not in 202511 yet. @deepak-singhal0408, please manually create the cherry pick PR for branch 202511.
You can ping the release branch owner(github account: vmittal-msft) to approve your cherry pick PR.
If this change is already in 202511, please comment "already in 202511". Thanks!

_{---Powered by SONiC BuildBot}

Upstream PR FRRouting/frr#21550 merged (2026-05-03). This is a temporary carry patch that will be dropped on the next FRR tag bump. The patch defers the post-startup RIB sweep in zebra until the metaqueue is fully drained, preventing premature deletion of routes that are still being processed from bgpd/staticd. Signed-off-by: Deepak Singhal <deepsinghal@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: mhchann <mhchann082@gmail.com>

Cherry-pick of #27093 to 202511 branch. Patch renumbered from 0108 to 0107 to match 202511 series. Signed-off-by: Deepak Singhal <deepsinghal@microsoft.com>

…#27210) Cherry-pick of sonic-net#27093 to 202511 branch. Patch renumbered from 0108 to 0107 to match 202511 series. Signed-off-by: Deepak Singhal <deepsinghal@microsoft.com> Signed-off-by: Kebo Liu <kebol@nvidia.com>

Cherry-pick of #27093 to 202511 branch. Patch renumbered from 0108 to 0107 to match 202511 series. Signed-off-by: Deepak Singhal <deepsinghal@microsoft.com>

Cherry-pick of sonic-net/sonic-buildimage#27093 to msft-202412 branch. Patch renumbered from 0108 to 0147 to match msft-202412 series. Signed-off-by: Deepak Singhal <deepsinghal@microsoft.com>

deepak-singhal0408 requested a review from lguohan as a code owner April 30, 2026 09:26

Copilot AI review requested due to automatic review settings April 30, 2026 09:26

Copilot started reviewing on behalf of deepak-singhal0408 April 30, 2026 09:27 View session

deepak-singhal0408 mentioned this pull request Apr 30, 2026

Bug: FRR failed to clean up static blackhole route when recovering from ungraceful exits. #27012

Closed

Copilot AI reviewed Apr 30, 2026

View reviewed changes

Comment thread src/sonic-frr/patch/0107-zebra-defer-rib-sweep-until-metaqueue-drained.patch Outdated

deepak-singhal0408 force-pushed the fix/zebra-sweep-metaqueue-drain branch from b5cdf91 to 977c06d Compare April 30, 2026 15:58

deepak-singhal0408 self-assigned this Apr 30, 2026

deepak-singhal0408 added this to SONiC Routing Dashboard Apr 30, 2026

github-project-automation Bot moved this to Todo in SONiC Routing Dashboard Apr 30, 2026

deepak-singhal0408 moved this from Todo to In Progress in SONiC Routing Dashboard Apr 30, 2026

r12f added Approved for 202511 Branch Request for msft-202412 Branch Request for 202511 Branch and removed Approved for 202511 Branch labels May 1, 2026

Copilot AI review requested due to automatic review settings May 5, 2026 05:27

deepak-singhal0408 force-pushed the fix/zebra-sweep-metaqueue-drain branch from 977c06d to 842b4c5 Compare May 5, 2026 05:27

Copilot started reviewing on behalf of deepak-singhal0408 May 5, 2026 05:28 View session

Copilot AI reviewed May 5, 2026

View reviewed changes

deepak-singhal0408 force-pushed the fix/zebra-sweep-metaqueue-drain branch from 842b4c5 to 2ad4c84 Compare May 5, 2026 05:30

Copilot AI review requested due to automatic review settings May 5, 2026 05:33

deepak-singhal0408 force-pushed the fix/zebra-sweep-metaqueue-drain branch from 2ad4c84 to 7fd24c2 Compare May 5, 2026 05:33

Copilot started reviewing on behalf of deepak-singhal0408 May 5, 2026 05:34 View session

Copilot AI reviewed May 5, 2026

View reviewed changes

Comment thread src/sonic-frr/patch/0108-SONiC-ONLY-zebra-defer-rib-sweep-until-metaqueue-drained.patch Outdated

deepak-singhal0408 force-pushed the fix/zebra-sweep-metaqueue-drain branch from 7fd24c2 to 0525ff4 Compare May 5, 2026 05:39

deepak-singhal0408 force-pushed the fix/zebra-sweep-metaqueue-drain branch from 1370bdb to 76c49af Compare May 6, 2026 04:29

Copilot started reviewing on behalf of deepak-singhal0408 May 6, 2026 04:30 View session

Copilot AI reviewed May 6, 2026

View reviewed changes

dgsudharsan approved these changes May 6, 2026

View reviewed changes

eddieruan-alibaba approved these changes May 6, 2026

View reviewed changes

eddieruan-alibaba merged commit 4d2fd24 into sonic-net:master May 6, 2026
29 checks passed

github-project-automation Bot moved this from In Progress to Done in SONiC Routing Dashboard May 6, 2026

This was referenced May 7, 2026

[202511] zebra: defer RIB sweep until metaqueue is drained #27210

Merged

[202412] zebra: defer RIB sweep until metaqueue is drained Azure/sonic-buildimage-msft#2260

Open

deepak-singhal0408 added Created PR to msft-202412 Branch Created PR to 202511 Branch and removed Request for msft-202412 Branch Cherry Pick Conflict_msft-202412 Request for 202511 Branch Cherry Pick Conflict_202511 labels May 7, 2026

Conversation

deepak-singhal0408 commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What I did

How I did it

How to verify it

Which release branch to backport

Upstream references

Uh oh!

mssonicbld commented Apr 30, 2026

Uh oh!

azure-pipelines Bot commented Apr 30, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

mssonicbld commented Apr 30, 2026

Uh oh!

azure-pipelines Bot commented Apr 30, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

mssonicbld commented May 5, 2026

Uh oh!

azure-pipelines Bot commented May 5, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

mssonicbld commented May 6, 2026

Uh oh!

azure-pipelines Bot commented May 6, 2026

Uh oh!

mssonicbld commented May 6, 2026

Uh oh!

mssonicbld commented May 6, 2026

Uh oh!

azure-pipelines Bot commented May 6, 2026

Uh oh!

azure-pipelines Bot commented May 6, 2026

Uh oh!

deepak-singhal0408 commented May 6, 2026

Uh oh!

mssonicbld commented May 6, 2026

Uh oh!

Uh oh!

mssonicbld commented May 6, 2026

Uh oh!

mssonicbld commented May 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

deepak-singhal0408 commented Apr 30, 2026 •

edited

Loading