[featured]: Skip redundant CONFIG_DB writes in sync_feature_scope by Bojun-Feng · Pull Request #368 · sonic-net/sonic-host-services

Bojun-Feng · 2026-03-31T02:19:16Z

Why I did it

Fix #369
Partially address sonic-net/sonic-buildimage#26350

sync_feature_scope() unconditionally writes to CONFIG_DB for every feature at startup and on feature state changes at runtime without checking whether the values have actually changed.

We should avoid redundant writes that does not update CONFIG_DB.

How I did it

Added a read-before-write check in sync_feature_scope(): read the current FEATURE|<name> entry from CONFIG_DB via get_entry(), and only call mod_entry() when has_per_asic_scope or has_global_scope actually differs from the intended value.
Guarded the get_entry() return with or {} to safely handle None returns, consistent with the defensive pattern already used in resync_feature_state() and sync_feature_delay_state() in the same file.
The conditional write applies to both the host CONFIG_DB and per-namespace CONFIG_DB on multi-ASIC platforms.
Added unit tests covering three cases: values unchanged (no write), values changed (write), and missing/empty entry (write).

How to verify it

I used the following script to test the update on a virtual switch in GNS3. We can iterate quickly since the modified Python file is at /usr/local/bin/featured and can be updated on the fly.

The virtual switch image is downloaded from official pipeline build on 2026/03/30 on commit ab081ee.

Bash Script & Setup

I had three files on the switch:

admin@sonic:~$ ls
new.py  old.py  test_featured.sh

old.py is copied from /usr/local/bin/featured on the switch. It is almost identical to the latest previous commit that modified the script, except due to compilation the first line changed from #!/usr/bin/env python3 to #!/usr/bin/python3.

new.py is almost identical to the script in PR, except the first line also changed from #!/usr/bin/env python3 to #!/usr/bin/python3.

The bash script test_featured.sh use either old.py or new.py to overwrite /usr/local/bin/featured, use systemctl to reload featured, then run config reload and see how many feature writes there are & how long did it take until the last write.

#!/bin/bash
# test_featured.sh - Test featured daemon CONFIG_DB write behavior
# Usage: ./test_featured.sh old|new
#
# Measures how many redundant FEATURE scope writes (has_per_asic_scope,
# has_global_scope) the featured daemon makes after config reload, and
# how long the CONFIG_DB mutation window stays open.

set -e

if [ -z "$1" ] || [[ "$1" != "old" && "$1" != "new" ]]; then
echo "Usage: $0 old|new"
exit 1
fi

VERSION="$1"
LOG="/tmp/monitor_${VERSION}.log"

# Clean up any leftover redis-cli MONITOR processes from previous runs
sudo pkill -f "redis-cli.*MONITOR" 2>/dev/null || true
sudo pkill -f "docker exec.*redis-cli.*MONITOR" 2>/dev/null || true
sleep 1
rm -f "$LOG"

# Deploy the selected version of featured
echo "==> Installing ${VERSION} version of featured..."
sudo cp ~/${VERSION}.py /usr/local/bin/featured
sudo systemctl restart featured
echo "==> Waiting for featured to settle..."
sleep 5

# Start capturing all CONFIG_DB operations
echo "==> Starting CONFIG_DB monitor..."
redis-cli -n 4 MONITOR < /dev/null > "$LOG" 2>&1 &
MONITOR_PID=$!
sleep 2

# Run config reload and measure wall-clock time
echo "==> Running config reload..."
RELOAD_START=$(date +%s.%N)
sudo config reload -y -f
RELOAD_END=$(date +%s.%N)
RELOAD_DURATION=$(python3 -c "print(f'{$RELOAD_END - $RELOAD_START:.2f}')")

# Wait for any background daemon writes to finish
echo "==> Waiting 90 seconds for background writes to complete..."
sleep 90

# Stop monitoring
kill $MONITOR_PID 2>/dev/null || true
sudo pkill -f "redis-cli.*MONITOR" 2>/dev/null || true
sleep 1

# --- Analysis ---

# Count: how many times featured wrote scope fields to FEATURE entries
SCOPE_WRITE_COUNT=$(grep --text -c 'HSET.*FEATURE.*\(has_per_asic_scope\|has_global_scope\)' "$LOG" || true)

# Timestamp of the first CONFIG_DB write (sonic-cfggen bulk load start)
FIRST_DB_WRITE=$(grep --text '"HSET"' "$LOG" | head -1 | awk '{print $1}')

# Timestamp of the last FEATURE scope field write by featured daemon
LAST_SCOPE_WRITE=$(grep --text 'HSET.*FEATURE.*\(has_per_asic_scope\|has_global_scope\)' "$LOG" | tail -1 | awk '{print $1}')

# Timestamp of the very last CONFIG_DB write of any kind
LAST_ANY_WRITE=$(grep --text '"HSET"' "$LOG" | tail -1 | awk '{print $1}')

# Calculate how long after config reload returned the scope writes continued
if [ -n "$LAST_SCOPE_WRITE" ] && [ -n "$FIRST_DB_WRITE" ]; then
SCOPE_WINDOW=$(python3 -c "print(f'{$LAST_SCOPE_WRITE - $RELOAD_END:.2f}')")
SCOPE_DURATION=$(python3 -c "print(f'{$LAST_SCOPE_WRITE - $FIRST_DB_WRITE:.2f}')")
else
SCOPE_WINDOW="N/A (no scope writes detected)"
SCOPE_DURATION="N/A"
fi

echo ""
echo "============================================================"
echo " Featured Daemon CONFIG_DB Write Test Results"
echo " Version: ${VERSION}"
echo "============================================================"
echo ""
echo " Config reload wall-clock duration: ${RELOAD_DURATION}s"
echo ""
echo " FEATURE scope writes (has_per_asic_scope / has_global_scope):"
echo " Total write count: ${SCOPE_WRITE_COUNT}"
echo " First CONFIG_DB write: ${FIRST_DB_WRITE:-N/A}"
echo " Last scope write by featured: ${LAST_SCOPE_WRITE:-none}"
echo " Last CONFIG_DB write (any): ${LAST_ANY_WRITE:-N/A}"
echo ""
echo " Seconds from first DB write to last scope write:"
echo " ${SCOPE_DURATION}"
echo ""
echo " Seconds from config reload RETURNING to last scope write:"
echo " ${SCOPE_WINDOW}"
echo ""
echo " Full log: ${LOG}"
echo "============================================================"

Full Logs Running the Script on VS

admin@sonic:~$ chmod +x test_featured.sh
# Run the old and new version alternatively, test each multiple times
# This output is caputred after running `sudo config save-y` and running the script multiple times previously
admin@sonic:~$ ./test_featured.sh old && \
sleep 20 && \
./test_featured.sh new && \
sleep 20 && \
./test_featured.sh old && \
sleep 20 && \
./test_featured.sh new

==> Installing old version of featured...
==> Waiting for featured to settle...
==> Starting CONFIG_DB monitor...
==> Running config reload...
Acquired lock on /etc/sonic/reload.lock
Disabling container and routeCheck monitoring ...
Running command: sudo systemctl stop aaastatsd.timer
Running command: sudo systemctl stop featured.timer
Running command: sudo systemctl stop hostcfgd.timer
Running command: sudo systemctl stop tacacs-config.timer
Stopping SONiC target ...
Running command: /usr/local/bin/sonic-cfggen -j /etc/sonic/init_cfg.json -j /etc/sonic/config_db.json --write-to-db
Running command: /usr/local/bin/db_migrator.py -o migrate
Running command: /usr/local/bin/sonic-cfggen -d -y /etc/sonic/sonic_version.yml -t /usr/share/sonic/templates/sonic-environment.j2,/etc/sonic/sonic-environment
Restarting SONiC target ...
Enabling container and routeCheck monitoring ...
Reloading Monit configuration ...
Reinitializing monit daemon
Released lock on /etc/sonic/reload.lock
==> Waiting 90 seconds for background writes to complete...

============================================================
 Featured Daemon CONFIG_DB Write Test Results
 Version: old
============================================================

 Config reload wall-clock duration: 30.63s

 FEATURE scope writes (has_per_asic_scope / has_global_scope):
 Total write count: 84
 First CONFIG_DB write: 1774906714.590470
 Last scope write by featured: 1774906746.737287
 Last CONFIG_DB write (any): 1774906836.590132

 Seconds from first DB write to last scope write:
 32.15

 Seconds from config reload RETURNING to last scope write:
 -0.30

 Full log: /tmp/monitor_old.log
============================================================
==> Installing new version of featured...
==> Waiting for featured to settle...
==> Starting CONFIG_DB monitor...
==> Running config reload...
Acquired lock on /etc/sonic/reload.lock
Disabling container and routeCheck monitoring ...
Running command: sudo systemctl stop aaastatsd.timer
Running command: sudo systemctl stop featured.timer
Running command: sudo systemctl stop hostcfgd.timer
Running command: sudo systemctl stop tacacs-config.timer
Stopping SONiC target ...
Running command: /usr/local/bin/sonic-cfggen -j /etc/sonic/init_cfg.json -j /etc/sonic/config_db.json --write-to-db
Running command: /usr/local/bin/db_migrator.py -o migrate
Running command: /usr/local/bin/sonic-cfggen -d -y /etc/sonic/sonic_version.yml -t /usr/share/sonic/templates/sonic-environment.j2,/etc/sonic/sonic-environment
Restarting SONiC target ...
Enabling container and routeCheck monitoring ...
Reloading Monit configuration ...
Reinitializing monit daemon
Released lock on /etc/sonic/reload.lock
==> Waiting 90 seconds for background writes to complete...

============================================================
 Featured Daemon CONFIG_DB Write Test Results
 Version: new
============================================================

 Config reload wall-clock duration: 30.62s

 FEATURE scope writes (has_per_asic_scope / has_global_scope):
 Total write count: 38
 First CONFIG_DB write: 1774906865.598661
 Last scope write by featured: 1774906886.589453
 Last CONFIG_DB write (any): 1774906987.320508

 Seconds from first DB write to last scope write:
 20.99

 Seconds from config reload RETURNING to last scope write:
 -11.33

 Full log: /tmp/monitor_new.log
============================================================
==> Installing old version of featured...
==> Waiting for featured to settle...
==> Starting CONFIG_DB monitor...
==> Running config reload...
Acquired lock on /etc/sonic/reload.lock
Disabling container and routeCheck monitoring ...
Running command: sudo systemctl stop aaastatsd.timer
Running command: sudo systemctl stop featured.timer
Running command: sudo systemctl stop hostcfgd.timer
Running command: sudo systemctl stop tacacs-config.timer
Stopping SONiC target ...
Running command: /usr/local/bin/sonic-cfggen -j /etc/sonic/init_cfg.json -j /etc/sonic/config_db.json --write-to-db
Running command: /usr/local/bin/db_migrator.py -o migrate
Running command: /usr/local/bin/sonic-cfggen -d -y /etc/sonic/sonic_version.yml -t /usr/share/sonic/templates/sonic-environment.j2,/etc/sonic/sonic-environment
Restarting SONiC target ...
Enabling container and routeCheck monitoring ...
Reloading Monit configuration ...
Reinitializing monit daemon
Released lock on /etc/sonic/reload.lock
==> Waiting 90 seconds for background writes to complete...

============================================================
 Featured Daemon CONFIG_DB Write Test Results
 Version: old
============================================================

 Config reload wall-clock duration: 30.14s

 FEATURE scope writes (has_per_asic_scope / has_global_scope):
 Total write count: 84
 First CONFIG_DB write: 1774907016.330388
 Last scope write by featured: 1774907046.978066
 Last CONFIG_DB write (any): 1774907137.935647

 Seconds from first DB write to last scope write:
 30.65

 Seconds from config reload RETURNING to last scope write:
 -1.36

 Full log: /tmp/monitor_old.log
============================================================
==> Installing new version of featured...
==> Waiting for featured to settle...
==> Starting CONFIG_DB monitor...
==> Running config reload...
Acquired lock on /etc/sonic/reload.lock
Disabling container and routeCheck monitoring ...
Running command: sudo systemctl stop aaastatsd.timer
Running command: sudo systemctl stop featured.timer
Running command: sudo systemctl stop hostcfgd.timer
Running command: sudo systemctl stop tacacs-config.timer
Stopping SONiC target ...
Running command: /usr/local/bin/sonic-cfggen -j /etc/sonic/init_cfg.json -j /etc/sonic/config_db.json --write-to-db
Running command: /usr/local/bin/db_migrator.py -o migrate
Running command: /usr/local/bin/sonic-cfggen -d -y /etc/sonic/sonic_version.yml -t /usr/share/sonic/templates/sonic-environment.j2,/etc/sonic/sonic-environment
Restarting SONiC target ...
Enabling container and routeCheck monitoring ...
Reloading Monit configuration ...
Reinitializing monit daemon
Released lock on /etc/sonic/reload.lock
==> Waiting 90 seconds for background writes to complete...

============================================================
 Featured Daemon CONFIG_DB Write Test Results
 Version: new
============================================================

 Config reload wall-clock duration: 30.02s

 FEATURE scope writes (has_per_asic_scope / has_global_scope):
 Total write count: 38
 First CONFIG_DB write: 1774907166.717689
 Last scope write by featured: 1774907187.528028
 Last CONFIG_DB write (any): 1774907288.738037

 Seconds from first DB write to last scope write:
 20.81

 Seconds from config reload RETURNING to last scope write:
 -11.11

 Full log: /tmp/monitor_new.log
============================================================

TLDR: The number of scope writes dropped from 84 to 38. The time from first CONFIG_DB write to last scope write dropped from ~30–32s to ~21s, so scope synchronization completes about 10s earlier, shortening the potential race-condition window.

Description for the changelog

Skip redundant CONFIG_DB writes in featured daemon's sync_feature_scope to speed up config reload.

* Add read-before-write check in sync_feature_scope to only write scope fields when values actually differ from CONFIG_DB * Add unit tests covering unchanged, changed, and missing entry cases Signed-off-by: Bojun Feng <bojundf@gmail.com>

mssonicbld · 2026-03-31T02:19:24Z

/azp run

azure-pipelines · 2026-03-31T02:19:33Z

Azure Pipelines successfully started running 1 pipeline(s).

* Move tests from TestFeatureDaemon to TestFeatureHandler to avoid class-level mock.patch argument injection issues * Improve assertions and add partial-change test case Signed-off-by: Bojun Feng <bojundf@gmail.com>

mssonicbld · 2026-03-31T02:47:56Z

/azp run

azure-pipelines · 2026-03-31T02:48:07Z

Azure Pipelines successfully started running 1 pipeline(s).

* Merge skip/write/missing-entry tests into one multi-scenario method following the test_feature_resync pattern * Remove redundant partial-change and empty-dict sub-cases Signed-off-by: Bojun Feng <bojundf@gmail.com>

mssonicbld · 2026-03-31T03:13:30Z

/azp run

azure-pipelines · 2026-03-31T03:13:39Z

Azure Pipelines successfully started running 1 pipeline(s).

Copilot

Pull request overview

This PR optimizes the featured host daemon by avoiding redundant FEATURE|<name> scope-field writes to CONFIG_DB, reducing unnecessary Redis churn during startup/config reload and on runtime feature updates.

Changes:

Added a read-before-write check in FeatureHandler.sync_feature_scope() to only update scope fields when values differ.
Applied the conditional write to both host CONFIG_DB and per-namespace CONFIG_DB (multi-ASIC).
Added unit tests for unchanged/changed/missing-entry behavior and namespace propagation.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.

File	Description
`scripts/featured`	Adds conditional scope syncing to avoid redundant `CONFIG_DB` writes.
`tests/featured/featured_test.py`	Adds unit tests validating conditional scope writes and namespace propagation.

tirupatihemanth · 2026-03-31T21:08:38Z

Hi @Bojun-Feng,
for the case if the host FEATURE entry already matches but a per-namespace FEATURE entry doesn't match with this implementation we won't update the per-namespace entries. Please check, thanks

Signed-off-by: Bojun-Feng <bojundf@gmail.com>

mssonicbld · 2026-04-01T04:59:06Z

/azp run

azure-pipelines · 2026-04-01T04:59:15Z

Azure Pipelines successfully started running 1 pipeline(s).

tirupatihemanth

NIT: can you move the duplicated getting and update fields if things changed into a function as it is being implemented twice now?

Signed-off-by: Bojun-Feng <bojundf@gmail.com>

mssonicbld · 2026-04-01T18:10:30Z

/azp run

azure-pipelines · 2026-04-01T18:10:40Z

Azure Pipelines successfully started running 1 pipeline(s).

Bojun-Feng · 2026-04-20T15:21:21Z

Hi @vmittal-msft @tirupatihemanth — just checking in on this PR. Seems like all checks are passing and it is approved for 202511 Branch. Please let me know if there's anything else needed from my side.

saiarcot895 · 2026-05-02T00:29:48Z

Before the update, config reload returns ~2 seconds after the last scope write; after the update, config reload returns ~11 seconds after the last scope write.

Is this written the other way around?

Bojun-Feng · 2026-05-02T00:38:54Z

Before the update, config reload returns ~2 seconds after the last scope write; after the update, config reload returns ~11 seconds after the last scope write.

Is this written the other way around?

@saiarcot895 It is correct. The phrasing there could definitely be clearer though.

What I meant is that scope write completes much faster without redundant operations, and config reload is expected to take roughly the same amount of time as before.

The referenced issue in sonic-buildimage potentially occurs when scope write takes too long, not completing even after config reload returns. In the example, since we’re dealing with small configs, that race window doesn’t really manifest, and scope writes always complete before config reload returns.

rookie-who

LGTM. Clean read-before-write optimization:

_conditional_update_scope helper avoids redundant Redis writes
Host and namespace checks are independent (namespace can be stale even when host matches)
Single mod_entry call with only changed fields
Tests cover: no-change, partial-change, missing entry, and namespace-stale-while-host-matches scenarios

No correctness concerns. The get_entry + mod_entry is not atomic, but the existing code was already non-atomic (unconditional mod_entry) and the race window is negligible for scope fields that rarely change.

mssonicbld · 2026-05-07T17:32:20Z

Cherry-pick PR to 202511: #382

mssonicbld · 2026-05-07T21:17:08Z

The cherry pick conflict has been handled manually. Removing cherry pick conflict label...

_{---Powered by SONiC BuildBot}

[featured]: Fix sync_feature_scope test placement and coverage

6d4557f

* Move tests from TestFeatureDaemon to TestFeatureHandler to avoid class-level mock.patch argument injection issues * Improve assertions and add partial-change test case Signed-off-by: Bojun Feng <bojundf@gmail.com>

Bojun-Feng force-pushed the bug/featured-redundant-configdb-writes branch from e947a49 to 6d4557f Compare March 31, 2026 02:47

[featured]: Consolidate sync_feature_scope tests

f1c56e1

* Merge skip/write/missing-entry tests into one multi-scenario method following the test_feature_resync pattern * Remove redundant partial-change and empty-dict sub-cases Signed-off-by: Bojun Feng <bojundf@gmail.com>

Bojun-Feng marked this pull request as ready for review March 31, 2026 03:25

tirupatihemanth requested a review from Copilot March 31, 2026 20:28

Copilot started reviewing on behalf of tirupatihemanth March 31, 2026 20:29 View session

Copilot AI reviewed Mar 31, 2026

View reviewed changes

Comment thread scripts/featured Outdated

Comment thread scripts/featured Outdated

Comment thread scripts/featured Outdated

Comment thread tests/featured/featured_test.py

Comment thread tests/featured/featured_test.py

tirupatihemanth requested review from dgsudharsan and tirupatihemanth March 31, 2026 20:48

dgsudharsan requested changes Apr 1, 2026

View reviewed changes

Comment thread scripts/featured Outdated

[featured]: Address comments

1a87fee

Signed-off-by: Bojun-Feng <bojundf@gmail.com>

Bojun-Feng requested a review from dgsudharsan April 1, 2026 06:05

dgsudharsan approved these changes Apr 1, 2026

View reviewed changes

dgsudharsan added the Request for 202511 Branch label Apr 1, 2026

tirupatihemanth reviewed Apr 1, 2026

View reviewed changes

Comment thread scripts/featured Outdated

tirupatihemanth approved these changes Apr 1, 2026

View reviewed changes

[featured]: Extract duplicated scope update logic into helper method

5efbc8c

Signed-off-by: Bojun-Feng <bojundf@gmail.com>

vmittal-msft added the Approved for 202511 Branch label Apr 6, 2026

rookie-who approved these changes May 7, 2026

View reviewed changes

vaibhavhd approved these changes May 7, 2026

View reviewed changes

saiarcot895 approved these changes May 7, 2026

View reviewed changes

saiarcot895 merged commit c5bbbe8 into sonic-net:master May 7, 2026
6 checks passed

mssonicbld mentioned this pull request May 7, 2026

[action] [PR:368] [featured]: Skip redundant CONFIG_DB writes in sync_feature_scope #382

Merged

mssonicbld added the Created PR to 202511 Branch label May 7, 2026

mssonicbld added Included in 202511 Branch and removed Created PR to 202511 Branch labels May 7, 2026

Bojun-Feng deleted the bug/featured-redundant-configdb-writes branch May 7, 2026 18:54

vmittal-msft added Cherry Pick Conflict_202511 Created PR to 202511 Branch labels May 7, 2026

mssonicbld removed the Cherry Pick Conflict_202511 label May 7, 2026

Conversation

Bojun-Feng commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why I did it

How I did it

How to verify it

Description for the changelog

Uh oh!

mssonicbld commented Mar 31, 2026

Uh oh!

azure-pipelines Bot commented Mar 31, 2026

Uh oh!

mssonicbld commented Mar 31, 2026

Uh oh!

azure-pipelines Bot commented Mar 31, 2026

Uh oh!

mssonicbld commented Mar 31, 2026

Uh oh!

azure-pipelines Bot commented Mar 31, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tirupatihemanth commented Mar 31, 2026

Uh oh!

Uh oh!

mssonicbld commented Apr 1, 2026

Uh oh!

azure-pipelines Bot commented Apr 1, 2026

Uh oh!

Uh oh!

tirupatihemanth left a comment

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Apr 1, 2026

Uh oh!

azure-pipelines Bot commented Apr 1, 2026

Uh oh!

Bojun-Feng commented Apr 20, 2026

Uh oh!

saiarcot895 commented May 2, 2026

Uh oh!

Bojun-Feng commented May 2, 2026

Uh oh!

rookie-who left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mssonicbld commented May 7, 2026

Uh oh!

mssonicbld commented May 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Bojun-Feng commented Mar 31, 2026 •

edited

Loading