Add 6 L3VNI DCI test cases (57, 58, 84, 85, 96, 97)

[bpar9]

Description of PR

Summary:
Adds 6 new L3VNI DCI test cases from the DCI Solution Testplan to the VXLAN DCI test automation framework, plus 4 supporting helper functions in vxlan_helper.py.

All test cases use continuous traffic via dci_flap_continuous streams (per reviewer feedback), following the InterfaceTrigger / test_dci_link_trigger pattern: check_traffic(action='start') → trigger → check_traffic(action='check'|'stop').

New test cases (in TestVxlanDCIBase):

Testplan ID	Method	Description
L3VNI_dci:57/58	test_base_dci_l3vni_mh_df_failover[v4/v6]	DF failover on DC2 MH leafs with continuous cross-DC L3VNI traffic
L3VNI_dci:84	test_base_dci_l3vni_simultaneous_bgw_reboot	Simultaneous reboot of both DC1 BGWs with continuous L2+L3 traffic
L3VNI_dci:85	test_base_dci_l3vni_rolling_reboot	Rolling reboot of DC1 fabric with continuous L2+L3 traffic
L3VNI_dci:96	test_base_dci_l3vni_remove_add_import_export_rt	Remove/re-add RT import/export on DC1 BGWs with continuous L2+L3 traffic
L3VNI_dci:97	test_base_dci_l3vni_remove_add_rt_rewrite_configs	Remove/re-add RT-REWRITE route-maps on DC1 BGWs with continuous L2+L3 traffic

New helper functions (in vxlan_helper.py):

• remove_bgw_import_export_rt() / add_bgw_import_export_rt() — targeted removal and restoration of BGP VRF route-target import/export statements
• remove_bgw_rt_rewrite_maps() / add_bgw_rt_rewrite_maps() — targeted removal and restoration of RT-REWRITE-WAN/DC route-maps, extcommunity-lists, and neighbor application

Updates since last revision

Round 4 — MH-only streams, disable-at-start removal, TC96/97 L2+L3 (reported by @vallabh78):

10. Removed disable-at-TC-start blocks from all 6 TCs: Streams are disabled by default — the explicit stop+disable at TC start was unnecessary and added execution time. Removed from TC57/58, TC84, TC85, TC96, TC97. Streams are now only enabled via check_traffic(action='start') during test execution and disabled in finally blocks at TC exit.

11. MH-only continuous streams in tgen_preconfig:

    • L2: _dci_fc_endpoints_for_vlan() now only uses l2_pc_cross endpoints (dropped l2_orphan_cross). Name prefix changed to DCI-FC-L2-MH-X-v{vlan}.
    • L3: New filtering selects only PortChannel-sourced (src_int.startswith('PortChannel')) L3 cross-DC endpoints, limited to 1 VLAN per VRF. Name prefix DCI-FC-L3-MH-X.

12. TC57/58 stream filtering — 1 L2 + L3 per VRF: After version filtering (v4 or v6), the TC now picks exactly 1 L2 stream + L3 streams (1 per VRF) by parsing L2/L3 and VRF tags from stream name field.

13. TC96/97 use ALL L2+L3 streams: Removed the VRF-based filtering that limited to 1 stream per VRF (which only picked L2). Both TCs now start/check/stop all dci_flap_continuous streams — L2 (VLANs 12, 18) + L3 (v4 + v6).

Round 3 — PortChannel discovery still failing + TC96/97 stream count (reported by @vallabh78):

8. PortChannel discovery third fallback (TC57/58): Added a third fallback that searches test_cfg['nodes'].get('leaf', []) for any node with 'dc2' in the name and port_channels data present. Also added debug logging when no PortChannels are found.

9. TC96/97 stream filtering — 1 VLAN per VRF: Superseded by item 13 above.

Round 2 — streams enabled before TC starts (reported by @vallabh78):

7. Disable all continuous streams at TC start: Superseded by item 10 above — disable-at-start blocks removed entirely.

Round 1 — 5 issues from testbed run (reported by @vallabh78):

1. PortChannel discovery fix (TC57/58): Replaced config_dict[leaf]['port_channel'] + get_config_interfaces_list() fallback with test_cfg.get(leaf, {}).get('port_channels', []) pattern — matches _get_leaf_portchannels_by_dc() in InterfaceTrigger.

2. Stream filtering by IP version (TC57/58): TC57 now runs only ipv4 continuous streams, TC58 only ipv6. Filters dci_flap_continuous dict by the new version field stored in each stream handle.

3. Stream disable after stop (all 6 TCs): Added explicit tg_traffic_config(mode='disable', stream_id=...) after every check_traffic(action='stop') in finally blocks so streams are not left enabled between test executions.

4. rate_percent=0.01 (all 6 TCs): Before starting continuous traffic, each stream is reconfigured via tg_traffic_config(mode='modify', stream_id=sid, rate_percent=0.01).

5. DF/NDF verification via show evpn es (TC57/58): Uses get_evpn_es() before shutdown to determine which leaf is DF (has L in type) vs NDF (has N in type). Prints show evpn es output on both DC2 leafs after PortChannel shutdown for debugging.

6. create_traffic_item change (vxlan_helper.py): Added name and version fields to stream_handles dict to enable stream filtering by IP version.

⚠️ Human review checklist

• MH-only L2 continuous streams: _dci_fc_endpoints_for_vlan() now only iterates l2_pc_cross. If l2_pc_cross is empty (no PortChannel cross-DC L2 endpoints), no L2 continuous streams will be created. Verify l2_pc_cross is populated for the testbed topology.
• MH-only L3 continuous streams: L3 continuous stream creation filters l3_cross_dc_endpoints by src_int.startswith('PortChannel') and limits to 1 VLAN per VRF. If no PortChannel-sourced L3 endpoints exist, only L2 streams are available. Verify this is acceptable.
• TC57/58 stream name parsing: Picks streams by checking 'L2' in name and 'L3' in name, then parses VRF from _vrf in name. This relies on name_prfx values DCI-FC-L2-MH-X-v{vlan} and DCI-FC-L3-MH-X set in tgen_preconfig. Verify stream names on testbed match these patterns.
• TC96/97 running ALL streams: Both TCs now start/check/stop all dci_flap_continuous streams (L2+L3, v4+v6). If there are many streams, verify IXIA handles this correctly and the test doesn't take excessive time.
• DC2 leaf discovery fallback chain: Three fallbacks tried in order: l2l3vni_bgw → l2l3vni → leaf (filtered by 'dc2' + has port_channels). Verify that test_cfg['nodes'].get('leaf', []) exists and contains DC2 leaf names on the testbed YAML.
• Stream version filtering correctness: TC57/58 filters fc_streams by v.get('version', '') == ixia_version. This relies on the version field added to create_traffic_item. Verify that tgen_preconfig creates dci_flap_continuous streams through create_traffic_item so they'll have this field populated.
• DF/NDF detection with multiple ES entries: The loop assigns df_leaf/ndf_leaf based on PortChannel ES entries. If a leaf has multiple PortChannel ESIs with mixed DF/NDF flags, the last match wins. Verify this is acceptable for the DC2 topology.
• tg_traffic_config(mode='disable', stream_id=list): Verify the IXIA API accepts a list of stream IDs vs requiring individual calls.
• rate_percent=0.01 persistence: Rate is set via mode='modify' before check_traffic(action='start') which calls mode='enable'. Verify enable doesn't reset rate_percent.

Type of change

• Bug fix
• Testbed and Framework(new/improvement)
• New Test case
  • Skipped for non-supported platforms
• Test case improvement

Back port request

(none)

Approach

What is the motivation for this PR?

Expand L3VNI DCI test coverage per the DCI Solution Testplan for multi-homing failover, system-level triggers (simultaneous/rolling reboots), and L3VNI config manipulation (RT and RT-REWRITE).

How did you do it?

• Test cases follow existing patterns in TestVxlanDCIBase (banner/log conventions, report_result, verify_base_setup_bgw, check_core).
• Continuous traffic uses tgen_handles['dci_flap_continuous'] with the check_traffic() three-phase pattern (start → check/stop). All traffic stop calls are in finally blocks to guarantee cleanup, followed by explicit disable.
• Stream lifecycle: Streams are disabled by default. TCs enable only the needed streams via check_traffic(action='start'), and stop+disable in finally blocks at TC exit.
• MH-only continuous streams: Both L2 and L3 continuous streams in tgen_preconfig are restricted to PortChannel (multihomed) sources only. L2 uses l2_pc_cross endpoints for VLANs 12, 18. L3 filters to PortChannel-sourced endpoints with 1 VLAN per VRF.
• DF failover (57/58) is parametrized on ip_version, filters streams to matching version only (then picks 1 L2 + L3 per VRF), uses show evpn es for DF/NDF identification, and uses a try/finally to guarantee PortChannel restoration. DC2 leaf discovery has a 3-tier fallback (l2l3vni_bgw → l2l3vni → leaf with port_channels check).
• Simultaneous BGW reboot (84) uses threading.Thread to reboot both BGWs concurrently.
• Rolling reboot (85) sequentially reboots each node and waits for docker recovery before proceeding.
• RT manipulation tests (96/97) wrap the remove/add cycle in try/except with recovery logic to restore config on failure. Both use all dci_flap_continuous streams (L2 + L3 v4/v6).
• Helper functions reuse _get_l3vni_bgw_params() for consistent parameter derivation and generate targeted FRR config strings (rather than full delete/recreate via delete_l3vni_bgw_frr_config).
• All 6 TCs configure rate_percent=0.01 on continuous streams before starting.

How did you verify/test it?

Python syntax verified (py_compile). Reviewer @vallabh78 ran TC57/58/84/85/96/97 on live testbed and reported issues addressed across four revision rounds.

Any platform specific information?

Requires 3-DC EVPN-VXLAN topology with BGW spines, IXIA traffic generation, and EVPN multi-homing on DC2.

Supported testbed topology if it's a new test case?

3-DC DCI topology: DC1 (4 leafs + 2 BGW spines), DC2 (2 leafs + 2 BGW spines), DC3 (1 leaf + 1 BGW spine).

Documentation

Test cases map to L3VNI_dci:57, 58, 84, 85, 96, 97 in DCI_Solution_Testplan.xlsx.

Link to Devin session: https://cisco-demo.devinenterprise.com/sessions/356a7bf56e254faaa8e996cefe35b12e
Requested by: @bpar9

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[vallabh78]

For all the testcases, we want to use continuous traffic. Refer Interfacetrigger class - dci_flap_continuous
Check this file to see if the testcase steps matches using continuous traffic.
tc.txt

[devin-ai-integration]

Thanks for the feedback @vallabh78! I've refactored all 6 test cases to use continuous traffic via dci_flap_continuous streams, following the InterfaceTrigger / test_dci_link_trigger pattern.

Changes made (commit 290621b):

• TC57/58 (DF failover): check_traffic(action='start') before PortChannel shutdown → check_traffic(action='check', min_perc=99.6) after failover → check_traffic(action='stop') in finally block
• TC84 (simultaneous BGW reboot): Start continuous L2+L3 traffic before reboot → stop after recovery (drop not measured per tc.txt) → burst verify across+within DC
• TC85 (rolling reboot): Same pattern as TC84 — continuous traffic during reboot, drop not measured, burst verification after recovery
• TC96 (RT remove/add): Start continuous traffic → remove/re-add RT → check_traffic(action='check', min_perc=99.6) to verify recovery → stop in finally
• TC97 (RT-REWRITE remove/add): Same pattern as TC96

I also reviewed the tc.txt steps and aligned the implementation accordingly. Please take another look when you get a chance.

[vallabh78]

After running all 4 testcases, below are the issues i found:

1. For L3VNI_dci:57/58 - No portchannel found for DC2 leafs.
   2026-04-09 22:02:09,069 T0000: INFO ################################################################################ 2026-04-09 22:02:09,069 T0000: INFO # Step 2: Pick Leaf0 and Leaf1 of DC2 as DF/NDF pair # 2026-04-09 22:02:09,070 T0000: INFO ################################################################################ 2026-04-09 22:02:23,636 T0000: INFO DC2 MH leaf pair: ['leaf0_dc2', 'leaf1_dc2'] 2026-04-09 22:03:34,646 T0000: INFO leaf0_dc2 PortChannels: [] 2026-04-09 22:04:19,227 T0000: INFO leaf1_dc2 PortChannels: []

2. For tc57 - We only want to run v4 continuous traffic. I see we are running all streams. Same for v6 testcase

3. Continuous streams are enabled all the time. we enable them only during the tc execution and then disable them

4. For continuous traffic, can we configure the rate_percent to 0.01?

5. Use show vxlan evpn es output to check df and ndf. the one with N is NDF for that ESI. Also, print it after we shut the portchannel.
   cisco@sonic:~$ show evpn es Type: B bypass, L local, R remote, N non-DF ESI Type ES-IF VTEPs 00:02:03:04:05:06:07:08:09:0d LR PortChannel4 2000:1::6 cisco@sonic:~$ show evpn es Type: B bypass, L local, R remote, N non-DF ESI Type ES-IF VTEPs 00:02:03:04:05:06:07:08:09:0d LRN PortChannel4 2000:1::5

[devin-ai-integration]

@vallabh78 Thanks for the detailed feedback from the testbed run! All 5 issues have been addressed in commit 5cc4e9d:

1. PortChannel discovery (TC57/58): Replaced the config_dict[leaf]['port_channel'] + get_config_interfaces_list() approach with test_cfg.get(leaf, {}).get('port_channels', []) — same pattern as _get_leaf_portchannels_by_dc() in InterfaceTrigger. This reads directly from the config file and constructs PortChannel{num} from the port_channel_num key.

2. Stream filtering by IP version (TC57/58): Added version field to stream_handles dict in create_traffic_item() (vxlan_helper.py). TC57 now filters dci_flap_continuous to only ipv4 streams, TC58 to only ipv6 streams.

3. Stream disable after stop (all 6 TCs): Added explicit tg_traffic_config(mode='disable', stream_id=...) after every check_traffic(action='stop') in finally blocks across all 6 test cases.

4. rate_percent=0.01 (all 6 TCs): Before starting continuous traffic, each stream is reconfigured via tg_traffic_config(mode='modify', stream_id=sid, rate_percent=0.01).

5. DF/NDF via show evpn es (TC57/58): Uses get_evpn_es() before shutdown to determine DF (type has L without N) vs NDF (type has N). Prints show evpn es output on both DC2 leafs after PortChannel shutdown for debugging.

[vallabh78]

Still see this issue where these streams are enabled even before the testcase starts.

[devin-ai-integration]

@vallabh78 Fixed in commit f1a448f — all 6 TCs now explicitly stop + disable all dci_flap_continuous streams at the very beginning of each test case, before any test logic runs.

The sequence is:

1. TC start: tg_traffic_control(action='stop') + tg_traffic_config(mode='disable') on all continuous streams
2. TC execution: check_traffic(action='start') enables only the needed streams
3. TC cleanup (finally): check_traffic(action='stop') + tg_traffic_config(mode='disable') again

This ensures streams are never left enabled between test executions, and are only active during the test execution phase.

[vallabh78]

portchannel again failing:
(Pdb) leaf_cfg.get('port_channel')
(Pdb) leaf_cfg.get('port_channels')
[{'port_channel_num': 4, 'member_ids': ['T1P2', 'T1P3'], 'sys_mac': '00:44:33:22:11:44', 'evpn_esi': '00:02:03:04:05:06:07:08:09:0d'}]

L3VNI_dci:96 and 97, we can have just 1VLAN each vrf for continuous traffic and only enable those 2 streams during the testcase.

@vallabh78 Thanks for the detailed feedback from the testbed run! All 5 issues have been addressed in commit 5cc4e9d:

1. PortChannel discovery (TC57/58): Replaced the config_dict[leaf]['port_channel'] + get_config_interfaces_list() approach with test_cfg.get(leaf, {}).get('port_channels', []) — same pattern as _get_leaf_portchannels_by_dc() in InterfaceTrigger. This reads directly from the config file and constructs PortChannel{num} from the port_channel_num key.
2. Stream filtering by IP version (TC57/58): Added version field to stream_handles dict in create_traffic_item() (vxlan_helper.py). TC57 now filters dci_flap_continuous to only ipv4 streams, TC58 to only ipv6 streams.
3. Stream disable after stop (all 6 TCs): Added explicit tg_traffic_config(mode='disable', stream_id=...) after every check_traffic(action='stop') in finally blocks across all 6 test cases.
4. rate_percent=0.01 (all 6 TCs): Before starting continuous traffic, each stream is reconfigured via tg_traffic_config(mode='modify', stream_id=sid, rate_percent=0.01).
5. DF/NDF via show evpn es (TC57/58): Uses get_evpn_es() before shutdown to determine DF (type has L without N) vs NDF (type has N). Prints show evpn es output on both DC2 leafs after PortChannel shutdown for debugging.

[devin-ai-integration]

@vallabh78 Both issues addressed in commit 6526379:

1. PortChannel discovery (TC57/58):
Added a third fallback that searches test_cfg['nodes'].get('leaf', []) for DC2 leafs with port_channels data. This covers the case where DC2 leafs aren't in l2l3vni_bgw or l2l3vni node lists but are in the leaf list. Also added debug logging that prints test_cfg keys for a leaf when no PortChannels are found, to help troubleshoot if it fails again.

2. TC96/97 stream filtering (1 VLAN per VRF):
Both TC96 and TC97 now filter fc_streams to only 2 streams — 1 per VRF. The filter extracts the VRF tag from the stream name field (e.g. _vrf101, _vrf102) and picks the first stream found for each VRF. The filtered _fc_subset dict is used for check_traffic(action='start'), check_traffic(action='check'), and check_traffic(action='stop'). All streams are still disabled at TC start and TC end (existing pattern unchanged).

[vallabh78]

@vallabh78 Both issues addressed in commit 6526379:

1. PortChannel discovery (TC57/58): Added a third fallback that searches test_cfg['nodes'].get('leaf', []) for DC2 leafs with port_channels data. This covers the case where DC2 leafs aren't in l2l3vni_bgw or l2l3vni node lists but are in the leaf list. Also added debug logging that prints test_cfg keys for a leaf when no PortChannels are found, to help troubleshoot if it fails again.

2. TC96/97 stream filtering (1 VLAN per VRF): Both TC96 and TC97 now filter fc_streams to only 2 streams — 1 per VRF. The filter extracts the VRF tag from the stream name field (e.g. _vrf101, _vrf102) and picks the first stream found for each VRF. The filtered _fc_subset dict is used for check_traffic(action='start'), check_traffic(action='check'), and check_traffic(action='stop'). All streams are still disabled at TC start and TC end (existing pattern unchanged).

1. Are we creating flap_continuous streams for all vlans by default? for both L2 and L3 traffic? if yes lets keep only 2 vlans per vlans for both L2 and L3 and same if we not creating default streams.

2. Its port_channels not port_channel
   (Pdb) leaf_cfg.get('port_channels')
   [{'port_channel_num': 4, 'member_ids': ['T1P2', 'T1P3'], 'sys_mac': '00:44:33:22:11:44', 'evpn_esi': '00:02:03:04:05:06:07:08:09:0d'}]

[devin-ai-integration]

@vallabh78 Responding to both points:

1. Flap_continuous stream creation:
The dci_flap_continuous streams are already limited — not created for all VLANs. Specifically:

• L2: Only VLANs 12 and 18 (_dci_fc_vlans = (12, 18)) — that's 1 VLAN per VRF for both IPv4 and IPv6
• L3: One traffic item per address family (v4/v6) using l3_cross_dc_endpoints grouped by VRF (multi_dst='vrf')

So we're already at 2 VLANs (1 per VRF) for L2, and VRF-based grouping for L3. This matches what you requested. The additional filtering in TC96/97 (_fc_subset) then picks just 1 stream per VRF from these to enable during the test.

2. port_channels key:
The code already uses port_channels (plural) — this was fixed in commit 5cc4e9d (line 6801: test_cfg.get(leaf, {}).get('port_channels', [])). The port_channel_num field within each dict entry is also correct — that's the key name inside the YAML data structure. If PortChannel discovery is still failing on your testbed run, it could be that you're running an older commit. The latest commit is 6526379 — please confirm you're running this version.

If the issue persists with the latest commit, the debug logging I added will print the test_cfg keys for the leaf when PCs are empty, which will help us narrow down the root cause.

[vallabh78]

check this log file
results_2026_04_10_10_16_33_logs.log

issues:

1. Why are we disabling streams every testcase. Those streams should be disabled by default and only enabled while running the testcases which are using it. Fix this as this takes time.
2. Tc 57 and 58 0 - We have multiple L3 streams for multiple vlans. We only want streams whose source is Portchannel means Multihomed host and only 1 vlans per VRF and 1 L2 stream
3. For 96 and 97, we want both continuous L2 and L3 (v4+v6) and i see only L2.

[devin-ai-integration]

@vallabh78 All 3 issues from the log file addressed in commit 067a1d5:

1. Removed disable-at-TC-start blocks from all 6 TCs (57/58, 84, 85, 96, 97):
Streams are disabled by default — no need to stop/disable at TC start. Removed the entire tg_traffic_control(action='stop') + tg_traffic_config(mode='disable') block from the beginning of each TC. This saves time at TC start.

2. TC57/58 — MH-only sources, 1 VLAN/VRF for L3, 1 L2 stream:

• tgen_preconfig: L2 continuous streams now only use l2_pc_cross endpoints (dropped orphan sources). Updated name prefix to DCI-FC-L2-MH-X-v{vlan}.
• tgen_preconfig: L3 continuous streams filter to PortChannel (MH) sources only, limited to 1 VLAN per VRF. Name prefix DCI-FC-L3-MH-X.
• TC filtering: After version filtering (v4/v6), picks 1 L2 stream + L3 streams (1 per VRF) by parsing L2/L3 and VRF tags from stream names.

3. TC96/97 — both L2 AND L3 (v4+v6) continuous streams:
Removed the VRF-based filtering that limited to 1 stream per VRF (which only picked L2). Both TCs now use all dci_flap_continuous streams — L2 (VLANs 12, 18) + L3 (v4 + v6). All streams are started/checked/stopped together.