chore(test): add gateway peering NAT tests for all cases in 26.01 by pau-hedgehog · Pull Request #1579 · githedgehog/fabricator

pau-hedgehog · 2026-03-19T17:07:58Z

Adds a full NAT test matrix for gateway peerings with external networks, covering all combinations of external type × NAT mode. Also refactors the external selection and NAT spec-building helpers to support the new test variants cleanly.

Part of https://github.com/githedgehog/internal/issues/305

Background

Gateway peering connects VPCs (or a VPC to an upstream network) through a dedicated gateway device that can apply NAT rules. This is different from fabric-native VPC peering: the gateway sits in the data path and can rewrite IP addresses before forwarding.

Externals are upstream network devices (routers/ISPs) that VPCs can reach through a gateway peering. Two types exist:

BGP external — the gateway and the upstream device exchange routes dynamically via BGP. When a peering is created, the gateway advertises the NAT pool CIDR to the upstream router; when it's torn down, the route is withdrawn.
Static external — routes are pre-configured as static entries on the shared edge device (DS2000). No route exchange happens at peering time; the upstream device always has the route.

NAT modes tested (per external type):

Mode	What happens
No NAT	VPC IPs are visible to the external device as-is
Static	Each VPC IP maps 1:1 to a fixed IP in the NAT pool (e.g. `10.50.1.5` → `192.168.85.5`)
Masquerade	All VPC IPs appear as one shared IP — like a home router. Connection-tracked.
Port Forward	Inbound connections on an external port are forwarded to a specific VPC server port
Masquerade + Port Forward	Both outbound masquerade and inbound port-forwarding active simultaneously

Why per-environment NAT pool CIDRs (the annotations)

All test environments share a single physical edge device (DS2000). If two environments used the same NAT pool CIDR, DS2000 could not route return traffic to the correct environment's gateway — it would go to whichever environment last advertised the route.

Each environment's External object carries an annotation with its assigned pool:

hhfab.githedgehog.com/test-bgp-nat-pool:    192.168.85.0/24  # env-5
hhfab.githedgehog.com/test-static-nat-pool: 192.168.81.0/24  # env-5

Tests read the annotation at runtime and skip if it's absent, so environments without the annotation configured are not affected.

The "inverted" port-forward tests

Port-forward is an inbound feature — an external client initiates a connection to a NATed IP, and the gateway forwards it to a server behind NAT. In our lab, "the external client" would be DS2000, but we have no SSH access to DS2000 to trigger connections from it.

The workaround: flip the NAT onto the external side. The external device's IP is exposed behind a virtual IP in the NAT pool (.200 by convention), with a port-forward rule 5201→15201. A VPC server then acts as the iperf3 client connecting to <natpool>.200:15201, and the gateway forwards that to <ds2000-ip>:5201 where an iperf3 server is listening. This exercises the port-forward dataplane without needing outbound access from DS2000.

Why MasqueradePortForward produces two expose entries

The gateway API schema does not allow a single expose entry to carry both NAT.Masquerade and NAT.PortForward blocks simultaneously. The workaround (buildExposes) emits two expose entries with identical IPs/As fields but different NAT blocks. The gateway processes both and applies both rule types to traffic.

Why BGP tests retry for up to 2 minutes after WaitReady

WaitReady confirms that fabric switches have programmed their forwarding tables. But for BGP externals, there is additional work: the gateway must advertise the NAT pool CIDR to DS2000 via BGP, and DS2000 must accept and install that route. This propagation takes extra time that is not visible from the fabric side. Static externals skip the wait because DS2000 already has a static route for the pool — there is nothing to propagate.

Changes

rt_nat_external_tests.go (new) — 10 new test cases covering the full BGP × static × NAT mode matrix, plus connectivity helpers (testNATExternalConnectivity, testIperfToExternal)
rt_utils.go — new NAT modes (NATModePortForward, NATModeMasqueradePortForward), new buildExposes helper, appendGwExtPeeringSpecWithNAT, appendGwExtInvertedPortForwardPeeringSpec; appendGwPeeringSpec now returns an error
rt_base.go — findExternals replaces inline external-selection loops; separately tracks extName (BGP) and staticExtName; NoStaticExternals skip flag is now wired into selectAndRunSuite
rt_multi_vpc_single_subnet_suite.go / rt_single_vpc_suite.go / rt_nat_tests.go — updated callers of appendGwPeeringSpec to handle the new error return
testing.go — removed expose.As guard that blocked NAT peerings from being verified
.github/workflows/run-vlab.yaml — pin lab-ci checkout to pau/nat-tests-annotations (contains per-environment NAT pool annotation configuration)

Copilot

Pull request overview

Adds comprehensive gateway peering NAT coverage to the hhfab release tests, expanding beyond the existing static/masquerade cases to include port-forward and combined NAT modes across internal VPC peerings and gateway externals.

Changes:

Extend gateway peering spec helpers to support port-forward NAT and combined masquerade+port-forward (via split expose entries).
Add new NAT test cases for internal VPC peerings (port-forward only, masquerade+port-forward).
Add a new external NAT test suite covering BGP and static externals, and wire it into the multi-VPC single-subnet suite; improve external selection/skip signaling.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
pkg/hhfab/testing.go	Removes a hard “unsupported” check so peering presence logic can tolerate `expose.As` (needed for NAT cases).
pkg/hhfab/rt_utils.go	Adds NAT-mode helpers (port-forward + combined), external NAT annotation helpers, and an inverted external port-forward peering builder.
pkg/hhfab/rt_nat_tests.go	Adds internal gateway peering tests for port-forward NAT and masquerade+port-forward NAT, including a new inbound port-forward connectivity helper.
pkg/hhfab/rt_nat_external_tests.go	Introduces a new external NAT test matrix for BGP and static externals (static/masq/port-forward/combined) with convergence handling.
pkg/hhfab/rt_multi_vpc_single_subnet_suite.go	Registers the external NAT test cases into the multi-VPC single-subnet suite.
pkg/hhfab/rt_base.go	Adds `findExternals` and enhances skip-flag handling for viable BGP vs static externals.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

github-actions · 2026-03-19T23:14:03Z

Release Tests

9 files ± 0 36 suites ±0 4h 14m 30s ⏱️ + 8m 37s
45 tests + 12 43 ✅ +12 2 💤 ± 0 0 ❌ ±0
405 runs +108 129 ✅ +18 276 💤 +90 0 ❌ ±0

Results for commit 357e6a5. ± Comparison against base commit 2632914.

♻️ This comment has been updated with latest results.

pau-hedgehog · 2026-03-23T09:06:23Z

The tbr commit (1b5a833) can be removed once githedgehog/lab-ci#17 is merged

pau-hedgehog · 2026-03-23T09:57:12Z

This passes on env-1:

14:32:38 INF *** Recap of the test results ***
14:32:38 INF Test suite results suite="No VPCs Suite"
14:32:38 INF PASS test="Breakout ports"
14:32:38 INF PASS test="Loki Observability"
14:32:38 INF PASS test="Prometheus Observability"
14:32:38 INF Test suite summary tests=3 passed=3 skipped=0 failed=0 duration=4m47s
14:32:38 INF Test suite results suite="Single VPC Suite"
14:32:38 INF PASS test="No restrictions"
14:32:38 INF PASS test="Single VPC with restrictions"
14:32:38 INF PASS test="DNS/NTP/MTU/DHCP lease"
14:32:38 INF PASS test="DHCP renewal"
14:32:38 INF PASS test="DHCP static lease"
14:32:38 WRN SKIP test="MCLAG Failover" reason="no MCLAG connections found"
14:32:38 WRN SKIP test="ESLAG Failover" reason="no ESLAG connections found"
14:32:38 WRN SKIP test="Bundled Failover" reason="no bundled connections found"
14:32:38 WRN SKIP test="Spine Failover" reason="There are no fabric (i.e. spine-leaf) links between the switches"
14:32:38 INF PASS test="Mesh Failover"
14:32:38 INF PASS test="RoCE flag and basic traffic marking"
14:32:38 INF Test suite summary tests=11 passed=7 skipped=4 failed=0 duration=24m38s
14:32:38 INF Test suite results suite="Multi-Subnet Multi-VPC Suite"
14:32:38 INF PASS test="Multi-Subnets no restrictions"
14:32:38 INF PASS test="Multi-Subnets isolation"
14:32:38 INF PASS test="Multi-Subnets with filtering"
14:32:38 INF PASS test=StaticExternal
14:32:38 INF Test suite summary tests=4 passed=4 skipped=0 failed=0 duration=16m32s
14:32:38 INF Test suite results suite="Multi-VPC Single-Subnet Suite"
14:32:38 WRN SKIP test="Starter Test" reason="not enough VPCs found"
14:32:38 INF PASS test="Only Externals"
14:32:38 INF PASS test="Full Mesh All Externals"
14:32:38 INF PASS test="Full Loop All Externals"
14:32:38 WRN SKIP test="Sergei's Special Test" reason="not enough VPCs found"
14:32:38 INF PASS test="Gateway Peering"
14:32:38 INF PASS test="Gateway Failover"
14:32:38 INF PASS test="Gateway Peering Loop"
14:32:38 INF PASS test="Mixed VPC and Gateway Peering Loop"
14:32:38 INF PASS test="Mixed Gateway and Fabric External Peering"
14:32:38 INF PASS test="Static External Peering"
14:32:38 INF PASS test="Gateway Peering Masquerade Source NAT"
14:32:38 INF PASS test="Gateway Peering Static Source NAT"
14:32:38 INF PASS test="Gateway Peering Bidirectional Static NAT"
14:32:38 INF PASS test="Gateway Peering Overlap NAT"
14:32:38 INF PASS test="Gateway Peering Port Forward NAT"
14:32:38 INF PASS test="Gateway Peering Masquerade and Port Forward NAT"
14:32:38 INF PASS test="BGP External No NAT"
14:32:38 INF PASS test="BGP External Static NAT"
14:32:38 INF PASS test="BGP External Masquerade NAT"
14:32:38 INF PASS test="BGP External Port Forward NAT"
14:32:38 INF PASS test="BGP External Masquerade and Port Forward NAT"
14:32:38 INF PASS test="Static External No NAT"
14:32:38 INF PASS test="Static External Static NAT"
14:32:38 INF PASS test="Static External Masquerade NAT"
14:32:38 INF PASS test="Static External Port Forward NAT"
14:32:38 INF PASS test="Static External Masquerade and Port Forward NAT"
14:32:38 INF Test suite summary tests=27 passed=25 skipped=2 failed=0 duration=47m5s
14:32:38 INF All tests completed duration=1h33m3.035286959s

And env-5:

02:51:54 INF *** Recap of the test results ***
02:51:54 INF Test suite results suite="No VPCs Suite"
02:51:54 INF PASS test="Breakout ports"
02:51:54 INF PASS test="Loki Observability"
02:51:54 INF PASS test="Prometheus Observability"
02:51:54 INF Test suite summary tests=3 passed=3 skipped=0 failed=0 duration=2m46s
02:51:54 INF Test suite results suite="Single VPC Suite"
02:51:54 INF PASS test="No restrictions"
02:51:54 INF PASS test="Single VPC with restrictions"
02:51:54 INF PASS test="DNS/NTP/MTU/DHCP lease"
02:51:54 INF PASS test="DHCP renewal"
02:51:54 INF PASS test="DHCP static lease"
02:51:54 WRN SKIP test="MCLAG Failover" reason="no MCLAG connections found"
02:51:54 WRN SKIP test="ESLAG Failover" reason="L3VNI mode is not compatible with ESLAG"
02:51:54 INF PASS test="Bundled Failover"
02:51:54 WRN SKIP test="Spine Failover" reason="There are no fabric (i.e. spine-leaf) links between the switches"
02:51:54 INF PASS test="Mesh Failover"
02:51:54 INF PASS test="RoCE flag and basic traffic marking"
02:51:54 INF Test suite summary tests=11 passed=8 skipped=3 failed=0 duration=22m25s
02:51:54 INF Test suite results suite="Multi-Subnet Multi-VPC Suite"
02:51:54 INF PASS test="Multi-Subnets no restrictions"
02:51:54 INF PASS test="Multi-Subnets isolation"
02:51:54 INF PASS test="Multi-Subnets with filtering"
02:51:54 INF PASS test=StaticExternal
02:51:54 INF Test suite summary tests=4 passed=4 skipped=0 failed=0 duration=12m44s
02:51:54 INF Test suite results suite="Multi-VPC Single-Subnet Suite"
02:51:54 WRN SKIP test="Starter Test" reason="not enough VPCs found"
02:51:54 INF PASS test="Only Externals"
02:51:54 INF PASS test="Full Mesh All Externals"
02:51:54 INF PASS test="Full Loop All Externals"
02:51:54 WRN SKIP test="Sergei's Special Test" reason="not enough VPCs found"
02:51:54 INF PASS test="Gateway Peering"
02:51:54 INF PASS test="Gateway Failover"
02:51:54 INF PASS test="Gateway Peering Loop"
02:51:54 INF PASS test="Mixed VPC and Gateway Peering Loop"
02:51:54 INF PASS test="Mixed Gateway and Fabric External Peering"
02:51:54 INF PASS test="Static External Peering"
02:51:54 INF PASS test="Gateway Peering Masquerade Source NAT"
02:51:54 INF PASS test="Gateway Peering Static Source NAT"
02:51:54 INF PASS test="Gateway Peering Bidirectional Static NAT"
02:51:54 INF PASS test="Gateway Peering Overlap NAT"
02:51:54 INF PASS test="Gateway Peering Port Forward NAT"
02:51:54 INF PASS test="Gateway Peering Masquerade and Port Forward NAT"
02:51:54 INF PASS test="BGP External No NAT"
02:51:54 INF PASS test="BGP External Static NAT"
02:51:54 INF PASS test="BGP External Masquerade NAT"
02:51:54 INF PASS test="BGP External Port Forward NAT"
02:51:54 INF PASS test="BGP External Masquerade and Port Forward NAT"
02:51:54 INF PASS test="Static External No NAT"
02:51:54 INF PASS test="Static External Static NAT"
02:51:54 INF PASS test="Static External Masquerade NAT"
02:51:54 INF PASS test="Static External Port Forward NAT"
02:51:54 INF PASS test="Static External Masquerade and Port Forward NAT"
02:51:54 INF Test suite summary tests=27 passed=25 skipped=2 failed=0 duration=45m13s
02:51:54 INF All tests completed duration=1h23m8.396663165s

There are some issues on env-ci-1 which I've been investigating and so far I believe we may need to raise a case:
https://github.com/githedgehog/fabricator/actions/runs/23374577938/job/68004359172

I may add Gateway Peering to the test names but I think it's ready for review

pau-hedgehog · 2026-03-23T15:35:01Z

Hit this while testing hlab job: #1593

edipascale

thanks @pau-hedgehog, this mostly looks good to me, I'm only concerned about the proliferation of testing methods for the various NAT types and the need to maintain slightly different copies of the same code. FWIW I think it's fine to keep the external related functions separate (and thanks for reusing the checkCurl method there!)

I'm also fine with merging this as is and iterating later on a separate PR, I see no major blocker here

pau-hedgehog · 2026-03-25T08:47:30Z

After the hlab issues I could run a whole tests harness and we have 2 failures we don't observe in env-1/env-5:
https://github.com/githedgehog/fabricator/actions/runs/23496297594/job/68379232155

19:22:11 INF *** Recap of the test results ***
19:22:11 INF Test suite results suite="No VPCs Suite"
19:22:11 INF PASS test="Breakout ports"
19:22:11 INF PASS test="Loki Observability"
19:22:11 INF PASS test="Prometheus Observability"
19:22:11 INF Test suite summary tests=3 passed=3 skipped=0 failed=0 duration=2m51s
19:22:11 INF Test suite results suite="Single VPC Suite"
19:22:11 INF PASS test="No restrictions"
19:22:11 INF PASS test="Single VPC with restrictions"
19:22:11 INF PASS test="DNS/NTP/MTU/DHCP lease"
19:22:11 INF PASS test="DHCP renewal"
19:22:11 INF PASS test="DHCP static lease"
19:22:11 INF PASS test="MCLAG Failover"
19:22:11 INF PASS test="ESLAG Failover"
19:22:11 INF PASS test="Bundled Failover"
19:22:11 INF PASS test="Spine Failover"
19:22:11 WRN SKIP test="Mesh Failover" reason="There are no mesh (i.e. leaf-leaf) links between the switches"
19:22:11 INF PASS test="RoCE flag and basic traffic marking"
19:22:11 INF Test suite summary tests=11 passed=10 skipped=1 failed=0 duration=1h2m20s
19:22:11 INF Test suite results suite="Multi-Subnet Multi-VPC Suite"
19:22:11 INF PASS test="Multi-Subnets no restrictions"
19:22:11 INF PASS test="Multi-Subnets isolation"
19:22:11 INF PASS test="Multi-Subnets with filtering"
19:22:11 INF PASS test=StaticExternal
19:22:11 INF Test suite summary tests=4 passed=4 skipped=0 failed=0 duration=20m37s
19:22:11 INF Test suite results suite="Multi-VPC Single-Subnet Suite"
19:22:11 WRN SKIP test="Starter Test" reason="not enough VPCs found"
19:22:11 INF PASS test="Only Externals"
19:22:11 INF PASS test="Full Mesh All Externals"
19:22:11 INF PASS test="Full Loop All Externals"
19:22:11 INF PASS test="Sergei's Special Test"
19:22:11 INF PASS test="Gateway Peering"
19:22:11 WRN SKIP test="Gateway Failover" reason="not enough gateways found"
19:22:11 ERR FAIL test="Gateway Peering Loop" error="testing gateway loop connectivity: ping server-3 -> server-5: unexpected ping result (sent 5, rcvd 4, expected true)"
19:22:11 INF PASS test="Mixed VPC and Gateway Peering Loop"
19:22:11 INF PASS test="Mixed Gateway and Fabric External Peering"
19:22:11 INF PASS test="Static External Peering"
19:22:11 INF PASS test="Gateway Peering Masquerade Source NAT"
19:22:11 INF PASS test="Gateway Peering Static Source NAT"
19:22:11 INF PASS test="Gateway Peering Bidirectional Static NAT"
19:22:11 INF PASS test="Gateway Peering Overlap NAT"
19:22:11 INF PASS test="Gateway Peering Port Forward NAT"
19:22:11 INF PASS test="Gateway Peering Masquerade and Port Forward NAT"
19:22:11 INF PASS test="Gateway Peering BGP External No NAT"
19:22:11 INF PASS test="Gateway Peering BGP External Static NAT"
19:22:11 INF PASS test="Gateway Peering BGP External Masquerade NAT"
19:22:11 INF PASS test="Gateway Peering BGP External Port Forward NAT"
19:22:11 INF PASS test="Gateway Peering BGP External Masquerade and Port Forward NAT"
19:22:11 ERR FAIL test="Gateway Peering Static External No NAT" error="testing static external connectivity: curl from server-1: should be reachable but curl failed (expected true)"
19:22:11 INF PASS test="Gateway Peering Static External Static NAT"
19:22:11 INF PASS test="Gateway Peering Static External Masquerade NAT"
19:22:11 INF PASS test="Gateway Peering Static External Port Forward NAT"
19:22:11 INF PASS test="Gateway Peering Static External Masquerade and Port Forward NAT"
19:22:11 INF Test suite summary tests=27 passed=23 skipped=2 failed=2 duration=1h23m29s

Frostman

Per the discussion on the fabric sync, @pau-hedgehog, please make sure that ALL gateway-related tests have "Gateway" (with capital G) in the name so we can run them with a simple regex and enable it on the dataplane repo

Frostman · 2026-03-31T18:51:51Z

@pau-hedgehog I've removed ci:+hlab for now as it's not working yet

pau-hedgehog · 2026-03-31T23:02:53Z

@pau-hedgehog I've removed ci:+hlab for now as it's not working yet

OK. Will re-add once ready to see this PR pass (hopefully)

BTW, I added a commit to fail the tests under https://github.com/githedgehog/internal/issues/348

The only reliable way I found was to ping the External IP at least 10 times

21:13:19 INF Test suite results suite="Multi-VPC Single-Subnet Suite"
21:13:19 WRN SKIP test="Starter Test" reason="not enough VPCs found"
21:13:19 INF PASS test="Only Externals"
21:13:19 INF PASS test="Full Mesh All Externals"
21:13:19 INF PASS test="Full Loop All Externals"
21:13:19 WRN SKIP test="Sergei's Special Test" reason="not enough VPCs found"
21:13:19 INF PASS test="Gateway Peering"
21:13:19 INF PASS test="Gateway Failover"
21:13:19 INF PASS test="Gateway Peering Loop"
21:13:19 INF PASS test="Mixed VPC and Gateway Peering Loop"
21:13:19 INF PASS test="Mixed Gateway and Fabric External Peering"
21:13:19 INF PASS test="Static External Peering"
21:13:19 INF PASS test="Gateway Peering Masquerade Source NAT"
21:13:19 INF PASS test="Gateway Peering Static Source NAT"
21:13:19 INF PASS test="Gateway Peering Bidirectional Static NAT"
21:13:19 INF PASS test="Gateway Peering Overlap NAT"
21:13:19 INF PASS test="Gateway Peering Port Forward NAT"
21:13:19 INF PASS test="Gateway Peering Masquerade and Port Forward NAT"
21:13:19 INF PASS test="Gateway Peering BGP External No NAT"
21:13:19 ERR FAIL test="Gateway Peering BGP External Static NAT" error="testing BGP external static NAT connectivity: NAT external connectivity ping stability check: ping server-1 -> 100.100.20.6: unexpected ping result (sent 10, rcvd 9, expected true)"
21:13:19 ERR FAIL test="Gateway Peering BGP External Masquerade NAT" error="testing BGP external masquerade NAT connectivity: NAT external connectivity ping stability check: ping server-1 -> 100.100.20.6: unexpected ping result (sent 10, rcvd 7, expected true)"
21:13:19 INF PASS test="Gateway Peering BGP External Port Forward NAT"
21:13:19 ERR FAIL test="Gateway Peering BGP External Masquerade and Port Forward NAT" error="testing BGP external masquerade+port-forward NAT connectivity: NAT external connectivity ping stability check: ping server-1 -> 100.100.0.6: unexpected ping result (sent 10, rcvd 9, expected true)"
21:13:19 INF PASS test="Gateway Peering Static External No NAT"
21:13:19 INF PASS test="Gateway Peering Static External Static NAT"
21:13:19 INF PASS test="Gateway Peering Static External Masquerade NAT"
21:13:19 INF PASS test="Gateway Peering Static External Port Forward NAT"
21:13:19 INF PASS test="Gateway Peering Static External Masquerade and Port Forward NAT"
21:13:19 INF PASS test="Gateway Peering Negative Both Sides Masquerade NAT"
21:13:19 INF PASS test="Gateway Peering Negative Static Plus Masquerade NAT"
21:13:19 INF PASS test="Gateway Peering Negative Static Plus Port Forward NAT"
21:13:19 INF PASS test="Gateway Peering Negative Nonexistent GatewayGroup"
21:13:19 INF PASS test="Gateway Peering Negative CIDR Overlap"
21:13:19 INF Test suite summary tests=32 passed=27 skipped=2 failed=3 duration=1h6m30s

pau-hedgehog · 2026-04-01T17:51:08Z

After env-ci-1 swap we still face failures on this branch (which passess clean on env-1/env-5):
https://github.com/githedgehog/fabricator/actions/runs/23820690253/job/69432620787

04:23:39 ERR FAIL test="Static External Peering" error="testing static external connectivity: curl from server-5: should be reachable but curl failed (expected true)"
04:23:39 INF PASS test="Gateway Peering Masquerade Source NAT"
04:23:39 INF PASS test="Gateway Peering Static Source NAT"
04:23:39 INF PASS test="Gateway Peering Bidirectional Static NAT"
04:23:39 INF PASS test="Gateway Peering Overlap NAT"
04:23:39 INF PASS test="Gateway Peering Port Forward NAT"
04:23:39 INF PASS test="Gateway Peering Masquerade and Port Forward NAT"
04:23:39 INF PASS test="Gateway Peering BGP External No NAT"
04:23:39 INF PASS test="Gateway Peering BGP External Static NAT"
04:23:39 INF PASS test="Gateway Peering BGP External Masquerade NAT"
04:23:39 INF PASS test="Gateway Peering BGP External Port Forward NAT"
04:23:39 INF PASS test="Gateway Peering BGP External Masquerade and Port Forward NAT"
04:23:39 ERR FAIL test="Gateway Peering Static External No NAT" error="testing static external connectivity: curl from server-1: should be reachable but curl failed (expected true)"
04:23:39 INF PASS test="Gateway Peering Static External Static NAT"
04:23:39 ERR FAIL test="Gateway Peering Static External Masquerade NAT" error="testing static external masquerade NAT connectivity: NAT external connectivity check: curl from server-1: should be reachable but curl failed (expected true)"
04:23:39 ERR FAIL test="Gateway Peering Static External Port Forward NAT" error="testing static external port-forward via iperf3: iperf3 from server-1 to 192.168.71.200:15201: running command: Process exited with status 124"
04:23:39 INF PASS test="Gateway Peering Static External Masquerade and Port Forward NAT"
04:23:39 INF Test suite summary tests=27 passed=22 skipped=1 failed=4 duration=1h35m10s
04:23:39 INF All tests completed duration=2h58m48.786057706s

Static External Peering is failing in master as well: #1627

pau-hedgehog · 2026-04-11T22:39:28Z

This branch is passing consistently now after githedgehog/lab-ci#20

All tests involving Gateway have the word on the test name so we can filter through release test regex

edipascale

Let's get this in!

Add tests for the full NAT matrix (no NAT, static, masquerade, port-forward, masquerade+port-forward) across all three peering target types: internal VPC, BGP external, and static external Signed-off-by: Pau Capdevila <pau@githedgehog.com>

Signed-off-by: Pau Capdevila <pau@githedgehog.com>

pau-hedgehog requested a review from Copilot March 19, 2026 17:07

pau-hedgehog self-assigned this Mar 19, 2026

pau-hedgehog added ci:+release Enable VLAB release tests ci:+hlab Enable hybrid VLAB tests labels Mar 19, 2026

Copilot started reviewing on behalf of pau-hedgehog March 19, 2026 17:08 View session

Copilot AI reviewed Mar 19, 2026

View reviewed changes

Comment thread pkg/hhfab/rt_utils.go

Comment thread pkg/hhfab/rt_base.go Outdated

Comment thread pkg/hhfab/rt_nat_external_tests.go

Comment thread pkg/hhfab/rt_utils.go

Comment thread pkg/hhfab/rt_utils.go Outdated

pau-hedgehog removed the ci:+release Enable VLAB release tests label Mar 20, 2026

pau-hedgehog force-pushed the pau/gw-peering-test branch from 170a1dd to 8e2b5dc Compare March 20, 2026 13:50

pau-hedgehog changed the title ~~chore(test): add gateway peering NAT tests for all cases in issue #305~~ chore(test): add gateway peering NAT tests for all cases in 26.01 Mar 21, 2026

pau-hedgehog force-pushed the pau/gw-peering-test branch 2 times, most recently from 7844d6a to bf540ad Compare March 22, 2026 11:00

pau-hedgehog added ci:+release Enable VLAB release tests and removed ci:+hlab Enable hybrid VLAB tests labels Mar 23, 2026

pau-hedgehog force-pushed the pau/gw-peering-test branch from bf540ad to 1b5a833 Compare March 23, 2026 07:24

pau-hedgehog marked this pull request as ready for review March 23, 2026 09:57

pau-hedgehog requested review from a team as code owners March 23, 2026 09:57

pau-hedgehog added the ci:+hlab Enable hybrid VLAB tests label Mar 23, 2026

pau-hedgehog force-pushed the pau/gw-peering-test branch from 1b5a833 to c275a9d Compare March 23, 2026 15:21

edipascale reviewed Mar 23, 2026

View reviewed changes

Comment thread pkg/hhfab/rt_base.go Outdated

Comment thread pkg/hhfab/rt_nat_tests.go

Comment thread pkg/hhfab/rt_utils.go

Frostman requested changes Mar 23, 2026

View reviewed changes

Comment thread .github/workflows/run-vlab.yaml Outdated

pau-hedgehog force-pushed the pau/gw-peering-test branch 4 times, most recently from 6235ab2 to 5e167fd Compare March 25, 2026 08:41

pau-hedgehog added ci:+release Enable VLAB release tests and removed ci:+release Enable VLAB release tests labels Mar 28, 2026

pau-hedgehog force-pushed the pau/gw-peering-test branch from dc6b610 to 32e95de Compare March 30, 2026 08:22

Frostman requested changes Mar 30, 2026

View reviewed changes

pau-hedgehog force-pushed the pau/gw-peering-test branch from 32e95de to da96854 Compare March 31, 2026 18:43

Frostman removed the ci:+hlab Enable hybrid VLAB tests label Mar 31, 2026

Frostman added the ci:+hlab Enable hybrid VLAB tests label Mar 31, 2026

pau-hedgehog force-pushed the pau/gw-peering-test branch from da96854 to e8a6f34 Compare March 31, 2026 21:39

pau-hedgehog force-pushed the pau/gw-peering-test branch from e8a6f34 to ec9a23e Compare April 1, 2026 21:25

Frostman force-pushed the pau/gw-peering-test branch from ec9a23e to ad1a036 Compare April 2, 2026 04:50

pau-hedgehog force-pushed the pau/gw-peering-test branch 3 times, most recently from bc4896c to f4897b5 Compare April 8, 2026 22:55

Frostman requested review from Frostman and edipascale April 9, 2026 15:42

pau-hedgehog force-pushed the pau/gw-peering-test branch 3 times, most recently from a52679e to 5c2fe59 Compare April 11, 2026 22:37

edipascale approved these changes Apr 15, 2026

View reviewed changes

Comment thread pkg/hhfab/rt_multi_vpc_single_subnet_suite.go

pau-hedgehog added 2 commits April 15, 2026 09:47

chore(release-test): add ICMP probe to external NAT connectivity check

357e6a5

Signed-off-by: Pau Capdevila <pau@githedgehog.com>

pau-hedgehog force-pushed the pau/gw-peering-test branch from 5c2fe59 to 357e6a5 Compare April 15, 2026 07:47

Frostman approved these changes Apr 15, 2026

View reviewed changes

Frostman merged commit 30099eb into master Apr 15, 2026
27 checks passed

Frostman deleted the pau/gw-peering-test branch April 15, 2026 17:42

Conversation

pau-hedgehog commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Background

Why per-environment NAT pool CIDRs (the annotations)

The "inverted" port-forward tests

Why MasqueradePortForward produces two expose entries

Why BGP tests retry for up to 2 minutes after WaitReady

Changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Release Tests

Uh oh!

pau-hedgehog commented Mar 23, 2026

Uh oh!

pau-hedgehog commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pau-hedgehog commented Mar 23, 2026

Uh oh!

edipascale left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pau-hedgehog commented Mar 25, 2026

Uh oh!

Frostman left a comment

Choose a reason for hiding this comment

Uh oh!

Frostman commented Mar 31, 2026

Uh oh!

pau-hedgehog commented Mar 31, 2026

Uh oh!

pau-hedgehog commented Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pau-hedgehog commented Apr 11, 2026

Uh oh!

edipascale left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pau-hedgehog commented Mar 19, 2026 •

edited

Loading

github-actions Bot commented Mar 19, 2026 •

edited

Loading

pau-hedgehog commented Mar 23, 2026 •

edited

Loading

pau-hedgehog commented Apr 1, 2026 •

edited

Loading