docs(roadmap): Test Infra & L2-Native Hardening backlog (#37-47) by gspivey · Pull Request #102 · gspivey/dpdk-stdlib-rust

gspivey · 2026-07-01T10:35:56Z

Adds the sequenced backlog for the testing/L2 program to ROADMAP.md (the agent-router's ordered list).

Context: perf run 28509354005 proved the DPDK TCP engine works on real hardware (tcp-echo real-vfio probe + listened on 10.0.1.10:9000) but failed on harness bugs: TRex started STL while the TCP benchmark needs ASTF, and an SSM TimeoutSeconds=15 (< AWS min 30) that left a stale DPDK process holding the EAL lock.

Items #37–47: fix perf harness (37) → fix SSM/ENI flake trio (38) → wire QUIC + add TCP/UDP-v6 Tier 1/2 tiers (39) → auto-run on PR + dev-merges, Graviton→nightly (40) → nightly perf + auto-PR halt gate + badges (41,42) → publish crates (43) → unify ARP into net crate + backend selection + Containerlab L2/VLAN/VXLAN/Geneve matrix (44,45,46) → publish again (47).

🤖 Generated with Claude Code

Program surfaced by the first end-to-end perf proof (run 28509354005), which validated the DPDK TCP engine on real hardware but exposed perf-harness bugs (TRex STL-vs-ASTF, SSM TimeoutSeconds<30). Sequence: fix perf/integration harness (37,38) -> expand Tier 1/2 functionality coverage (39,40) -> nightly perf + halt gate + badges (41,42) -> publish crates (43) -> L2-native ARP unification + backend selection + Containerlab correctness matrix (44,45,46) -> publish again (47). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

github-actions · 2026-07-01T10:37:06Z

Synthetic Performance Results (run)

Commit: a1dfd252

✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)

Synthetic UDP Performance Results

Measures framework overhead: sync dpdk_udp::UdpSocket vs async (std::sync::Mutex + try_recv_from).

IPv4 Baseline

Test	Payload	Sync PPS	Async PPS	Ratio (sync/async)	Async ns/op
TX send_to	64B	11.8M	11.2M	1.1x	89
RX recv_from	64B	3.6M	4.3M	0.8x	230
TX send_to	1400B	1.8M	1.8M	1.0x	547
RX recv_from	1400B	1.1M	1.2M	0.9x	851

IPv6

Test	Payload	Sync PPS	Async PPS	Ratio (sync/async)	Async ns/op
TX send_to (IPv6)	64B	9.0M	8.9M	1.0x	112
RX recv_from (IPv6)	64B	4.0M	5.1M	0.8x	197
TX send_to (IPv6)	1400B	3.0M	3.0M	1.0x	328
RX recv_from (IPv6)	1400B	1.2M	1.3M	0.9x	792

IPv6 vs IPv4 Comparison (sync path)

Test	Payload	IPv4 PPS	IPv6 PPS	IPv4/IPv6 Ratio
TX send_to (sync)	64B	11.8M	9.0M	1.31x
RX recv_from (sync)	64B	3.6M	4.0M	0.89x
TX send_to (sync)	1400B	1.8M	3.0M	0.61x
RX recv_from (sync)	1400B	1.1M	1.2M	0.93x

IPv4 avg sync/async ratio: 1.0x, worst: 1.1x | IPv6 vs IPv4 worst ratio: 1.31x (OK)

OK: IPv6 is 31.2% slower than IPv4 — within acceptable threshold (<50%). Expected due to larger headers (40B vs 20B) and mandatory UDP checksum.

Good: Async wrapper is within 1.1x of sync — minimal framework overhead.

github-actions · 2026-07-01T10:37:08Z

Synthetic Performance Results — Graviton (run)

Commit: a1dfd252

✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)

Synthetic UDP Performance Results

Measures framework overhead: sync dpdk_udp::UdpSocket vs async (std::sync::Mutex + try_recv_from).

IPv4 Baseline

Test	Payload	Sync PPS	Async PPS	Ratio (sync/async)	Async ns/op
TX send_to	64B	15.7M	13.8M	1.1x	72
RX recv_from	64B	4.4M	5.4M	0.8x	186
TX send_to	1400B	2.9M	2.9M	1.0x	341
RX recv_from	1400B	1.5M	1.6M	0.9x	610

IPv6

Test	Payload	Sync PPS	Async PPS	Ratio (sync/async)	Async ns/op
TX send_to (IPv6)	64B	11.8M	11.1M	1.1x	89
RX recv_from (IPv6)	64B	4.8M	5.9M	0.8x	170
TX send_to (IPv6)	1400B	4.2M	4.1M	1.0x	244
RX recv_from (IPv6)	1400B	1.4M	1.5M	0.9x	657

IPv6 vs IPv4 Comparison (sync path)

Test	Payload	IPv4 PPS	IPv6 PPS	IPv4/IPv6 Ratio
TX send_to (sync)	64B	15.7M	11.8M	1.33x
RX recv_from (sync)	64B	4.4M	4.8M	0.93x
TX send_to (sync)	1400B	2.9M	4.2M	0.70x
RX recv_from (sync)	1400B	1.5M	1.4M	1.11x

IPv4 avg sync/async ratio: 1.0x, worst: 1.1x | IPv6 vs IPv4 worst ratio: 1.33x (OK)

OK: IPv6 is 32.6% slower than IPv4 — within acceptable threshold (<50%). Expected due to larger headers (40B vs 20B) and mandatory UDP checksum.

Good: Async wrapper is within 1.1x of sync — minimal framework overhead.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

github-actions · 2026-07-01T10:47:14Z

[CI] Stage: Deploy

Infrastructure ready.

Sender: i-0d2db60ba68b24691 (DPDK ENI: 10.0.1.134)
Receiver: i-040a08ceaaa6d438a (DPDK ENI: 10.0.1.49)
Both instances SSM-ready.

github-actions · 2026-07-01T10:48:34Z

[CI] Stage: Deploy

Infrastructure ready.

Sender: i-0eb7b891a75600bd2 (DPDK ENI: 10.0.1.236)
Receiver: i-05b78bda71e224062 (DPDK ENI: 10.0.1.105)
Both instances SSM-ready.

github-actions · 2026-07-01T11:05:10Z

[CI] Stage: Summary

All tests PASSED.

ARP seeding: kernel /proc/net/arp (automatic)

tier1-dpdk-echo: 6 tests, 0 failures
tier2-kernel-interop: 4 tests, 0 failures
tier3-iperf-interop: 1 tests, 0 failures
tier3-iperf-interop: 1 tests, 0 failures

github-actions · 2026-07-01T11:05:15Z

[CI] Stage: Summary

All tests PASSED.

ARP seeding: kernel /proc/net/arp (automatic)

tier1-dpdk-echo: 6 tests, 0 failures
tier2-kernel-interop: 4 tests, 0 failures
tier3-iperf-interop: 1 tests, 0 failures
tier3-iperf-interop: 1 tests, 0 failures

github-actions · 2026-07-01T11:08:11Z

✅ Integration Tests Passed (Run 28511400911)

Branch: 102/merge | Commit: a1dfd252

Test Results

tier1-dpdk-echo.xml: 6 tests, 0 failures, skipped
tier2-kernel-interop.xml: 4 tests, 0 failures, skipped
tier3-iperf-sends.xml: 1 tests, 0 failures, skipped
tier3-our-app-sends.xml: 1 tests, 0 failures, skipped

Application Logs (last 20 lines)

receiver-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.105:9000 (MAC: 02:79:33:34:f6:11)
echo listening on 10.0.1.105:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.236:9000 (MAC: 02:80:f5:5d:1f:89)
echo listening on 10.0.1.236:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-test-client.log

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
DPDK bind failed (Port init failed: Invalid port ID: 0), falling back to tokio
Backend: tokio
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.105:9000: 'arp-probe #1'
Test complete
[2026-07-01T10:52:51Z] INFO: ARP resolution succeeded (got response from peer)
[2026-07-01T10:52:51Z] INFO: Test: udp_send_receive
[2026-07-01T10:52:52Z] INFO: UDP send/receive succeeded
[2026-07-01T10:52:52Z] INFO: Test: echo_roundtrip
[2026-07-01T10:52:53Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-07-01T10:52:53Z] INFO: Test: payload_integrity
[2026-07-01T10:52:53Z] INFO: Response received, checking payload match...
[2026-07-01T10:52:53Z] INFO: Payload integrity verified (found in response)
[2026-07-01T10:52:53Z] INFO: JUnit XML written to: /tmp/test-results/tier2-kernel-interop.xml
[2026-07-01T10:52:53Z] INFO: Tier 2 sender tests complete. Results: /tmp/test-results/tier2-kernel-interop.xml

receiver-test-client-iperf.log

[2026-07-01T11:00:02Z] INFO: iperf-sends: sent 10 packets, received 10 responses
[2026-07-01T11:00:02Z] INFO: iperf-sends: PASS (sent >= 5 packets)
[2026-07-01T11:00:02Z] INFO: JUnit XML written to: /tmp/test-results/tier3-iperf-sends.xml
[2026-07-01T11:00:02Z] INFO: iperf-sends test complete

sender-test-client-iperf.log

Received 30 bytes from 10.0.1.105:9000: 'dpdk-to-kernel-test-payload #3'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #4'
Received 30 bytes from 10.0.1.105:9000: 'dpdk-to-kernel-test-payload #4'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #5'
Received 30 bytes from 10.0.1.105:9000: 'dpdk-to-kernel-test-payload #5'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #6'
Received 30 bytes from 10.0.1.105:9000: 'dpdk-to-kernel-test-payload #6'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #7'
Received 30 bytes from 10.0.1.105:9000: 'dpdk-to-kernel-test-payload #7'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #8'
Received 30 bytes from 10.0.1.105:9000: 'dpdk-to-kernel-test-payload #8'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #9'
Received 30 bytes from 10.0.1.105:9000: 'dpdk-to-kernel-test-payload #9'
Sent 31 bytes: 'dpdk-to-kernel-test-payload #10'
Received 31 bytes from 10.0.1.105:9000: 'dpdk-to-kernel-test-payload #10'
Test complete
[2026-07-01T10:59:02Z] INFO: our-app-sends: sent 10 packets, received 10 responses
[2026-07-01T10:59:02Z] INFO: our-app-sends: PASS (sent >= 5 packets)
[2026-07-01T10:59:02Z] INFO: JUnit XML written to: /tmp/test-results/tier3-our-app-sends.xml
[2026-07-01T10:59:02Z] INFO: our-app-sends test complete

Full Application Logs (last 200 lines each)

receiver-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.105:9000 (MAC: 02:79:33:34:f6:11)
echo listening on 10.0.1.105:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.236:9000 (MAC: 02:80:f5:5d:1f:89)
echo listening on 10.0.1.236:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-test-client.log

[2026-07-01T10:49:51Z] INFO: Test: arp_resolution
UDP Test Client
Target: 10.0.1.105:9000
Bind address: 10.0.1.236:0
Message: 'arp-probe'
Count: 1
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.236:32768 (MAC: 02:80:f5:5d:1f:89)
Backend: dpdk
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.105:9000: 'arp-probe #1'
Test complete
[2026-07-01T10:49:54Z] INFO: ARP resolution succeeded (got response from peer)
[2026-07-01T10:49:54Z] INFO: Test: udp_send_receive
[2026-07-01T10:49:55Z] INFO: UDP send/receive succeeded
[2026-07-01T10:49:55Z] INFO: Test: echo_roundtrip
[2026-07-01T10:49:57Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-07-01T10:49:57Z] INFO: Test: payload_integrity
[2026-07-01T10:49:58Z] INFO: Response received, checking payload match...
[2026-07-01T10:49:58Z] INFO: Payload integrity verified (found in response)
[2026-07-01T10:49:58Z] INFO: Test: jumbo_diagnostics
[2026-07-01T10:49:58Z] INFO: === JUMBO FRAME DIAGNOSTICS ===
[2026-07-01T10:49:58Z] INFO: Interface MTU:
  9001
  65536
[2026-07-01T10:49:58Z] INFO:   ens5: MTU=9001
[2026-07-01T10:49:58Z] INFO:   lo: MTU=65536
[2026-07-01T10:49:58Z] INFO: Routing table MTU column:
Iface	Destination	Gateway 	Flags	RefCnt	Use	Metric	Mask		MTU	Window	IRTT                                                       
ens5	00000000	0101000A	0003	0	0	512	00000000	0	0	0                                                                             
ens5	0200000A	0101000A	0007	0	0	512	FFFFFFFF	0	0	0                                                                             
ens5	0001000A	00000000	0001	0	0	512	00FFFFFF	0	0	0                                                                             
ens5	0101000A	00000000	0005	0	0	512	FFFFFFFF	0	0	0                                                                             
[2026-07-01T10:49:58Z] INFO: DPDK port config (from echo server log):
[2026-07-01T10:49:58Z] INFO:   (no MTU info in echo log)
[2026-07-01T10:49:58Z] INFO: === END JUMBO DIAGNOSTICS ===
[2026-07-01T10:49:58Z] INFO: Test: jumbo_echo_8000
[2026-07-01T10:49:59Z] INFO: Jumbo output: UDP Test Client
Target: 10.0.1.105:9000
Bind address: 10.0.1.236:0
Payload size: 8000 bytes
Count: 3
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.236:32768 (MAC: 02:80:f5:5d:1f:89)
Backend: dpdk
Sending packets...
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.105:9000 (expected 8000, OK)
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.105:9000 (expected 8000, OK)
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.105:9000 (expected 8000, OK)
Test complete
[2026-07-01T10:49:59Z] INFO: Jumbo frame echo succeeded: 3/3 responses with correct size
[2026-07-01T10:49:59Z] INFO: JUnit XML written to: /tmp/test-results/tier1-dpdk-echo.xml
[2026-07-01T10:49:59Z] INFO: Tier 1 sender tests complete. Results: /tmp/test-results/tier1-dpdk-echo.xml
[2026-07-01T10:52:51Z] INFO: Test: arp_resolution
UDP Test Client
Target: 10.0.1.105:9000
Bind address: 0.0.0.0:0
Message: 'arp-probe'
Count: 1
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
DPDK bind failed (Port init failed: Invalid port ID: 0), falling back to tokio
Backend: tokio
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.105:9000: 'arp-probe #1'
Test complete
[2026-07-01T10:52:51Z] INFO: ARP resolution succeeded (got response from peer)
[2026-07-01T10:52:51Z] INFO: Test: udp_send_receive
[2026-07-01T10:52:52Z] INFO: UDP send/receive succeeded
[2026-07-01T10:52:52Z] INFO: Test: echo_roundtrip
[2026-07-01T10:52:53Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-07-01T10:52:53Z] INFO: Test: payload_integrity
[2026-07-01T10:52:53Z] INFO: Response received, checking payload match...
[2026-07-01T10:52:53Z] INFO: Payload integrity verified (found in response)
[2026-07-01T10:52:53Z] INFO: JUnit XML written to: /tmp/test-results/tier2-kernel-interop.xml
[2026-07-01T10:52:53Z] INFO: Tier 2 sender tests complete. Results: /tmp/test-results/tier2-kernel-interop.xml

receiver-test-client-iperf.log

[2026-07-01T11:00:02Z] INFO: iperf-sends: sent 10 packets, received 10 responses
[2026-07-01T11:00:02Z] INFO: iperf-sends: PASS (sent >= 5 packets)
[2026-07-01T11:00:02Z] INFO: JUnit XML written to: /tmp/test-results/tier3-iperf-sends.xml
[2026-07-01T11:00:02Z] INFO: iperf-sends test complete

sender-test-client-iperf.log

[2026-07-01T10:58:59Z] INFO: Pre-flight: checking DPDK state and ARP cache...
[2026-07-01T10:58:59Z] INFO: Local IP: 10.0.1.236, Peer IP: 10.0.1.105, Port: 9000
[2026-07-01T10:58:59Z] INFO: /proc/net/arp contents:
IP address       HW type     Flags       HW address            Mask     Device
10.0.1.105       0x1         0x2         02:79:33:34:f6:11     *        ens5
10.0.1.236       0x1         0x2         02:80:f5:5d:1f:89     *        ens5
10.0.1.1         0x1         0x2         02:c3:19:cb:b9:29     *        ens5
10.0.1.183       0x1         0x2         02:9d:fc:6d:5c:bb     *        ens5
[2026-07-01T10:58:59Z] INFO: DPDK runtime state:
No /var/run/dpdk/ directory
[2026-07-01T10:58:59Z] INFO: vfio-pci bindings:
0000:00:06.0
bind
module
new_id
remove_id
uevent
unbind
[2026-07-01T10:58:59Z] INFO: Test binary: /opt/dpdk-stdlib/target/release/test-client
-rwxr-xr-x. 2 root root 1950584 Jul  1 10:46 /opt/dpdk-stdlib/target/release/test-client
[2026-07-01T10:58:59Z] INFO: Launching test-client: /opt/dpdk-stdlib/target/release/test-client --target 10.0.1.105 --port 9000 --bind-ip 10.0.1.236 --count 10 --delay 200
[2026-07-01T10:59:02Z] INFO: Test client output: UDP Test Client
Target: 10.0.1.105:9000
Bind address: 10.0.1.236:0
Message: 'dpdk-to-kernel-test-payload'
Count: 10
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.236:32768 (MAC: 02:80:f5:5d:1f:89)
Backend: dpdk
Sending packets...
Sent 30 bytes: 'dpdk-to-kernel-test-payload #1'
Received 30 bytes from 10.0.1.105:9000: 'dpdk-to-kernel-test-payload #1'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #2'
Received 30 bytes from 10.0.1.105:9000: 'dpdk-to-kernel-test-payload #2'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #3'
Received 30 bytes from 10.0.1.105:9000: 'dpdk-to-kernel-test-payload #3'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #4'
Received 30 bytes from 10.0.1.105:9000: 'dpdk-to-kernel-test-payload #4'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #5'
Received 30 bytes from 10.0.1.105:9000: 'dpdk-to-kernel-test-payload #5'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #6'
Received 30 bytes from 10.0.1.105:9000: 'dpdk-to-kernel-test-payload #6'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #7'
Received 30 bytes from 10.0.1.105:9000: 'dpdk-to-kernel-test-payload #7'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #8'
Received 30 bytes from 10.0.1.105:9000: 'dpdk-to-kernel-test-payload #8'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #9'
Received 30 bytes from 10.0.1.105:9000: 'dpdk-to-kernel-test-payload #9'
Sent 31 bytes: 'dpdk-to-kernel-test-payload #10'
Received 31 bytes from 10.0.1.105:9000: 'dpdk-to-kernel-test-payload #10'
Test complete
[2026-07-01T10:59:02Z] INFO: our-app-sends: sent 10 packets, received 10 responses
[2026-07-01T10:59:02Z] INFO: our-app-sends: PASS (sent >= 5 packets)
[2026-07-01T10:59:02Z] INFO: JUnit XML written to: /tmp/test-results/tier3-our-app-sends.xml
[2026-07-01T10:59:02Z] INFO: our-app-sends test complete

⚠️ SSM Command Failures (receiver-ssm-failure.log)

=== Polling timeout after 30s ===
Status: InProgress
Instance: i-05b78bda71e224062 (receiver)
Command ID: 9156832b-17d8-47db-941e-f0b243e52d11

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-05b78bda71e224062 (receiver)
Command ID: 0095c748-b385-498a-9e48-4e7ef694b89f

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-05b78bda71e224062 (receiver)
Command ID: c0a74144-97a5-4a87-aadb-ac623e0c633c

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-05b78bda71e224062 (receiver)
Command ID: 9df94b4a-bac3-4492-96bc-7d58dff4783f

=== STDOUT ===


=== STDERR ===

⚠️ SSM Command Failures (sender-ssm-failure.log)

=== Polling timeout after 30s ===
Status: InProgress
Instance: i-0eb7b891a75600bd2 (sender)
Command ID: 40f0eecb-e0a8-4fdf-a75d-96e707ee7942

=== STDOUT ===


=== STDERR ===

Network & PCI State

receiver-network-interfaces.log

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:ff:ea:d4:4b:ed brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-005ae4b0c09e62302
    altname device-number-0.0
    inet 10.0.1.11/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 2119sec preferred_lft 2119sec
    inet6 fe80::ff:eaff:fed4:4bed/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

sender-network-interfaces.log

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:24:78:ff:8e:bb brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-0e5be7840134f1435
    altname device-number-0.0
    inet 10.0.1.56/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 2141sec preferred_lft 2141sec
    inet6 fe80::24:78ff:feff:8ebb/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

receiver-networking-diag-baseline.txt

=== NETWORKING DIAGNOSTICS ===
timestamp: 2026-07-01T10:49:12Z
hostname: ip-10-0-1-11.ec2.internal
kernel: 6.18.35-68.129.amzn2023.x86_64

=== DPDK PORT STATUS ===

Network devices using DPDK-compatible driver
============================================
0000:00:06.0 'Elastic Network Adapter (ENA) ec20' drv=vfio-pci unused=ena

Network devices using kernel driver
===================================
0000:00:05.0 'Elastic Network Adapter (ENA) ec20' if=ens5 drv=ena unused=vfio-pci *Active*

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'DMA' devices detected
=========================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

No 'Regex' devices detected
===========================

=== IP ADDRESSES ===
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:ff:ea:d4:4b:ed brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-005ae4b0c09e62302
    altname device-number-0.0
    inet 10.0.1.11/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 3067sec preferred_lft 3067sec
    inet6 fe80::ff:eaff:fed4:4bed/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

=== ARP TABLE ===
10.0.1.218 dev ens5 lladdr 02:2b:29:27:95:f7 STALE 
10.0.1.1 dev ens5 lladdr 02:c3:19:cb:b9:29 REACHABLE 
10.0.1.183 dev ens5 lladdr 02:9d:fc:6d:5c:bb REACHABLE 

=== ROUTE TABLE ===
default via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.11 metric 512 
10.0.0.2 via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.11 metric 512 
10.0.1.0/24 dev ens5 proto kernel scope link src 10.0.1.11 metric 512 
10.0.1.1 dev ens5 proto dhcp scope link src 10.0.1.11 metric 512 

=== IMDS: ENI INFORMATION ===
ENI MACs found: 02:79:33:34:f6:11/ 02:ff:ea:d4:4b:ed/ 

--- ENI: 02:79:33:34:f6:11/ ---
  device-number: 1
  local-ipv4s: 10.0.1.105
  subnet-id: subnet-09594936f268f8322
  subnet-cidr: 10.0.1.0/24

--- ENI: 02:ff:ea:d4:4b:ed/ ---
  device-number: 0
  local-ipv4s: 10.0.1.11
  subnet-id: subnet-09594936f268f8322
  subnet-cidr: 10.0.1.0/24


=== GATEWAY ARP TEST ===
Gateway IP: 10.0.1.1
Gateway ARP entry:
10.0.1.1 dev ens5 lladdr 02:c3:19:cb:b9:29 REACHABLE 

arping result:
ARPING 10.0.1.1 from 10.0.1.11 ens5
Unicast reply from 10.0.1.1 [02:C3:19:CB:B9:29]  0.534ms
Sent 1 probes (1 broadcast(s))
Received 1 response(s)

=== HUGEPAGE STATUS ===
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:     51200 kB
HugePages_Total:    1024
HugePages_Free:     1024
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         2097152 kB

=== VFIO STATUS ===
total 0
drwxr-xr-x.  2 root root       80 Jul  1 10:48 .
drwxr-xr-x. 14 root root     3100 Jul  1 10:48 ..
crw-------.  1 root root 243,   0 Jul  1 10:48 noiommu-0
crw-rw-rw-.  1 root root  10, 196 Jul  1 10:40 vfio

noiommu mode:
Y

=== DPDK SHARED MEMORY ===
no /var/run/dpdk/ directory (clean state)

=== DPDK-RELATED DMESG (last 30 lines) ===
[    0.055023] printk: legacy console [ttyS0] enabled
[    0.056068] x2apic enabled
[    0.060358] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto
[    0.060469] x86/fpu: Enabled xstate features 0x2ff, context size is 2568 bytes, using 'compacted' format.
[    0.070200] audit: type=2000 audit(1782902408.391:1): state=initialized audit_enabled=0 res=1
[    0.089680] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
[    0.149989] ACPI: Interpreter enabled
[    0.150021] ACPI: Enabled 2 GPEs in block 00 to 0F
[    0.158543] pci 0000:00:05.0: enabling Extended Tags
[    0.254277] SGI XFS with ACLs, security attributes, quota, no debug enabled
[    0.259952] ACPI: \_SB_.LNKD: Enabled at IRQ 11
[    0.355081] IPI shorthand broadcast: enabled
[    7.246877] systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
[    7.279200] systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
[    7.351936] VFIO - User Level meta-driver version: 0.3
[    8.074395] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.17.1g
[    8.090647] ena 0000:00:05.0: ENA device version: 0.10
[    8.091321] ena 0000:00:05.0: ENA controller version: 0.0.1 implementation version 1
[    8.194803] ena 0000:00:05.0: ENA Large LLQ is disabled
[    8.206912] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem c0500000, mac addr 02:ff:ea:d4:4b:ed
[    8.256957] ena 0000:00:05.0 ens5: renamed from eth0
[  491.457705] pci 0000:00:06.0: enabling Extended Tags
[  491.461615] ena 0000:00:06.0: enabling device (0000 -> 0002)
[  491.473495] ena 0000:00:06.0: ENA device version: 0.10
[  491.474260] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  491.573479] ena 0000:00:06.0: ENA Large LLQ is disabled
[  491.587440] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:79:33:34:f6:11
[  491.593001] ena 0000:00:06.0 ens6: renamed from eth0
[  530.450086] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  530.451546] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device

=== DPDK-RELATED PROCESSES ===
no DPDK processes running

=== END DIAGNOSTICS ===

sender-networking-diag-baseline.txt

=== NETWORKING DIAGNOSTICS ===
timestamp: 2026-07-01T10:49:04Z
hostname: ip-10-0-1-56.ec2.internal
kernel: 6.18.35-68.129.amzn2023.x86_64

=== DPDK PORT STATUS ===

Network devices using DPDK-compatible driver
============================================
0000:00:06.0 'Elastic Network Adapter (ENA) ec20' drv=vfio-pci unused=ena

Network devices using kernel driver
===================================
0000:00:05.0 'Elastic Network Adapter (ENA) ec20' if=ens5 drv=ena unused=vfio-pci *Active*

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'DMA' devices detected
=========================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

No 'Regex' devices detected
===========================

=== IP ADDRESSES ===
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:24:78:ff:8e:bb brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-0e5be7840134f1435
    altname device-number-0.0
    inet 10.0.1.56/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 3074sec preferred_lft 3074sec
    inet6 fe80::24:78ff:feff:8ebb/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

=== ARP TABLE ===
10.0.1.1 dev ens5 lladdr 02:c3:19:cb:b9:29 REACHABLE 
10.0.1.183 dev ens5 lladdr 02:9d:fc:6d:5c:bb REACHABLE 

=== ROUTE TABLE ===
default via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.56 metric 512 
10.0.0.2 via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.56 metric 512 
10.0.1.0/24 dev ens5 proto kernel scope link src 10.0.1.56 metric 512 
10.0.1.1 dev ens5 proto dhcp scope link src 10.0.1.56 metric 512 

=== IMDS: ENI INFORMATION ===
ENI MACs found: 02:24:78:ff:8e:bb/ 02:80:f5:5d:1f:89/ 

--- ENI: 02:24:78:ff:8e:bb/ ---
  device-number: 0
  local-ipv4s: 10.0.1.56
  subnet-id: subnet-09594936f268f8322
  subnet-cidr: 10.0.1.0/24

--- ENI: 02:80:f5:5d:1f:89/ ---
  device-number: 1
  local-ipv4s: 10.0.1.236
  subnet-id: subnet-09594936f268f8322
  subnet-cidr: 10.0.1.0/24


=== GATEWAY ARP TEST ===
Gateway IP: 10.0.1.1
Gateway ARP entry:
10.0.1.1 dev ens5 lladdr 02:c3:19:cb:b9:29 REACHABLE 

arping result:
ARPING 10.0.1.1 from 10.0.1.56 ens5
Unicast reply from 10.0.1.1 [02:C3:19:CB:B9:29]  0.537ms
Sent 1 probes (1 broadcast(s))
Received 1 response(s)

=== HUGEPAGE STATUS ===
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:     51200 kB
HugePages_Total:    1024
HugePages_Free:     1024
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         2097152 kB

=== VFIO STATUS ===
total 0
drwxr-xr-x.  2 root root       80 Jul  1 10:48 .
drwxr-xr-x. 14 root root     3100 Jul  1 10:48 ..
crw-------.  1 root root 243,   0 Jul  1 10:48 noiommu-0
crw-rw-rw-.  1 root root  10, 196 Jul  1 10:40 vfio

noiommu mode:
Y

=== DPDK SHARED MEMORY ===
no /var/run/dpdk/ directory (clean state)

=== DPDK-RELATED DMESG (last 30 lines) ===
[    0.052575] printk: legacy console [ttyS0] enabled
[    0.053585] x2apic enabled
[    0.057709] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto
[    0.057817] x86/fpu: Enabled xstate features 0x2ff, context size is 2568 bytes, using 'compacted' format.
[    0.067567] audit: type=2000 audit(1782902409.655:1): state=initialized audit_enabled=0 res=1
[    0.087071] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
[    0.147562] ACPI: Interpreter enabled
[    0.147562] ACPI: Enabled 2 GPEs in block 00 to 0F
[    0.155999] pci 0000:00:05.0: enabling Extended Tags
[    0.239000] SGI XFS with ACLs, security attributes, quota, no debug enabled
[    0.256580] ACPI: \_SB_.LNKD: Enabled at IRQ 11
[    0.364602] IPI shorthand broadcast: enabled
[    6.087198] systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
[    6.117692] systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
[    6.263050] VFIO - User Level meta-driver version: 0.3
[    6.803986] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.17.1g
[    6.816419] ena 0000:00:05.0: ENA device version: 0.10
[    6.817106] ena 0000:00:05.0: ENA controller version: 0.0.1 implementation version 1
[    6.924561] ena 0000:00:05.0: ENA Large LLQ is disabled
[    6.936587] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem c0500000, mac addr 02:24:78:ff:8e:bb
[    6.984999] ena 0000:00:05.0 ens5: renamed from eth0
[  470.818542] pci 0000:00:06.0: enabling Extended Tags
[  470.822480] ena 0000:00:06.0: enabling device (0000 -> 0002)
[  470.836729] ena 0000:00:06.0: ENA device version: 0.10
[  470.837480] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  470.937540] ena 0000:00:06.0: ENA Large LLQ is disabled
[  470.950461] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:80:f5:5d:1f:89
[  470.957747] ena 0000:00:06.0 ens6: renamed from eth0
[  522.364100] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  522.365539] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device

=== DPDK-RELATED PROCESSES ===
no DPDK processes running

=== END DIAGNOSTICS ===

⚠️ Crash Diagnostics

receiver-dmesg-crashes.log

[    0.069585] pid_max: default: 32768 minimum: 301
[    0.162367] iommu: Default domain type: Translated
[    0.168192] NetLabel:  unlabeled traffic allowed by default
[    0.189951] PCI: CLS 0 bytes, default 64
[    0.281298] nvme nvme0: 2/0/0 default/read/poll queues
[    0.429709] systemd[1]: systemd 252.23-12.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    0.429835] systemd[1]: No hostname configured, using default hostname.
[    0.508621] systemd[1]: Queued start job for default target initrd.target.
[    5.341341] systemd[1]: systemd 252.23-12.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[  567.516496] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:8338)
[  747.336133] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:8771)
[ 1351.584576] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:9993)

sender-dmesg-crashes.log

[    0.066970] pid_max: default: 32768 minimum: 301
[    0.159797] iommu: Default domain type: Translated
[    0.167229] NetLabel:  unlabeled traffic allowed by default
[    0.186524] PCI: CLS 0 bytes, default 64
[    0.280825] nvme nvme0: 2/0/0 default/read/poll queues
[    0.449056] systemd[1]: systemd 252.23-12.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    0.449178] systemd[1]: No hostname configured, using default hostname.
[    0.509052] systemd[1]: Queued start job for default target initrd.target.
[    4.128852] systemd[1]: systemd 252.23-12.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[ 1175.077550] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:9585)

Kernel Console (dmesg)

receiver-console-output.log (PCI/driver events only)

[  567.514953] vfio-pci 0000:00:06.0: reset done
[  567.516496] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:8338)
[  567.517877] vfio-pci 0000:00:06.0: resetting
[  567.734829] vfio-pci 0000:00:06.0: reset done
[  648.183735] vfio-pci 0000:00:06.0: Removing from iommu group 0
[  649.193129] ena 0000:00:06.0: ENA device version: 0.10
[  649.193861] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  649.293106] ena 0000:00:06.0: ENA Large LLQ is disabled
[  649.305427] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:79:33:34:f6:11
[  649.316168] ena 0000:00:06.0 ens6: renamed from eth0 (while UP)
[  669.086828] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  669.088202] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[  747.119161] vfio-pci 0000:00:06.0: resetting
[  747.334553] vfio-pci 0000:00:06.0: reset done
[  747.336133] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:8771)
[  747.337546] vfio-pci 0000:00:06.0: resetting
[  747.554420] vfio-pci 0000:00:06.0: reset done
[  822.668042] vfio-pci 0000:00:06.0: Removing from iommu group 0
[  823.680958] ena 0000:00:06.0: ENA device version: 0.10
[  823.681731] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  823.782634] ena 0000:00:06.0: ENA Large LLQ is disabled
[  823.794723] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:79:33:34:f6:11
[  823.802070] ena 0000:00:06.0 ens6: renamed from eth0
[ 1277.005262] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1277.006669] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1351.356598] vfio-pci 0000:00:06.0: resetting
[ 1351.582965] vfio-pci 0000:00:06.0: reset done
[ 1351.584576] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:9993)
[ 1351.586030] vfio-pci 0000:00:06.0: resetting
[ 1351.802823] vfio-pci 0000:00:06.0: reset done

sender-console-output.log (PCI/driver events only)

[ 1175.078985] vfio-pci 0000:00:06.0: resetting
[ 1175.295846] vfio-pci 0000:00:06.0: reset done
[ 1237.987866] vfio-pci 0000:00:06.0: Removing from iommu group 0
[ 1239.000696] ena 0000:00:06.0: ENA device version: 0.10
[ 1239.001461] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[ 1239.102413] ena 0000:00:06.0: ENA Large LLQ is disabled
[ 1239.116131] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:80:f5:5d:1f:89
[ 1239.123365] ena 0000:00:06.0 ens6: renamed from eth0
[ 1268.839725] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1268.840987] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1366.047475] vfio-pci 0000:00:06.0: resetting
[ 1366.264405] vfio-pci 0000:00:06.0: reset done
[ 1366.265932] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:10028)
[ 1366.267465] vfio-pci 0000:00:06.0: resetting
[ 1366.484279] vfio-pci 0000:00:06.0: reset done
[ 1366.771918] vfio-pci 0000:00:06.0: resetting
[ 1366.994291] vfio-pci 0000:00:06.0: reset done
[ 1366.995840] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:10048)
[ 1366.997385] vfio-pci 0000:00:06.0: resetting
[ 1367.214233] vfio-pci 0000:00:06.0: reset done
[ 1368.503745] vfio-pci 0000:00:06.0: resetting
[ 1368.734297] vfio-pci 0000:00:06.0: reset done
[ 1368.735826] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:10071)
[ 1368.737420] vfio-pci 0000:00:06.0: resetting
[ 1368.954229] vfio-pci 0000:00:06.0: reset done
[ 1370.243707] vfio-pci 0000:00:06.0: resetting
[ 1370.464278] vfio-pci 0000:00:06.0: reset done
[ 1370.465841] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:10094)
[ 1370.467415] vfio-pci 0000:00:06.0: resetting
[ 1370.684232] vfio-pci 0000:00:06.0: reset done

github-actions · 2026-07-01T11:08:21Z

✅ Integration Tests Passed — Graviton (run)

Branch: 102/merge | Commit: a1dfd252

Test Results

tier1-dpdk-echo.xml: 6 tests, 0 failures
tier2-kernel-interop.xml: 4 tests, 0 failures
tier3-iperf-sends.xml: 1 tests, 0 failures
tier3-our-app-sends.xml: 1 tests, 0 failures

Application Logs (last 20 lines)

receiver-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.49:9000 (MAC: 02:b7:a1:d7:6a:01)
echo listening on 10.0.1.49:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.134:9000 (MAC: 02:e2:b5:cd:2b:7d)
echo listening on 10.0.1.134:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-test-client.log

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
DPDK bind failed (Port init failed: Invalid port ID: 0), falling back to tokio
Backend: tokio
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.49:9000: 'arp-probe #1'
Test complete
[2026-07-01T10:52:59Z] INFO: ARP resolution succeeded (got response from peer)
[2026-07-01T10:52:59Z] INFO: Test: udp_send_receive
[2026-07-01T10:53:00Z] INFO: UDP send/receive succeeded
[2026-07-01T10:53:00Z] INFO: Test: echo_roundtrip
[2026-07-01T10:53:00Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-07-01T10:53:00Z] INFO: Test: payload_integrity
[2026-07-01T10:53:00Z] INFO: Response received, checking payload match...
[2026-07-01T10:53:00Z] INFO: Payload integrity verified (found in response)
[2026-07-01T10:53:00Z] INFO: JUnit XML written to: /tmp/test-results/tier2-kernel-interop.xml
[2026-07-01T10:53:01Z] INFO: Tier 2 sender tests complete. Results: /tmp/test-results/tier2-kernel-interop.xml

docs: testing-hardening notes list + roadmap trim

5415a9b

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

gspivey closed this Jul 1, 2026

gspivey deleted the agent/testing-backlog branch July 1, 2026 11:10

Conversation

gspivey commented Jul 1, 2026

Uh oh!

github-actions Bot commented Jul 1, 2026

Synthetic Performance Results (run)

Synthetic UDP Performance Results

IPv4 Baseline

IPv6

IPv6 vs IPv4 Comparison (sync path)

Uh oh!

github-actions Bot commented Jul 1, 2026

Synthetic Performance Results — Graviton (run)

Synthetic UDP Performance Results

IPv4 Baseline

IPv6

IPv6 vs IPv4 Comparison (sync path)

Uh oh!

github-actions Bot commented Jul 1, 2026

[CI] Stage: Deploy

Uh oh!

github-actions Bot commented Jul 1, 2026

[CI] Stage: Deploy

Uh oh!

github-actions Bot commented Jul 1, 2026

[CI] Stage: Summary

Uh oh!

github-actions Bot commented Jul 1, 2026

[CI] Stage: Summary

Uh oh!

github-actions Bot commented Jul 1, 2026

✅ Integration Tests Passed (Run 28511400911)

Test Results

Application Logs (last 20 lines)

receiver-echo-server.log

sender-echo-server.log

sender-test-client.log

receiver-test-client-iperf.log

sender-test-client-iperf.log

receiver-network-interfaces.log

sender-network-interfaces.log

receiver-networking-diag-baseline.txt

sender-networking-diag-baseline.txt

receiver-dmesg-crashes.log

sender-dmesg-crashes.log

receiver-console-output.log (PCI/driver events only)

sender-console-output.log (PCI/driver events only)

Uh oh!

github-actions Bot commented Jul 1, 2026

✅ Integration Tests Passed — Graviton (run)

Test Results

Application Logs (last 20 lines)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant