fix(perf): TRex ASTF needs --lro-disable on AWS ENA by gspivey · Pull Request #104 · gspivey/dpdk-stdlib-rust

gspivey · 2026-07-01T12:14:08Z

Follow-up to #103. That fix was validated by run 28514788453: TRex now runs in ASTF mode, all DUTs start, and tcp-echo binds on the real NIC. It exposed the next blocker:

Requested RX offload TCP_LRO is not supported   <- TRex dies here
... TRexError: *** [RPC] - Failed to get server response from tcp://localhost:4501

TRex ASTF requests hardware TCP_LRO; AWS ENA doesn't support it, so t-rex-64 -i --astf dies at startup and the ASTF RPC never comes up. (STL/UDP never hit this — no LRO requested, which is why UDP perf always worked.)

Fix

Extract trex_launch_cmd(mode) and pass --lro-disable (harmless no-op for STL).
Readiness check: drop LOG_GROWING — a growing log doesn't mean TRex is alive, and it masked the dead server for the whole 30-min run. Require PROCESS_FOUND/API_PORT (:4501).
Regression test asserts the ASTF launch includes --astf + --lro-disable (catches this locally in <1s).

Scripts-only, bash -n clean, harness tests green. Merge → re-dispatch the TCP perf suite to get real numbers.

🤖 Generated with Claude Code

…LRO) Run 28514788453 (post-#103) proved the harness fix works — TRex now starts in ASTF mode, all DUTs start, tcp-echo binds on the real NIC — but exposed a deeper blocker: TRex ASTF requests hardware TCP_LRO, which AWS ENA does not support, so t-rex-64 dies at startup ('Requested RX offload TCP_LRO is not supported') and the ASTF RPC (:4501) never comes up -> clients get 'Failed to get server response'. UDP/STL never hit this (no LRO requested). - Extract trex_launch_cmd(mode) and pass --lro-disable (harmless for STL). - Readiness check: drop LOG_GROWING as a pass signal — a growing log does not mean TRex is alive; it masked the dead server for the whole run. Require PROCESS_FOUND or API_PORT (:4501). - Regression test asserts ASTF launch includes --astf + --lro-disable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

github-actions · 2026-07-01T12:15:18Z

Synthetic Performance Results (run)

Commit: 1360b0a9

✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)

Synthetic UDP Performance Results

Measures framework overhead: sync dpdk_udp::UdpSocket vs async (std::sync::Mutex + try_recv_from).

IPv4 Baseline

Test	Payload	Sync PPS	Async PPS	Ratio (sync/async)	Async ns/op
TX send_to	64B	11.8M	11.2M	1.0x	89
RX recv_from	64B	3.6M	4.6M	0.8x	217
TX send_to	1400B	1.9M	1.9M	1.0x	539
RX recv_from	1400B	1.1M	1.2M	0.9x	837

IPv6

Test	Payload	Sync PPS	Async PPS	Ratio (sync/async)	Async ns/op
TX send_to (IPv6)	64B	9.1M	8.9M	1.0x	112
RX recv_from (IPv6)	64B	4.0M	5.4M	0.7x	183
TX send_to (IPv6)	1400B	3.1M	3.1M	1.0x	327
RX recv_from (IPv6)	1400B	1.2M	1.3M	0.9x	777

IPv6 vs IPv4 Comparison (sync path)

Test	Payload	IPv4 PPS	IPv6 PPS	IPv4/IPv6 Ratio
TX send_to (sync)	64B	11.8M	9.1M	1.29x
RX recv_from (sync)	64B	3.6M	4.0M	0.88x
TX send_to (sync)	1400B	1.9M	3.1M	0.61x
RX recv_from (sync)	1400B	1.1M	1.2M	0.94x

IPv4 avg sync/async ratio: 0.9x, worst: 1.0x | IPv6 vs IPv4 worst ratio: 1.29x (OK)

OK: IPv6 is 29.5% slower than IPv4 — within acceptable threshold (<50%). Expected due to larger headers (40B vs 20B) and mandatory UDP checksum.

Good: Async wrapper is within 1.0x of sync — minimal framework overhead.

github-actions · 2026-07-01T12:15:20Z

Synthetic Performance Results — Graviton (run)

Commit: 1360b0a9

✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)

Synthetic UDP Performance Results

Measures framework overhead: sync dpdk_udp::UdpSocket vs async (std::sync::Mutex + try_recv_from).

IPv4 Baseline

Test	Payload	Sync PPS	Async PPS	Ratio (sync/async)	Async ns/op
TX send_to	64B	11.7M	11.2M	1.0x	89
RX recv_from	64B	3.6M	4.7M	0.8x	210
TX send_to	1400B	1.8M	1.8M	1.0x	541
RX recv_from	1400B	1.1M	1.2M	0.9x	802

IPv6

Test	Payload	Sync PPS	Async PPS	Ratio (sync/async)	Async ns/op
TX send_to (IPv6)	64B	9.1M	8.3M	1.1x	120
RX recv_from (IPv6)	64B	4.1M	5.5M	0.7x	182
TX send_to (IPv6)	1400B	3.1M	3.0M	1.0x	337
RX recv_from (IPv6)	1400B	1.2M	1.3M	0.9x	761

IPv6 vs IPv4 Comparison (sync path)

Test	Payload	IPv4 PPS	IPv6 PPS	IPv4/IPv6 Ratio
TX send_to (sync)	64B	11.7M	9.1M	1.29x
RX recv_from (sync)	64B	3.6M	4.1M	0.89x
TX send_to (sync)	1400B	1.8M	3.1M	0.60x
RX recv_from (sync)	1400B	1.1M	1.2M	0.95x

IPv4 avg sync/async ratio: 0.9x, worst: 1.0x | IPv6 vs IPv4 worst ratio: 1.29x (OK)

OK: IPv6 is 28.8% slower than IPv4 — within acceptable threshold (<50%). Expected due to larger headers (40B vs 20B) and mandatory UDP checksum.

Good: Async wrapper is within 1.0x of sync — minimal framework overhead.

github-actions · 2026-07-01T12:23:34Z

[CI] Stage: Deploy

Infrastructure ready.

Sender: i-0e33173a67b517e1b (DPDK ENI: 10.0.1.228)
Receiver: i-01b65745a6a180e18 (DPDK ENI: 10.0.1.104)
Both instances SSM-ready.

github-actions · 2026-07-01T12:25:14Z

[CI] Stage: Deploy

Infrastructure ready.

Sender: i-04c5de817f89d555e (DPDK ENI: 10.0.1.208)
Receiver: i-0b9ee3dcc9ae2fa0f (DPDK ENI: 10.0.1.81)
Both instances SSM-ready.

github-actions · 2026-07-01T12:40:17Z

[CI] Stage: Summary

All tests PASSED.

ARP seeding: kernel /proc/net/arp (automatic)

tier1-dpdk-echo: 6 tests, 0 failures
tier2-kernel-interop: 4 tests, 0 failures
tier3-iperf-interop: 1 tests, 0 failures
tier3-iperf-interop: 1 tests, 0 failures

github-actions · 2026-07-01T12:41:34Z

[CI] Stage: Summary

All tests PASSED.

ARP seeding: kernel /proc/net/arp (automatic)

tier1-dpdk-echo: 6 tests, 0 failures
tier2-kernel-interop: 4 tests, 0 failures
tier3-iperf-interop: 1 tests, 0 failures
tier3-iperf-interop: 1 tests, 0 failures

github-actions · 2026-07-01T12:42:38Z

✅ Integration Tests Passed — Graviton (run)

Branch: 104/merge | Commit: 1360b0a9

Test Results

tier1-dpdk-echo.xml: 6 tests, 0 failures
tier2-kernel-interop.xml: 4 tests, 0 failures
tier3-iperf-sends.xml: 1 tests, 0 failures
tier3-our-app-sends.xml: 1 tests, 0 failures

Application Logs (last 20 lines)

receiver-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.104:9000 (MAC: 02:6d:fe:f8:53:e5)
echo listening on 10.0.1.104:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.228:9000 (MAC: 02:b3:35:a8:7e:bb)
echo listening on 10.0.1.228:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-test-client.log

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
DPDK bind failed (Port init failed: Invalid port ID: 0), falling back to tokio
Backend: tokio
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.104:9000: 'arp-probe #1'
Test complete
[2026-07-01T12:27:53Z] INFO: ARP resolution succeeded (got response from peer)
[2026-07-01T12:27:53Z] INFO: Test: udp_send_receive
[2026-07-01T12:27:54Z] INFO: UDP send/receive succeeded
[2026-07-01T12:27:54Z] INFO: Test: echo_roundtrip
[2026-07-01T12:27:55Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-07-01T12:27:55Z] INFO: Test: payload_integrity
[2026-07-01T12:27:55Z] INFO: Response received, checking payload match...
[2026-07-01T12:27:55Z] INFO: Payload integrity verified (found in response)
[2026-07-01T12:27:55Z] INFO: JUnit XML written to: /tmp/test-results/tier2-kernel-interop.xml
[2026-07-01T12:27:55Z] INFO: Tier 2 sender tests complete. Results: /tmp/test-results/tier2-kernel-interop.xml

github-actions · 2026-07-01T12:44:36Z

✅ Integration Tests Passed (Run 28516698984)

Branch: 104/merge | Commit: 1360b0a9

Test Results

tier1-dpdk-echo.xml: 6 tests, 0 failures, skipped
tier2-kernel-interop.xml: 4 tests, 0 failures, skipped
tier3-iperf-sends.xml: 1 tests, 0 failures, skipped
tier3-our-app-sends.xml: 1 tests, 0 failures, skipped

Application Logs (last 20 lines)

receiver-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.81:9000 (MAC: 02:74:58:97:55:83)
echo listening on 10.0.1.81:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.208:9000 (MAC: 02:71:3e:0c:2e:49)
echo listening on 10.0.1.208:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-test-client.log

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
DPDK bind failed (Port init failed: Invalid port ID: 0), falling back to tokio
Backend: tokio
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.81:9000: 'arp-probe #1'
Test complete
[2026-07-01T12:30:52Z] INFO: ARP resolution succeeded (got response from peer)
[2026-07-01T12:30:52Z] INFO: Test: udp_send_receive
[2026-07-01T12:30:54Z] INFO: UDP send/receive succeeded
[2026-07-01T12:30:54Z] INFO: Test: echo_roundtrip
[2026-07-01T12:30:54Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-07-01T12:30:54Z] INFO: Test: payload_integrity
[2026-07-01T12:30:54Z] INFO: Response received, checking payload match...
[2026-07-01T12:30:54Z] INFO: Payload integrity verified (found in response)
[2026-07-01T12:30:54Z] INFO: JUnit XML written to: /tmp/test-results/tier2-kernel-interop.xml
[2026-07-01T12:30:54Z] INFO: Tier 2 sender tests complete. Results: /tmp/test-results/tier2-kernel-interop.xml

receiver-test-client-iperf.log

[2026-07-01T12:36:15Z] INFO: iperf-sends: sent 10 packets, received 10 responses
[2026-07-01T12:36:15Z] INFO: iperf-sends: PASS (sent >= 5 packets)
[2026-07-01T12:36:15Z] INFO: JUnit XML written to: /tmp/test-results/tier3-iperf-sends.xml
[2026-07-01T12:36:15Z] INFO: iperf-sends test complete

sender-test-client-iperf.log

Received 30 bytes from 10.0.1.81:9000: 'dpdk-to-kernel-test-payload #3'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #4'
Received 30 bytes from 10.0.1.81:9000: 'dpdk-to-kernel-test-payload #4'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #5'
Received 30 bytes from 10.0.1.81:9000: 'dpdk-to-kernel-test-payload #5'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #6'
Received 30 bytes from 10.0.1.81:9000: 'dpdk-to-kernel-test-payload #6'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #7'
Received 30 bytes from 10.0.1.81:9000: 'dpdk-to-kernel-test-payload #7'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #8'
Received 30 bytes from 10.0.1.81:9000: 'dpdk-to-kernel-test-payload #8'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #9'
Received 30 bytes from 10.0.1.81:9000: 'dpdk-to-kernel-test-payload #9'
Sent 31 bytes: 'dpdk-to-kernel-test-payload #10'
Received 31 bytes from 10.0.1.81:9000: 'dpdk-to-kernel-test-payload #10'
Test complete
[2026-07-01T12:35:18Z] INFO: our-app-sends: sent 10 packets, received 10 responses
[2026-07-01T12:35:18Z] INFO: our-app-sends: PASS (sent >= 5 packets)
[2026-07-01T12:35:18Z] INFO: JUnit XML written to: /tmp/test-results/tier3-our-app-sends.xml
[2026-07-01T12:35:18Z] INFO: our-app-sends test complete

Full Application Logs (last 200 lines each)

receiver-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.81:9000 (MAC: 02:74:58:97:55:83)
echo listening on 10.0.1.81:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.208:9000 (MAC: 02:71:3e:0c:2e:49)
echo listening on 10.0.1.208:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-test-client.log

[2026-07-01T12:26:28Z] INFO: Test: arp_resolution
UDP Test Client
Target: 10.0.1.81:9000
Bind address: 10.0.1.208:0
Message: 'arp-probe'
Count: 1
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.208:32768 (MAC: 02:71:3e:0c:2e:49)
Backend: dpdk
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.81:9000: 'arp-probe #1'
Test complete
[2026-07-01T12:26:28Z] INFO: ARP resolution succeeded (got response from peer)
[2026-07-01T12:26:28Z] INFO: Test: udp_send_receive
[2026-07-01T12:26:30Z] INFO: UDP send/receive succeeded
[2026-07-01T12:26:30Z] INFO: Test: echo_roundtrip
[2026-07-01T12:26:32Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-07-01T12:26:32Z] INFO: Test: payload_integrity
[2026-07-01T12:26:32Z] INFO: Response received, checking payload match...
[2026-07-01T12:26:32Z] INFO: Payload integrity verified (found in response)
[2026-07-01T12:26:32Z] INFO: Test: jumbo_diagnostics
[2026-07-01T12:26:32Z] INFO: === JUMBO FRAME DIAGNOSTICS ===
[2026-07-01T12:26:32Z] INFO: Interface MTU:
  9001
  65536
[2026-07-01T12:26:32Z] INFO:   ens5: MTU=9001
[2026-07-01T12:26:32Z] INFO:   lo: MTU=65536
[2026-07-01T12:26:32Z] INFO: Routing table MTU column:
Iface	Destination	Gateway 	Flags	RefCnt	Use	Metric	Mask		MTU	Window	IRTT                                                       
ens5	00000000	0101000A	0003	0	0	512	00000000	0	0	0                                                                             
ens5	0200000A	0101000A	0007	0	0	512	FFFFFFFF	0	0	0                                                                             
ens5	0001000A	00000000	0001	0	0	512	00FFFFFF	0	0	0                                                                             
ens5	0101000A	00000000	0005	0	0	512	FFFFFFFF	0	0	0                                                                             
[2026-07-01T12:26:32Z] INFO: DPDK port config (from echo server log):
[2026-07-01T12:26:32Z] INFO:   (no MTU info in echo log)
[2026-07-01T12:26:32Z] INFO: === END JUMBO DIAGNOSTICS ===
[2026-07-01T12:26:32Z] INFO: Test: jumbo_echo_8000
[2026-07-01T12:26:34Z] INFO: Jumbo output: UDP Test Client
Target: 10.0.1.81:9000
Bind address: 10.0.1.208:0
Payload size: 8000 bytes
Count: 3
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.208:32768 (MAC: 02:71:3e:0c:2e:49)
Backend: dpdk
Sending packets...
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.81:9000 (expected 8000, OK)
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.81:9000 (expected 8000, OK)
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.81:9000 (expected 8000, OK)
Test complete
[2026-07-01T12:26:34Z] INFO: Jumbo frame echo succeeded: 3/3 responses with correct size
[2026-07-01T12:26:34Z] INFO: JUnit XML written to: /tmp/test-results/tier1-dpdk-echo.xml
[2026-07-01T12:26:34Z] INFO: Tier 1 sender tests complete. Results: /tmp/test-results/tier1-dpdk-echo.xml
[2026-07-01T12:30:52Z] INFO: Test: arp_resolution
UDP Test Client
Target: 10.0.1.81:9000
Bind address: 0.0.0.0:0
Message: 'arp-probe'
Count: 1
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
DPDK bind failed (Port init failed: Invalid port ID: 0), falling back to tokio
Backend: tokio
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.81:9000: 'arp-probe #1'
Test complete
[2026-07-01T12:30:52Z] INFO: ARP resolution succeeded (got response from peer)
[2026-07-01T12:30:52Z] INFO: Test: udp_send_receive
[2026-07-01T12:30:54Z] INFO: UDP send/receive succeeded
[2026-07-01T12:30:54Z] INFO: Test: echo_roundtrip
[2026-07-01T12:30:54Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-07-01T12:30:54Z] INFO: Test: payload_integrity
[2026-07-01T12:30:54Z] INFO: Response received, checking payload match...
[2026-07-01T12:30:54Z] INFO: Payload integrity verified (found in response)
[2026-07-01T12:30:54Z] INFO: JUnit XML written to: /tmp/test-results/tier2-kernel-interop.xml
[2026-07-01T12:30:54Z] INFO: Tier 2 sender tests complete. Results: /tmp/test-results/tier2-kernel-interop.xml

receiver-test-client-iperf.log

[2026-07-01T12:36:15Z] INFO: iperf-sends: sent 10 packets, received 10 responses
[2026-07-01T12:36:15Z] INFO: iperf-sends: PASS (sent >= 5 packets)
[2026-07-01T12:36:15Z] INFO: JUnit XML written to: /tmp/test-results/tier3-iperf-sends.xml
[2026-07-01T12:36:15Z] INFO: iperf-sends test complete

sender-test-client-iperf.log

[2026-07-01T12:35:16Z] INFO: Pre-flight: checking DPDK state and ARP cache...
[2026-07-01T12:35:16Z] INFO: Local IP: 10.0.1.208, Peer IP: 10.0.1.81, Port: 9000
[2026-07-01T12:35:16Z] INFO: /proc/net/arp contents:
IP address       HW type     Flags       HW address            Mask     Device
10.0.1.118       0x1         0x2         02:26:e2:66:cf:59     *        ens5
10.0.1.1         0x1         0x2         02:56:54:04:88:a3     *        ens5
10.0.1.208       0x1         0x2         02:71:3e:0c:2e:49     *        ens5
10.0.1.18        0x1         0x2         02:63:f9:24:ea:ff     *        ens5
10.0.1.81        0x1         0x2         02:74:58:97:55:83     *        ens5
[2026-07-01T12:35:16Z] INFO: DPDK runtime state:
No /var/run/dpdk/ directory
[2026-07-01T12:35:16Z] INFO: vfio-pci bindings:
0000:00:06.0
bind
module
new_id
remove_id
uevent
unbind
[2026-07-01T12:35:16Z] INFO: Test binary: /opt/dpdk-stdlib/target/release/test-client
-rwxr-xr-x. 2 root root 1950584 Jul  1 12:23 /opt/dpdk-stdlib/target/release/test-client
[2026-07-01T12:35:16Z] INFO: Launching test-client: /opt/dpdk-stdlib/target/release/test-client --target 10.0.1.81 --port 9000 --bind-ip 10.0.1.208 --count 10 --delay 200
[2026-07-01T12:35:18Z] INFO: Test client output: UDP Test Client
Target: 10.0.1.81:9000
Bind address: 10.0.1.208:0
Message: 'dpdk-to-kernel-test-payload'
Count: 10
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.208:32768 (MAC: 02:71:3e:0c:2e:49)
Backend: dpdk
Sending packets...
Sent 30 bytes: 'dpdk-to-kernel-test-payload #1'
Received 30 bytes from 10.0.1.81:9000: 'dpdk-to-kernel-test-payload #1'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #2'
Received 30 bytes from 10.0.1.81:9000: 'dpdk-to-kernel-test-payload #2'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #3'
Received 30 bytes from 10.0.1.81:9000: 'dpdk-to-kernel-test-payload #3'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #4'
Received 30 bytes from 10.0.1.81:9000: 'dpdk-to-kernel-test-payload #4'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #5'
Received 30 bytes from 10.0.1.81:9000: 'dpdk-to-kernel-test-payload #5'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #6'
Received 30 bytes from 10.0.1.81:9000: 'dpdk-to-kernel-test-payload #6'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #7'
Received 30 bytes from 10.0.1.81:9000: 'dpdk-to-kernel-test-payload #7'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #8'
Received 30 bytes from 10.0.1.81:9000: 'dpdk-to-kernel-test-payload #8'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #9'
Received 30 bytes from 10.0.1.81:9000: 'dpdk-to-kernel-test-payload #9'
Sent 31 bytes: 'dpdk-to-kernel-test-payload #10'
Received 31 bytes from 10.0.1.81:9000: 'dpdk-to-kernel-test-payload #10'
Test complete
[2026-07-01T12:35:18Z] INFO: our-app-sends: sent 10 packets, received 10 responses
[2026-07-01T12:35:18Z] INFO: our-app-sends: PASS (sent >= 5 packets)
[2026-07-01T12:35:18Z] INFO: JUnit XML written to: /tmp/test-results/tier3-our-app-sends.xml
[2026-07-01T12:35:18Z] INFO: our-app-sends test complete

⚠️ SSM Command Failures (receiver-ssm-failure.log)

=== Polling timeout after 30s ===
Status: InProgress
Instance: i-0b9ee3dcc9ae2fa0f (receiver)
Command ID: f966b992-490f-4c37-af75-dcc945083723

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-0b9ee3dcc9ae2fa0f (receiver)
Command ID: d5a55382-955a-4b55-ab51-378afebfe0dd

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-0b9ee3dcc9ae2fa0f (receiver)
Command ID: a191e0e7-3cc5-4248-968e-14d391c690d2

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-0b9ee3dcc9ae2fa0f (receiver)
Command ID: 37e868de-0f2b-414a-a0b4-0dcb8ea118e9

=== STDOUT ===


=== STDERR ===

⚠️ SSM Command Failures (sender-ssm-failure.log)

=== Polling timeout 120s (SSM timeout: 90s, command: cd /opt/dpdk-stdlib && bash scripts/integration-tests/configure-eni.sh --action unbind && bash scripts/integration-tests/configure-eni.sh --action assign-ip --ip 10.0.1.208) ===
Status: InProgress
Instance: i-04c5de817f89d555e (sender)
Command ID: a7824bd3-68b8-41bf-b6a0-02bb1683691e

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-04c5de817f89d555e (sender)
Command ID: 10a3a9c9-5af3-4eba-806a-82d8acf56995

=== STDOUT ===


=== STDERR ===

Network & PCI State

receiver-network-interfaces.log

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:c5:93:ef:57:c3 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-025d6d16fc246c1fc
    altname device-number-0.0
    inet 10.0.1.188/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 2212sec preferred_lft 2212sec
    inet6 fe80::c5:93ff:feef:57c3/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

sender-network-interfaces.log

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:ed:3c:cc:1b:91 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-0f32bfe0e029241d2
    altname device-number-0.0
    inet 10.0.1.75/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 2234sec preferred_lft 2234sec
    inet6 fe80::ed:3cff:fecc:1b91/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

receiver-networking-diag-baseline.txt

=== NETWORKING DIAGNOSTICS ===
timestamp: 2026-07-01T12:25:50Z
hostname: ip-10-0-1-188.ec2.internal
kernel: 6.18.35-68.129.amzn2023.x86_64

=== DPDK PORT STATUS ===

Network devices using DPDK-compatible driver
============================================
0000:00:06.0 'Elastic Network Adapter (ENA) ec20' drv=vfio-pci unused=ena

Network devices using kernel driver
===================================
0000:00:05.0 'Elastic Network Adapter (ENA) ec20' if=ens5 drv=ena unused=vfio-pci *Active*

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'DMA' devices detected
=========================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

No 'Regex' devices detected
===========================

=== IP ADDRESSES ===
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:c5:93:ef:57:c3 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-025d6d16fc246c1fc
    altname device-number-0.0
    inet 10.0.1.188/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 3146sec preferred_lft 3146sec
    inet6 fe80::c5:93ff:feef:57c3/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

=== ARP TABLE ===
10.0.1.18 dev ens5 lladdr 02:63:f9:24:ea:ff REACHABLE 
10.0.1.1 dev ens5 lladdr 02:56:54:04:88:a3 REACHABLE 

=== ROUTE TABLE ===
default via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.188 metric 512 
10.0.0.2 via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.188 metric 512 
10.0.1.0/24 dev ens5 proto kernel scope link src 10.0.1.188 metric 512 
10.0.1.1 dev ens5 proto dhcp scope link src 10.0.1.188 metric 512 

=== IMDS: ENI INFORMATION ===
ENI MACs found: 02:74:58:97:55:83/ 02:c5:93:ef:57:c3/ 

--- ENI: 02:74:58:97:55:83/ ---
  device-number: 1
  local-ipv4s: 10.0.1.81
  subnet-id: subnet-08833a1d280ac3180
  subnet-cidr: 10.0.1.0/24

--- ENI: 02:c5:93:ef:57:c3/ ---
  device-number: 0
  local-ipv4s: 10.0.1.188
  subnet-id: subnet-08833a1d280ac3180
  subnet-cidr: 10.0.1.0/24


=== GATEWAY ARP TEST ===
Gateway IP: 10.0.1.1
Gateway ARP entry:
10.0.1.1 dev ens5 lladdr 02:56:54:04:88:a3 REACHABLE 

arping result:
ARPING 10.0.1.1 from 10.0.1.188 ens5
Unicast reply from 10.0.1.1 [02:56:54:04:88:A3]  0.533ms
Sent 1 probes (1 broadcast(s))
Received 1 response(s)

=== HUGEPAGE STATUS ===
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:     51200 kB
HugePages_Total:    1024
HugePages_Free:     1024
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         2097152 kB

=== VFIO STATUS ===
total 0
drwxr-xr-x.  2 root root       80 Jul  1 12:25 .
drwxr-xr-x. 14 root root     3100 Jul  1 12:25 ..
crw-------.  1 root root 243,   0 Jul  1 12:25 noiommu-0
crw-rw-rw-.  1 root root  10, 196 Jul  1 12:18 vfio

noiommu mode:
Y

=== DPDK SHARED MEMORY ===
no /var/run/dpdk/ directory (clean state)

=== DPDK-RELATED DMESG (last 30 lines) ===
[    0.053543] printk: legacy console [ttyS0] enabled
[    0.054560] x2apic enabled
[    0.058722] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto
[    0.058832] x86/fpu: Enabled xstate features 0x2ff, context size is 2568 bytes, using 'compacted' format.
[    0.068577] audit: type=2000 audit(1782908291.590:1): state=initialized audit_enabled=0 res=1
[    0.088060] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
[    0.140288] ACPI: Interpreter enabled
[    0.140288] ACPI: Enabled 2 GPEs in block 00 to 0F
[    0.158117] pci 0000:00:05.0: enabling Extended Tags
[    0.244404] SGI XFS with ACLs, security attributes, quota, no debug enabled
[    0.256866] ACPI: \_SB_.LNKD: Enabled at IRQ 11
[    0.353478] IPI shorthand broadcast: enabled
[    2.466897] systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
[    2.506666] systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
[    2.572133] VFIO - User Level meta-driver version: 0.3
[    2.932385] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.17.1g
[    2.947087] ena 0000:00:05.0: ENA device version: 0.10
[    2.947808] ena 0000:00:05.0: ENA controller version: 0.0.1 implementation version 1
[    3.044584] ena 0000:00:05.0: ENA Large LLQ is disabled
[    3.057530] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem c0500000, mac addr 02:c5:93:ef:57:c3
[    3.079137] ena 0000:00:05.0 ens5: renamed from eth0
[  411.586018] pci 0000:00:06.0: enabling Extended Tags
[  411.590204] ena 0000:00:06.0: enabling device (0000 -> 0002)
[  411.604210] ena 0000:00:06.0: ENA device version: 0.10
[  411.604985] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  411.704695] ena 0000:00:06.0: ENA Large LLQ is disabled
[  411.716957] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:74:58:97:55:83
[  411.724429] ena 0000:00:06.0 ens6: renamed from eth0
[  446.448881] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  446.450459] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device

=== DPDK-RELATED PROCESSES ===
no DPDK processes running

=== END DIAGNOSTICS ===

sender-networking-diag-baseline.txt

=== NETWORKING DIAGNOSTICS ===
timestamp: 2026-07-01T12:25:43Z
hostname: ip-10-0-1-75.ec2.internal
kernel: 6.18.35-68.129.amzn2023.x86_64

=== DPDK PORT STATUS ===

Network devices using DPDK-compatible driver
============================================
0000:00:06.0 'Elastic Network Adapter (ENA) ec20' drv=vfio-pci unused=ena

Network devices using kernel driver
===================================
0000:00:05.0 'Elastic Network Adapter (ENA) ec20' if=ens5 drv=ena unused=vfio-pci *Active*

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'DMA' devices detected
=========================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

No 'Regex' devices detected
===========================

=== IP ADDRESSES ===
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:ed:3c:cc:1b:91 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-0f32bfe0e029241d2
    altname device-number-0.0
    inet 10.0.1.75/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 3155sec preferred_lft 3155sec
    inet6 fe80::ed:3cff:fecc:1b91/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

=== ARP TABLE ===
10.0.1.118 dev ens5 lladdr 02:26:e2:66:cf:59 REACHABLE 
10.0.1.1 dev ens5 lladdr 02:56:54:04:88:a3 REACHABLE 
10.0.1.18 dev ens5 lladdr 02:63:f9:24:ea:ff REACHABLE 

=== ROUTE TABLE ===
default via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.75 metric 512 
10.0.0.2 via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.75 metric 512 
10.0.1.0/24 dev ens5 proto kernel scope link src 10.0.1.75 metric 512 
10.0.1.1 dev ens5 proto dhcp scope link src 10.0.1.75 metric 512 

=== IMDS: ENI INFORMATION ===
ENI MACs found: 02:71:3e:0c:2e:49/ 02:ed:3c:cc:1b:91/ 

--- ENI: 02:71:3e:0c:2e:49/ ---
  device-number: 1
  local-ipv4s: 10.0.1.208
  subnet-id: subnet-08833a1d280ac3180
  subnet-cidr: 10.0.1.0/24

--- ENI: 02:ed:3c:cc:1b:91/ ---
  device-number: 0
  local-ipv4s: 10.0.1.75
  subnet-id: subnet-08833a1d280ac3180
  subnet-cidr: 10.0.1.0/24


=== GATEWAY ARP TEST ===
Gateway IP: 10.0.1.1
Gateway ARP entry:
10.0.1.1 dev ens5 lladdr 02:56:54:04:88:a3 REACHABLE 

arping result:
ARPING 10.0.1.1 from 10.0.1.75 ens5
Unicast reply from 10.0.1.1 [02:56:54:04:88:A3]  0.533ms
Sent 1 probes (1 broadcast(s))
Received 1 response(s)

=== HUGEPAGE STATUS ===
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:     51200 kB
HugePages_Total:    1024
HugePages_Free:     1024
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         2097152 kB

=== VFIO STATUS ===
total 0
drwxr-xr-x.  2 root root       80 Jul  1 12:25 .
drwxr-xr-x. 14 root root     3100 Jul  1 12:24 ..
crw-------.  1 root root 243,   0 Jul  1 12:25 noiommu-0
crw-rw-rw-.  1 root root  10, 196 Jul  1 12:18 vfio

noiommu mode:
Y

=== DPDK SHARED MEMORY ===
no /var/run/dpdk/ directory (clean state)

=== DPDK-RELATED DMESG (last 30 lines) ===
[    0.052718] printk: legacy console [ttyS0] enabled
[    0.053710] x2apic enabled
[    0.057811] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto
[    0.057917] x86/fpu: Enabled xstate features 0x2ff, context size is 2568 bytes, using 'compacted' format.
[    0.067695] audit: type=2000 audit(1782908291.843:1): state=initialized audit_enabled=0 res=1
[    0.087180] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
[    0.139090] ACPI: Interpreter enabled
[    0.139090] ACPI: Enabled 2 GPEs in block 00 to 0F
[    0.157225] pci 0000:00:05.0: enabling Extended Tags
[    0.226068] SGI XFS with ACLs, security attributes, quota, no debug enabled
[    0.238896] ACPI: \_SB_.LNKD: Enabled at IRQ 11
[    0.348653] IPI shorthand broadcast: enabled
[    3.736469] systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
[    3.764185] systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
[    3.830376] VFIO - User Level meta-driver version: 0.3
[    4.224914] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.17.1g
[    4.235590] ena 0000:00:05.0: ENA device version: 0.10
[    4.236304] ena 0000:00:05.0: ENA controller version: 0.0.1 implementation version 1
[    4.341097] ena 0000:00:05.0: ENA Large LLQ is disabled
[    4.353829] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem c0500000, mac addr 02:ed:3c:cc:1b:91
[    4.378576] ena 0000:00:05.0 ens5: renamed from eth0
[  392.182199] pci 0000:00:06.0: enabling Extended Tags
[  392.186095] ena 0000:00:06.0: enabling device (0000 -> 0002)
[  392.198855] ena 0000:00:06.0: ENA device version: 0.10
[  392.199590] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  392.300731] ena 0000:00:06.0: ENA Large LLQ is disabled
[  392.314482] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:71:3e:0c:2e:49
[  392.320229] ena 0000:00:06.0 ens6: renamed from eth0
[  439.677058] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  439.678462] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device

=== DPDK-RELATED PROCESSES ===
no DPDK processes running

=== END DIAGNOSTICS ===

⚠️ Crash Diagnostics

receiver-dmesg-crashes.log

[    0.067963] pid_max: default: 32768 minimum: 301
[    0.161902] iommu: Default domain type: Translated
[    0.161927] NetLabel:  unlabeled traffic allowed by default
[    0.181749] PCI: CLS 0 bytes, default 64
[    0.275179] nvme nvme0: 2/0/0 default/read/poll queues
[    0.418488] systemd[1]: systemd 252.23-12.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    0.418610] systemd[1]: No hostname configured, using default hostname.
[    0.489816] systemd[1]: Queued start job for default target initrd.target.
[    1.835354] systemd[1]: systemd 252.23-12.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[  481.168048] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:8277)
[  745.918232] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:8765)
[ 1251.118498] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:9920)

sender-dmesg-crashes.log

[    0.067082] pid_max: default: 32768 minimum: 301
[    0.161016] iommu: Default domain type: Translated
[    0.161054] NetLabel:  unlabeled traffic allowed by default
[    0.182354] PCI: CLS 0 bytes, default 64
[    0.263941] nvme nvme0: 2/0/0 default/read/poll queues
[    0.428992] systemd[1]: systemd 252.23-12.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    0.429115] systemd[1]: No hostname configured, using default hostname.
[    0.494038] systemd[1]: Queued start job for default target initrd.target.
[    3.012092] systemd[1]: systemd 252.23-12.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[ 1066.608392] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:9505)

Kernel Console (dmesg)

receiver-console-output.log (PCI/driver events only)

[  481.166388] vfio-pci 0000:00:06.0: reset done
[  481.168048] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:8277)
[  481.169544] vfio-pci 0000:00:06.0: resetting
[  481.386184] vfio-pci 0000:00:06.0: reset done
[  558.373218] vfio-pci 0000:00:06.0: Removing from iommu group 0
[  559.384857] ena 0000:00:06.0: ENA device version: 0.10
[  559.385672] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  559.484840] ena 0000:00:06.0: ENA Large LLQ is disabled
[  559.497101] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:74:58:97:55:83
[  559.505043] ena 0000:00:06.0 ens6: renamed from eth0
[  578.822748] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  578.824268] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[  745.695060] vfio-pci 0000:00:06.0: resetting
[  745.916591] vfio-pci 0000:00:06.0: reset done
[  745.918232] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:8765)
[  745.919702] vfio-pci 0000:00:06.0: resetting
[  746.136308] vfio-pci 0000:00:06.0: reset done
[  817.654988] vfio-pci 0000:00:06.0: Removing from iommu group 0
[  818.664784] ena 0000:00:06.0: ENA device version: 0.10
[  818.665513] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  818.764771] ena 0000:00:06.0: ENA Large LLQ is disabled
[  818.777643] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:74:58:97:55:83
[  818.785407] ena 0000:00:06.0 ens6: renamed from eth0
[ 1162.979265] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1162.980784] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1250.884850] vfio-pci 0000:00:06.0: resetting
[ 1251.116847] vfio-pci 0000:00:06.0: reset done
[ 1251.118498] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:9920)
[ 1251.119989] vfio-pci 0000:00:06.0: resetting
[ 1251.336530] vfio-pci 0000:00:06.0: reset done

sender-console-output.log (PCI/driver events only)

[ 1066.609733] vfio-pci 0000:00:06.0: resetting
[ 1066.826748] vfio-pci 0000:00:06.0: reset done
[ 1126.366308] vfio-pci 0000:00:06.0: Removing from iommu group 0
[ 1127.374690] ena 0000:00:06.0: ENA device version: 0.10
[ 1127.375435] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[ 1127.474670] ena 0000:00:06.0: ENA Large LLQ is disabled
[ 1127.486880] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:71:3e:0c:2e:49
[ 1127.494334] ena 0000:00:06.0 ens6: renamed from eth0
[ 1156.347912] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1156.349336] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1266.400162] vfio-pci 0000:00:06.0: resetting
[ 1266.615091] vfio-pci 0000:00:06.0: reset done
[ 1266.616609] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:9947)
[ 1266.618136] vfio-pci 0000:00:06.0: resetting
[ 1266.834971] vfio-pci 0000:00:06.0: reset done
[ 1267.122429] vfio-pci 0000:00:06.0: resetting
[ 1267.334924] vfio-pci 0000:00:06.0: reset done
[ 1267.336446] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:9967)
[ 1267.337990] vfio-pci 0000:00:06.0: resetting
[ 1267.554935] vfio-pci 0000:00:06.0: reset done
[ 1268.844520] vfio-pci 0000:00:06.0: resetting
[ 1269.064985] vfio-pci 0000:00:06.0: reset done
[ 1269.066505] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:9990)
[ 1269.068024] vfio-pci 0000:00:06.0: resetting
[ 1269.284900] vfio-pci 0000:00:06.0: reset done
[ 1270.574661] vfio-pci 0000:00:06.0: resetting
[ 1270.794976] vfio-pci 0000:00:06.0: reset done
[ 1270.796504] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:10013)
[ 1270.798048] vfio-pci 0000:00:06.0: resetting
[ 1271.014924] vfio-pci 0000:00:06.0: reset done

gspivey merged commit cb10887 into development Jul 1, 2026
7 checks passed

gspivey deleted the agent/trex-lro-fix branch July 1, 2026 12:40

gspivey mentioned this pull request Jul 1, 2026

fix(perf): TRex ASTFIPGenDist needs a [start,end] IP range #105

Merged

Conversation

gspivey commented Jul 1, 2026

Fix

Uh oh!

github-actions Bot commented Jul 1, 2026

Synthetic Performance Results (run)

Synthetic UDP Performance Results

IPv4 Baseline

IPv6

IPv6 vs IPv4 Comparison (sync path)

Uh oh!

github-actions Bot commented Jul 1, 2026

Synthetic Performance Results — Graviton (run)

Synthetic UDP Performance Results

IPv4 Baseline

IPv6

IPv6 vs IPv4 Comparison (sync path)

Uh oh!

github-actions Bot commented Jul 1, 2026

[CI] Stage: Deploy

Uh oh!

github-actions Bot commented Jul 1, 2026

[CI] Stage: Deploy

Uh oh!

Uh oh!

github-actions Bot commented Jul 1, 2026

[CI] Stage: Summary

Uh oh!

github-actions Bot commented Jul 1, 2026

[CI] Stage: Summary

Uh oh!

github-actions Bot commented Jul 1, 2026

✅ Integration Tests Passed — Graviton (run)

Test Results

Application Logs (last 20 lines)

Uh oh!

github-actions Bot commented Jul 1, 2026

✅ Integration Tests Passed (Run 28516698984)

Test Results

Application Logs (last 20 lines)

receiver-echo-server.log

sender-echo-server.log

sender-test-client.log

receiver-test-client-iperf.log

sender-test-client-iperf.log

receiver-network-interfaces.log

sender-network-interfaces.log

receiver-networking-diag-baseline.txt

sender-networking-diag-baseline.txt

receiver-dmesg-crashes.log

sender-dmesg-crashes.log

receiver-console-output.log (PCI/driver events only)

sender-console-output.log (PCI/driver events only)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant