Skip to content

feat: dpdk-stdlib-tcp Performance CI workflow#97

Open
gspivey wants to merge 1 commit into
developmentfrom
agent/tcp-perf-ci
Open

feat: dpdk-stdlib-tcp Performance CI workflow#97
gspivey wants to merge 1 commit into
developmentfrom
agent/tcp-perf-ci

Conversation

@gspivey

@gspivey gspivey commented Jun 20, 2026

Copy link
Copy Markdown
Owner

Summary

Implements ROADMAP #33: dpdk-stdlib-tcp Performance CI workflow.

Spec: .kiro/specs/tcp-support/ task 15.5

Changes

  • scripts/run-perf-tests-tcp.sh — TCP performance test orchestrator (TRex ASTF mode)
    • Deploys CDK PerfTestStack
    • Runs TRex TCP benchmarks across 3 DUT configs: plain-rust-tcp, rust-dpdk-tcp, tokio-dpdk-tcp
    • Configurable payload sizes (default: 64,512,1400,65536 B), CPS rates (100-5000), duration
    • Posts results to PR comment with throughput, latency percentiles, CPS metrics
    • Safety-net teardown on failure

Blocked: Workflow YAML requires workflow scope on PAT

.github/workflows/perf-tests-tcp.yml cannot be pushed because the current GitHub fine-grained PAT lacks the workflow scope. The file is ready locally (validated YAML + syntax) but GitHub rejects the push.

To unblock: Update the GITHUB_TOKEN / GITHUB_TOKEN_DPDK fine-grained PAT to include the workflow permission, then push the workflow file:

git checkout agent/tcp-perf-ci
# Add the workflow file and push

Tests

  • cargo build — ✅ passes
  • bash -n scripts/run-perf-tests-tcp.sh — ✅ syntax valid
  • YAML validated via Python yaml.safe_load — ✅ valid
  • cargo test — pre-existing linker crash (Bus error / memory constraint) unrelated to this scripts-only change

@github-actions

github-actions Bot commented Jun 20, 2026

Copy link
Copy Markdown

Synthetic Performance Results (run)

Commit: c9ee553a

✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)

Synthetic UDP Performance Results

Measures framework overhead: sync dpdk_udp::UdpSocket vs async (std::sync::Mutex + try_recv_from).

IPv4 Baseline

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to 64B 11.8M 11.4M 1.0x 87
RX recv_from 64B 3.7M 4.8M 0.8x 209
TX send_to 1400B 1.8M 1.8M 1.0x 553
RX recv_from 1400B 1.1M 1.3M 0.9x 788

IPv6

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to (IPv6) 64B 9.0M 8.9M 1.0x 112
RX recv_from (IPv6) 64B 4.1M 5.6M 0.7x 177
TX send_to (IPv6) 1400B 2.9M 2.9M 1.0x 347
RX recv_from (IPv6) 1400B 1.2M 1.2M 0.9x 801

IPv6 vs IPv4 Comparison (sync path)

Test Payload IPv4 PPS IPv6 PPS IPv4/IPv6 Ratio
TX send_to (sync) 64B 11.8M 9.0M 1.31x
RX recv_from (sync) 64B 3.7M 4.1M 0.91x
TX send_to (sync) 1400B 1.8M 2.9M 0.63x
RX recv_from (sync) 1400B 1.1M 1.2M 0.99x

IPv4 avg sync/async ratio: 0.9x, worst: 1.0x | IPv6 vs IPv4 worst ratio: 1.31x (OK)

OK: IPv6 is 31.0% slower than IPv4 — within acceptable threshold (<50%). Expected due to larger headers (40B vs 20B) and mandatory UDP checksum.

Good: Async wrapper is within 1.0x of sync — minimal framework overhead.

@github-actions

github-actions Bot commented Jun 20, 2026

Copy link
Copy Markdown

Synthetic Performance Results — Graviton (run)

Commit: c9ee553a

✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)

Synthetic UDP Performance Results

Measures framework overhead: sync dpdk_udp::UdpSocket vs async (std::sync::Mutex + try_recv_from).

IPv4 Baseline

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to 64B 10.8M 10.1M 1.1x 98
RX recv_from 64B 3.3M 3.6M 0.9x 278
TX send_to 1400B 1.7M 1.6M 1.0x 607
RX recv_from 1400B 966.5K 996.4K 1.0x 1003

IPv6

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to (IPv6) 64B 9.0M 8.5M 1.1x 118
RX recv_from (IPv6) 64B 3.7M 4.1M 0.9x 243
TX send_to (IPv6) 1400B 2.7M 2.6M 1.0x 379
RX recv_from (IPv6) 1400B 952.2K 991.5K 1.0x 1008

IPv6 vs IPv4 Comparison (sync path)

Test Payload IPv4 PPS IPv6 PPS IPv4/IPv6 Ratio
TX send_to (sync) 64B 10.8M 9.0M 1.20x
RX recv_from (sync) 64B 3.3M 3.7M 0.89x
TX send_to (sync) 1400B 1.7M 2.7M 0.62x
RX recv_from (sync) 1400B 966.5K 952.2K 1.01x

IPv4 avg sync/async ratio: 1.0x, worst: 1.1x | IPv6 vs IPv4 worst ratio: 1.20x (OK)

OK: IPv6 is 20.4% slower than IPv4 — within acceptable threshold (<50%). Expected due to larger headers (40B vs 20B) and mandatory UDP checksum.

Good: Async wrapper is within 1.1x of sync — minimal framework overhead.

@github-actions

github-actions Bot commented Jun 20, 2026

Copy link
Copy Markdown

[CI] Stage: Deploy

Infrastructure ready.

  • Sender: i-053064e45960cb2b0 (DPDK ENI: 10.0.1.65)
  • Receiver: i-05e1591e6ff3d7614 (DPDK ENI: 10.0.1.170)
  • Both instances SSM-ready.

@github-actions

github-actions Bot commented Jun 20, 2026

Copy link
Copy Markdown

[CI] Stage: Deploy

Infrastructure ready.

  • Sender: i-0002c2d63f412360d (DPDK ENI: 10.0.1.190)
  • Receiver: i-01a623c3bb29bb016 (DPDK ENI: 10.0.1.198)
  • Both instances SSM-ready.

@github-actions

github-actions Bot commented Jun 20, 2026

Copy link
Copy Markdown

[CI] Stage: Summary

Some tests FAILED (exit code: 1).

ARP seeding: kernel /proc/net/arp (automatic)

  • tier1-dpdk-echo: 6 tests, 0 failures
  • tier2-kernel-interop: 4 tests, 0 failures
  • tier3-iperf-sends: 1 tests, 1 failures
  • tier3-our-app-sends: 1 tests, 1 failures

@github-actions

github-actions Bot commented Jun 20, 2026

Copy link
Copy Markdown

[CI] Stage: Summary

All tests PASSED.

ARP seeding: kernel /proc/net/arp (automatic)

  • tier1-dpdk-echo: 6 tests, 0 failures
  • tier2-kernel-interop: 4 tests, 0 failures
  • tier3-iperf-interop: 1 tests, 0 failures
  • tier3-iperf-interop: 1 tests, 0 failures

@github-actions

github-actions Bot commented Jun 20, 2026

Copy link
Copy Markdown

Integration Test Failure — Graviton (Run 27856186190)

Branch: 97/merge | Commit: c9ee553a1c4f326e587ac2be61b4f1666f1bfcbc

receiver-user-data.log (7011 bytes, last 80 lines)

   Compiling anstream v1.0.0
   Compiling memoffset v0.9.1
   Compiling futures-macro v0.3.32
   Compiling clap_lex v1.1.0
   Compiling futures-io v0.3.32
   Compiling lazy_static v1.5.0
   Compiling futures-task v0.3.32
   Compiling heck v0.5.0
   Compiling s2n-tls v0.3.38
   Compiling slab v0.4.12
   Compiling strsim v0.11.1
   Compiling clap_builder v4.6.0
   Compiling futures-util v0.3.32
   Compiling clap_derive v4.6.1
   Compiling rand_core v0.5.1
   Compiling ppv-lite86 v0.2.21
   Compiling getrandom v0.4.3
   Compiling rand_core v0.10.1
   Compiling hex v0.4.3
   Compiling rand_chacha v0.2.2
   Compiling clap v4.6.1
   Compiling futures-executor v0.3.32
   Compiling dpdk-stdlib-tcp v0.2.0 (/opt/dpdk-stdlib/dpdk-stdlib-tcp)
   Compiling ring v0.17.14
   Compiling allocator-api2 v0.2.21
   Compiling num-conv v0.2.2
   Compiling equivalent v1.0.2
   Compiling deranged v0.5.8
   Compiling foldhash v0.2.0
   Compiling powerfmt v0.2.0
   Compiling time-core v0.1.9
   Compiling s2n-quic-platform v0.81.0
   Compiling time v0.3.49
   Compiling hashbrown v0.17.1
   Compiling futures v0.3.32
   Compiling intrusive-collections v0.10.2
   Compiling rand v0.7.3
   Compiling chacha20 v0.10.0
   Compiling fnv v1.0.7
   Compiling untrusted v0.9.0
   Compiling base64 v0.22.1
   Compiling pem v3.0.6
   Compiling s2n-quic-crypto v0.81.0
   Compiling cuckoofilter v0.5.0
   Compiling rand v0.10.1
   Compiling s2n-quic-transport v0.81.0
   Compiling yasna v0.5.2
   Compiling rustls-pki-types v1.14.1
   Compiling hash_hasher v2.0.4
   Compiling serde_core v1.0.228
   Compiling rcgen v0.13.2
   Compiling async-trait v0.1.89
   Compiling zmij v1.0.21
   Compiling dpdk-stdlib-tokio v0.2.0 (/opt/dpdk-stdlib/dpdk-tokio)
   Compiling serde v1.0.228
   Compiling serde_json v1.0.150
   Compiling serde_derive v1.0.228
   Compiling itoa v1.0.18
   Compiling synthetic-bench v0.2.0 (/opt/dpdk-stdlib/apps/synthetic-bench)
   Compiling tcp-synthetic-bench v0.2.0 (/opt/dpdk-stdlib/apps/tcp-synthetic-bench)
   Compiling tokio-tcp-echo v0.1.0 (/opt/dpdk-stdlib/apps/tokio-tcp-echo)
   Compiling tokio-echo v0.1.0 (/opt/dpdk-stdlib/apps/tokio-echo)
   Compiling test-client v0.1.0 (/opt/dpdk-stdlib/apps/test-client)
   Compiling tcp-echo v0.1.0 (/opt/dpdk-stdlib/apps/tcp-echo)
   Compiling tcp-test-client v0.1.0 (/opt/dpdk-stdlib/apps/tcp-test-client)
   Compiling plain-tcp-echo v0.2.0 (/opt/dpdk-stdlib/apps/plain-tcp-echo)
   Compiling echo v0.1.0 (/opt/dpdk-stdlib/apps/echo)
   Compiling s2n-quic-tls v0.81.0
   Compiling s2n-quic-tls-default v0.81.0
   Compiling s2n-quic v1.81.0
   Compiling plain-echo v0.1.0 (/opt/dpdk-stdlib/apps/plain-echo)
   Compiling dpdk-stdlib-quic v0.2.0 (/opt/dpdk-stdlib/dpdk-stdlib-quic)
    Finished `release` profile [optimized] target(s) in 3m 18s
=== Build complete ===
-rwxr-xr-x. 2 root root 1428072 Jun 20 01:41 target/release/echo
-rwxr-xr-x. 2 root root 1934640 Jun 20 01:41 target/release/test-client
=== Setup complete! ===
Rust project built successfully
Instance ready for testing
Project location: /opt/dpdk-stdlib

sender-user-data.log (7054 bytes, last 80 lines)

   Compiling anstream v1.0.0
   Compiling memoffset v0.9.1
   Compiling futures-macro v0.3.32
   Compiling lazy_static v1.5.0
   Compiling slab v0.4.12
   Compiling futures-task v0.3.32
   Compiling futures-io v0.3.32
   Compiling strsim v0.11.1
   Compiling heck v0.5.0
   Compiling clap_lex v1.1.0
   Compiling s2n-tls v0.3.38
   Compiling clap_builder v4.6.0
   Compiling clap_derive v4.6.1
   Compiling futures-util v0.3.32
   Compiling rand_core v0.5.1
   Compiling ppv-lite86 v0.2.21
   Compiling hex v0.4.3
   Compiling rand_core v0.10.1
   Compiling getrandom v0.4.3
   Compiling rand_chacha v0.2.2
   Compiling futures-executor v0.3.32
   Compiling clap v4.6.1
   Compiling dpdk-stdlib-tcp v0.2.0 (/opt/dpdk-stdlib/dpdk-stdlib-tcp)
   Compiling ring v0.17.14
   Compiling deranged v0.5.8
   Compiling num-conv v0.2.2
   Compiling time-core v0.1.9
   Compiling equivalent v1.0.2
   Compiling powerfmt v0.2.0
   Compiling s2n-quic-platform v0.81.0
   Compiling allocator-api2 v0.2.21
   Compiling foldhash v0.2.0
   Compiling hashbrown v0.17.1
   Compiling time v0.3.49
   Compiling futures v0.3.32
   Compiling intrusive-collections v0.10.2
   Compiling rand v0.7.3
   Compiling chacha20 v0.10.0
   Compiling untrusted v0.9.0
   Compiling base64 v0.22.1
   Compiling fnv v1.0.7
   Compiling cuckoofilter v0.5.0
   Compiling pem v3.0.6
   Compiling rand v0.10.1
   Compiling s2n-quic-transport v0.81.0
   Compiling s2n-quic-crypto v0.81.0
   Compiling yasna v0.5.2
   Compiling rustls-pki-types v1.14.1
   Compiling hash_hasher v2.0.4
   Compiling serde_core v1.0.228
   Compiling rcgen v0.13.2
   Compiling async-trait v0.1.89
   Compiling zmij v1.0.21
   Compiling dpdk-stdlib-tokio v0.2.0 (/opt/dpdk-stdlib/dpdk-tokio)
   Compiling serde v1.0.228
   Compiling serde_json v1.0.150
   Compiling serde_derive v1.0.228
   Compiling itoa v1.0.18
   Compiling synthetic-bench v0.2.0 (/opt/dpdk-stdlib/apps/synthetic-bench)
   Compiling tcp-synthetic-bench v0.2.0 (/opt/dpdk-stdlib/apps/tcp-synthetic-bench)
   Compiling tokio-echo v0.1.0 (/opt/dpdk-stdlib/apps/tokio-echo)
   Compiling tokio-tcp-echo v0.1.0 (/opt/dpdk-stdlib/apps/tokio-tcp-echo)
   Compiling test-client v0.1.0 (/opt/dpdk-stdlib/apps/test-client)
   Compiling tcp-test-client v0.1.0 (/opt/dpdk-stdlib/apps/tcp-test-client)
   Compiling tcp-echo v0.1.0 (/opt/dpdk-stdlib/apps/tcp-echo)
   Compiling plain-echo v0.1.0 (/opt/dpdk-stdlib/apps/plain-echo)
   Compiling plain-tcp-echo v0.2.0 (/opt/dpdk-stdlib/apps/plain-tcp-echo)
   Compiling echo v0.1.0 (/opt/dpdk-stdlib/apps/echo)
   Compiling s2n-quic-tls v0.81.0
   Compiling s2n-quic-tls-default v0.81.0
   Compiling s2n-quic v1.81.0
   Compiling dpdk-stdlib-quic v0.2.0 (/opt/dpdk-stdlib/dpdk-stdlib-quic)
    Finished `release` profile [optimized] target(s) in 3m 07s
=== Build complete ===
-rwxr-xr-x. 2 root root 1428072 Jun 20 01:41 target/release/echo
-rwxr-xr-x. 2 root root 1934640 Jun 20 01:40 target/release/test-client
=== Setup complete! ===
Rust project built successfully
Instance ready for testing
Project location: /opt/dpdk-stdlib

receiver-console-output.log (44376 bytes, last 80 lines)

[  331.993679] cloud-init[2011]: -rwxr-xr-x. 2 root root 1934640 Jun 20 01:41 target/release/test-client
[  331.996427] cloud-init[2011]: === Setup complete! ===
[  331.997689] cloud-init[2011]: Rust project built successfully
[  332.001190] cloud-init[2011]: Instance ready for testing
[  332.002492] cloud-init[2011]: Project location: /opt/dpdk-stdlib
ci-info: +++++++++++++++++++++++++++++++++++++++++++++Authorized keys from /home/ec2-user/.ssh/authorized_keys for user ec2-user++++++++++++++++++++++++++++++++++++++++++++++
ci-info: +---------+-------------------------------------------------------------------------------------------------+---------+---------------------------------------------+
ci-info: | Keytype |                                       Fingerprint (sha256)                                      | Options |                   Comment                   |
ci-info: +---------+-------------------------------------------------------------------------------------------------+---------+---------------------------------------------+
ci-info: | ssh-rsa | 87:74:24:48:b8:87:f3:01:d3:56:66:dc:36:8f:6c:18:f5:43:04:54:be:9a:69:46:1b:49:38:7b:1e:3c:39:86 |    -    | packer_6a32352b-7474-b657-8f55-7b911f4e3a9c |
ci-info: +---------+-------------------------------------------------------------------------------------------------+---------+---------------------------------------------+
<14>Jun 20 01:42:04 cloud-init: #############################################################
<14>Jun 20 01:42:04 cloud-init: -----BEGIN SSH HOST KEY FINGERPRINTS-----
<14>Jun 20 01:42:04 cloud-init: 256 SHA256:+DgtEtqo5P1Z5gXcabEaOeX56OkcZ4cUYr6WZ4GXXOk root@ip-10-0-1-200.ec2.internal (ECDSA)
<14>Jun 20 01:42:04 cloud-init: 256 SHA256:3XJb10/SsghKlZCYUyV4LQK85RzRXSZoxJosJSIjvqg root@ip-10-0-1-200.ec2.internal (ED25519)
<14>Jun 20 01:42:04 cloud-init: -----END SSH HOST KEY FINGERPRINTS-----
<14>Jun 20 01:42:04 cloud-init: #############################################################
-----BEGIN SSH HOST KEY KEYS-----
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMmbfui8oMfi48c8dpvdqEU7HqWo+K3mdw5yZlRvQRm8UDoHZvfZYLz2qqzQV0STrU7fZzV660+YfnXWQ+NtQZY= root@ip-10-0-1-200.ec2.internal
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBwWlCs1z15KsvT+36DvM4o7XcrCkBAGneajuo7zsWOx root@ip-10-0-1-200.ec2.internal
-----END SSH HOST KEY KEYS-----
[  332.761284] cloud-init[2011]: Cloud-init v. 22.2.2 finished at Sat, 20 Jun 2026 01:42:04 +0000. Datasource DataSourceEc2.  Up 332.74 seconds
[  351.922587] pci 0000:00:06.0: [1d0f:ec20] type 00 class 0x020000 PCIe Root Complex Integrated Endpoint
[  351.923951] pci 0000:00:06.0: BAR 0 [mem 0x00000000-0x00001fff]
[  351.924615] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00001fff]
[  351.925423] pci 0000:00:06.0: BAR 2 [mem 0x00000000-0x0003ffff pref]
[  351.926338] pci 0000:00:06.0: enabling Extended Tags
[  351.927145] pci 0000:00:06.0: PME# supported from D0 D1 D2 D3hot D3cold
[  351.928142] pci 0000:00:06.0: BAR 2 [mem 0x80080000-0x800bffff pref]: assigned
[  351.928992] pci 0000:00:06.0: BAR 0 [mem 0x8004a000-0x8004bfff]: assigned
[  351.929913] pci 0000:00:06.0: BAR 1 [mem 0x8004c000-0x8004dfff]: assigned
[  351.931130] ena 0000:00:06.0: enabling device (0000 -> 0002)
[  351.940467] ena 0000:00:06.0: ENA device version: 0.10
[  351.941072] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  352.040302] ena 0000:00:06.0: Forcing large headers and decreasing maximum TX queue size to 512
[  352.041369] ena 0000:00:06.0: ENA Large LLQ is enabled
[  352.052262] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem 8004a000, mac addr 02:0c:9b:21:42:af
[  352.061183] ena 0000:00:06.0 ens6: renamed from eth0
[  352.210814] zram_generator::config[7787]: zram0: system has too much memory (3806MB), limit is 800MB, ignoring.
[  386.889680] zram_generator::config[8033]: zram0: system has too much memory (3806MB), limit is 800MB, ignoring.
[  387.816993] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  387.817930] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[  422.748858] vfio-pci 0000:00:06.0: enabling device (0000 -> 0002)
[  422.750886] vfio-pci 0000:00:06.0: resetting
[  422.972613] vfio-pci 0000:00:06.0: reset done
[  422.973442] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:8138)
[  422.974546] vfio-pci 0000:00:06.0: resetting
[  423.182615] vfio-pci 0000:00:06.0: reset done
[  500.794354] vfio-pci 0000:00:06.0: Removing from iommu group 0
[  501.802239] ena 0000:00:06.0: ENA device version: 0.10
[  501.802832] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  501.902121] ena 0000:00:06.0: Forcing large headers and decreasing maximum TX queue size to 512
[  501.903217] ena 0000:00:06.0: ENA Large LLQ is enabled
[  501.913709] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem 8004a000, mac addr 02:0c:9b:21:42:af
[  501.922447] ena 0000:00:06.0 ens6: renamed from eth0 (while UP)
[  502.087626] zram_generator::config[8302]: zram0: system has too much memory (3806MB), limit is 800MB, ignoring.
[  520.233635] zram_generator::config[8489]: zram0: system has too much memory (3806MB), limit is 800MB, ignoring.
[  521.186255] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  521.187120] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[  690.451513] vfio-pci 0000:00:06.0: resetting
[  690.666027] vfio-pci 0000:00:06.0: reset done
[  690.666854] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:8574)
[  690.667955] vfio-pci 0000:00:06.0: resetting
[  690.885940] vfio-pci 0000:00:06.0: reset done
[  763.064284] vfio-pci 0000:00:06.0: Removing from iommu group 0
[  764.075461] ena 0000:00:06.0: ENA device version: 0.10
[  764.076059] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  764.175324] ena 0000:00:06.0: Forcing large headers and decreasing maximum TX queue size to 512
[  764.176410] ena 0000:00:06.0: ENA Large LLQ is enabled
[  764.186919] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem 8004a000, mac addr 02:0c:9b:21:42:af
[  764.191468] ena 0000:00:06.0 ens6: renamed from eth0
[  764.286480] zram_generator::config[8734]: zram0: system has too much memory (3806MB), limit is 800MB, ignoring.
[ 1099.731051] zram_generator::config[9433]: zram0: system has too much memory (3806MB), limit is 800MB, ignoring.
[ 1100.643474] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1100.644423] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1194.467595] vfio-pci 0000:00:06.0: resetting
[ 1194.682144] vfio-pci 0000:00:06.0: reset done
[ 1194.682976] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:9518)
[ 1194.684078] vfio-pci 0000:00:06.0: resetting
[ 1194.892059] vfio-pci 0000:00:06.0: reset done

sender-console-output.log (46041 bytes, last 80 lines)

[  331.899378] pci 0000:00:06.0: BAR 1 [mem 0x8004c000-0x8004dfff]: assigned
[  331.900584] ena 0000:00:06.0: enabling device (0000 -> 0002)
[  331.910260] ena 0000:00:06.0: ENA device version: 0.10
[  331.910849] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  332.010096] ena 0000:00:06.0: Forcing large headers and decreasing maximum TX queue size to 512
[  332.011175] ena 0000:00:06.0: ENA Large LLQ is enabled
[  332.021778] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem 8004a000, mac addr 02:08:02:3e:e8:e7
[  332.030633] ena 0000:00:06.0 ens6: renamed from eth0
[  332.199258] zram_generator::config[7818]: zram0: system has too much memory (3806MB), limit is 800MB, ignoring.
[  381.304720] zram_generator::config[8061]: zram0: system has too much memory (3806MB), limit is 800MB, ignoring.
[  382.175383] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  382.176292] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[  439.051443] vfio-pci 0000:00:06.0: enabling device (0000 -> 0002)
[  439.053495] vfio-pci 0000:00:06.0: resetting
[  439.272601] vfio-pci 0000:00:06.0: reset done
[  439.273458] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:8174)
[  439.274711] vfio-pci 0000:00:06.0: resetting
[  439.492561] vfio-pci 0000:00:06.0: reset done
[  439.776341] vfio-pci 0000:00:06.0: resetting
[  439.992657] vfio-pci 0000:00:06.0: reset done
[  439.993504] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:8195)
[  439.994734] vfio-pci 0000:00:06.0: resetting
[  440.212612] vfio-pci 0000:00:06.0: reset done
[  441.488527] vfio-pci 0000:00:06.0: resetting
[  441.702693] vfio-pci 0000:00:06.0: reset done
[  441.703533] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:8217)
[  441.704769] vfio-pci 0000:00:06.0: resetting
[  441.922603] vfio-pci 0000:00:06.0: reset done
[  442.998419] vfio-pci 0000:00:06.0: resetting
[  443.212768] vfio-pci 0000:00:06.0: reset done
[  443.213620] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:8240)
[  443.214861] vfio-pci 0000:00:06.0: resetting
[  443.432616] vfio-pci 0000:00:06.0: reset done
[  443.724393] vfio-pci 0000:00:06.0: resetting
[  443.942693] vfio-pci 0000:00:06.0: reset done
[  443.943547] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:8296)
[  443.944765] vfio-pci 0000:00:06.0: resetting
[  444.162665] vfio-pci 0000:00:06.0: reset done
[  491.891986] vfio-pci 0000:00:06.0: Removing from iommu group 0
[  492.901925] ena 0000:00:06.0: ENA device version: 0.10
[  492.902521] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  493.001770] ena 0000:00:06.0: Forcing large headers and decreasing maximum TX queue size to 512
[  493.002856] ena 0000:00:06.0: ENA Large LLQ is enabled
[  493.013388] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem 8004a000, mac addr 02:08:02:3e:e8:e7
[  493.017195] ena 0000:00:06.0 ens6: renamed from eth0
[  493.104077] zram_generator::config[8456]: zram0: system has too much memory (3806MB), limit is 800MB, ignoring.
[  783.133667] zram_generator::config[9233]: zram0: system has too much memory (3806MB), limit is 800MB, ignoring.
[  784.048631] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  784.049539] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1064.717424] vfio-pci 0000:00:06.0: Removing from iommu group 0
[ 1065.727957] ena 0000:00:06.0: ENA device version: 0.10
[ 1065.728538] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[ 1065.827767] ena 0000:00:06.0: Forcing large headers and decreasing maximum TX queue size to 512
[ 1065.828863] ena 0000:00:06.0: ENA Large LLQ is enabled
[ 1065.839333] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem 8004a000, mac addr 02:08:02:3e:e8:e7
[ 1065.843436] ena 0000:00:06.0 ens6: renamed from eth0
[ 1065.969171] zram_generator::config[9491]: zram0: system has too much memory (3806MB), limit is 800MB, ignoring.
[ 1093.788657] zram_generator::config[9676]: zram0: system has too much memory (3806MB), limit is 800MB, ignoring.
[ 1094.764673] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1094.765587] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1210.775187] vfio-pci 0000:00:06.0: resetting
[ 1210.990731] vfio-pci 0000:00:06.0: reset done
[ 1210.991578] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:9775)
[ 1210.992798] vfio-pci 0000:00:06.0: resetting
[ 1211.210650] vfio-pci 0000:00:06.0: reset done
[ 1211.484283] vfio-pci 0000:00:06.0: resetting
[ 1211.700741] vfio-pci 0000:00:06.0: reset done
[ 1211.701603] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:9795)
[ 1211.702850] vfio-pci 0000:00:06.0: resetting
[ 1211.920659] vfio-pci 0000:00:06.0: reset done
[ 1213.195569] vfio-pci 0000:00:06.0: resetting
[ 1213.410746] vfio-pci 0000:00:06.0: reset done
[ 1213.411587] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:9818)
[ 1213.412806] vfio-pci 0000:00:06.0: resetting
[ 1213.630681] vfio-pci 0000:00:06.0: reset done
[ 1214.905448] vfio-pci 0000:00:06.0: resetting
[ 1215.130846] vfio-pci 0000:00:06.0: reset done
[ 1215.131693] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:9841)
[ 1215.132911] vfio-pci 0000:00:06.0: resetting
[ 1215.350754] vfio-pci 0000:00:06.0: reset done

@github-actions

github-actions Bot commented Jun 20, 2026

Copy link
Copy Markdown

❌ Integration Tests Failed — Graviton (run)

Branch: 97/merge | Commit: c9ee553a

Test Results

  • tier1-dpdk-echo.xml: 6 tests, 0 failures
  • tier2-kernel-interop.xml: 4 tests, 0 failures
  • tier3-iperf-sends.xml: 1 tests, 1 failures
  • tier3-our-app-sends.xml: 1 tests, 1 failures

Application Logs (last 20 lines)

receiver-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.170:9000 (MAC: 02:0c:9b:21:42:af)
echo listening on 10.0.1.170:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-test-client.log

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
DPDK bind failed (Port init failed: Invalid port ID: 0), falling back to tokio
Backend: tokio
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.170:9000: 'arp-probe #1'
Test complete
[2026-06-20T01:48:17Z] INFO: ARP resolution succeeded (got response from peer)
[2026-06-20T01:48:17Z] INFO: Test: udp_send_receive
[2026-06-20T01:48:18Z] INFO: UDP send/receive succeeded
[2026-06-20T01:48:18Z] INFO: Test: echo_roundtrip
[2026-06-20T01:48:19Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-06-20T01:48:19Z] INFO: Test: payload_integrity
[2026-06-20T01:48:19Z] INFO: Response received, checking payload match...
[2026-06-20T01:48:19Z] INFO: Payload integrity verified (found in response)
[2026-06-20T01:48:19Z] INFO: JUnit XML written to: /tmp/test-results/tier2-kernel-interop.xml
[2026-06-20T01:48:19Z] INFO: Tier 2 sender tests complete. Results: /tmp/test-results/tier2-kernel-interop.xml

@github-actions

github-actions Bot commented Jun 20, 2026

Copy link
Copy Markdown

✅ Integration Tests Passed (Run 27856186184)

Branch: 97/merge | Commit: c9ee553a

Test Results

  • tier1-dpdk-echo.xml: 6 tests, 0 failures, skipped
  • tier2-kernel-interop.xml: 4 tests, 0 failures, skipped
  • tier3-iperf-sends.xml: 1 tests, 0 failures, skipped
  • tier3-our-app-sends.xml: 1 tests, 0 failures, skipped

Application Logs (last 20 lines)

receiver-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.198:9000 (MAC: 02:be:03:ad:bb:95)
echo listening on 10.0.1.198:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.190:9000 (MAC: 02:48:2a:5f:c8:41)
echo listening on 10.0.1.190:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-test-client.log

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
DPDK bind failed (Port init failed: Invalid port ID: 0), falling back to tokio
Backend: tokio
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.198:9000: 'arp-probe #1'
Test complete
[2026-06-20T01:48:40Z] INFO: ARP resolution succeeded (got response from peer)
[2026-06-20T01:48:40Z] INFO: Test: udp_send_receive
[2026-06-20T01:48:41Z] INFO: UDP send/receive succeeded
[2026-06-20T01:48:41Z] INFO: Test: echo_roundtrip
[2026-06-20T01:48:42Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-06-20T01:48:42Z] INFO: Test: payload_integrity
[2026-06-20T01:48:42Z] INFO: Response received, checking payload match...
[2026-06-20T01:48:42Z] INFO: Payload integrity verified (found in response)
[2026-06-20T01:48:42Z] INFO: JUnit XML written to: /tmp/test-results/tier2-kernel-interop.xml
[2026-06-20T01:48:42Z] INFO: Tier 2 sender tests complete. Results: /tmp/test-results/tier2-kernel-interop.xml

receiver-test-client-iperf.log

[2026-06-20T01:55:40Z] INFO: iperf-sends: sent 10 packets, received 10 responses
[2026-06-20T01:55:40Z] INFO: iperf-sends: PASS (sent >= 5 packets)
[2026-06-20T01:55:40Z] INFO: JUnit XML written to: /tmp/test-results/tier3-iperf-sends.xml
[2026-06-20T01:55:40Z] INFO: iperf-sends test complete

sender-test-client-iperf.log

Received 30 bytes from 10.0.1.198:9000: 'dpdk-to-kernel-test-payload #3'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #4'
Received 30 bytes from 10.0.1.198:9000: 'dpdk-to-kernel-test-payload #4'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #5'
Received 30 bytes from 10.0.1.198:9000: 'dpdk-to-kernel-test-payload #5'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #6'
Received 30 bytes from 10.0.1.198:9000: 'dpdk-to-kernel-test-payload #6'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #7'
Received 30 bytes from 10.0.1.198:9000: 'dpdk-to-kernel-test-payload #7'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #8'
Received 30 bytes from 10.0.1.198:9000: 'dpdk-to-kernel-test-payload #8'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #9'
Received 30 bytes from 10.0.1.198:9000: 'dpdk-to-kernel-test-payload #9'
Sent 31 bytes: 'dpdk-to-kernel-test-payload #10'
Received 31 bytes from 10.0.1.198:9000: 'dpdk-to-kernel-test-payload #10'
Test complete
[2026-06-20T01:54:41Z] INFO: our-app-sends: sent 10 packets, received 10 responses
[2026-06-20T01:54:41Z] INFO: our-app-sends: PASS (sent >= 5 packets)
[2026-06-20T01:54:41Z] INFO: JUnit XML written to: /tmp/test-results/tier3-our-app-sends.xml
[2026-06-20T01:54:41Z] INFO: our-app-sends test complete
Full Application Logs (last 200 lines each)

receiver-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.198:9000 (MAC: 02:be:03:ad:bb:95)
echo listening on 10.0.1.198:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.190:9000 (MAC: 02:48:2a:5f:c8:41)
echo listening on 10.0.1.190:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-test-client.log

[2026-06-20T01:44:10Z] INFO: Test: arp_resolution
UDP Test Client
Target: 10.0.1.198:9000
Bind address: 10.0.1.190:0
Message: 'arp-probe'
Count: 1
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.190:32768 (MAC: 02:48:2a:5f:c8:41)
Backend: dpdk
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.198:9000: 'arp-probe #1'
Test complete
[2026-06-20T01:44:11Z] INFO: ARP resolution succeeded (got response from peer)
[2026-06-20T01:44:11Z] INFO: Test: udp_send_receive
[2026-06-20T01:44:12Z] INFO: UDP send/receive succeeded
[2026-06-20T01:44:12Z] INFO: Test: echo_roundtrip
[2026-06-20T01:44:14Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-06-20T01:44:14Z] INFO: Test: payload_integrity
[2026-06-20T01:44:14Z] INFO: Response received, checking payload match...
[2026-06-20T01:44:14Z] INFO: Payload integrity verified (found in response)
[2026-06-20T01:44:14Z] INFO: Test: jumbo_diagnostics
[2026-06-20T01:44:14Z] INFO: === JUMBO FRAME DIAGNOSTICS ===
[2026-06-20T01:44:14Z] INFO: Interface MTU:
  9001
  65536
[2026-06-20T01:44:15Z] INFO:   ens5: MTU=9001
[2026-06-20T01:44:15Z] INFO:   lo: MTU=65536
[2026-06-20T01:44:15Z] INFO: Routing table MTU column:
Iface	Destination	Gateway 	Flags	RefCnt	Use	Metric	Mask		MTU	Window	IRTT                                                       
ens5	00000000	0101000A	0003	0	0	512	00000000	0	0	0                                                                             
ens5	0200000A	0101000A	0007	0	0	512	FFFFFFFF	0	0	0                                                                             
ens5	0001000A	00000000	0001	0	0	512	00FFFFFF	0	0	0                                                                             
ens5	0101000A	00000000	0005	0	0	512	FFFFFFFF	0	0	0                                                                             
[2026-06-20T01:44:15Z] INFO: DPDK port config (from echo server log):
[2026-06-20T01:44:15Z] INFO:   (no MTU info in echo log)
[2026-06-20T01:44:15Z] INFO: === END JUMBO DIAGNOSTICS ===
[2026-06-20T01:44:15Z] INFO: Test: jumbo_echo_8000
[2026-06-20T01:44:16Z] INFO: Jumbo output: UDP Test Client
Target: 10.0.1.198:9000
Bind address: 10.0.1.190:0
Payload size: 8000 bytes
Count: 3
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.190:32768 (MAC: 02:48:2a:5f:c8:41)
Backend: dpdk
Sending packets...
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.198:9000 (expected 8000, OK)
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.198:9000 (expected 8000, OK)
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.198:9000 (expected 8000, OK)
Test complete
[2026-06-20T01:44:16Z] INFO: Jumbo frame echo succeeded: 3/3 responses with correct size
[2026-06-20T01:44:16Z] INFO: JUnit XML written to: /tmp/test-results/tier1-dpdk-echo.xml
[2026-06-20T01:44:16Z] INFO: Tier 1 sender tests complete. Results: /tmp/test-results/tier1-dpdk-echo.xml
[2026-06-20T01:48:40Z] INFO: Test: arp_resolution
UDP Test Client
Target: 10.0.1.198:9000
Bind address: 0.0.0.0:0
Message: 'arp-probe'
Count: 1
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
DPDK bind failed (Port init failed: Invalid port ID: 0), falling back to tokio
Backend: tokio
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.198:9000: 'arp-probe #1'
Test complete
[2026-06-20T01:48:40Z] INFO: ARP resolution succeeded (got response from peer)
[2026-06-20T01:48:40Z] INFO: Test: udp_send_receive
[2026-06-20T01:48:41Z] INFO: UDP send/receive succeeded
[2026-06-20T01:48:41Z] INFO: Test: echo_roundtrip
[2026-06-20T01:48:42Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-06-20T01:48:42Z] INFO: Test: payload_integrity
[2026-06-20T01:48:42Z] INFO: Response received, checking payload match...
[2026-06-20T01:48:42Z] INFO: Payload integrity verified (found in response)
[2026-06-20T01:48:42Z] INFO: JUnit XML written to: /tmp/test-results/tier2-kernel-interop.xml
[2026-06-20T01:48:42Z] INFO: Tier 2 sender tests complete. Results: /tmp/test-results/tier2-kernel-interop.xml

receiver-test-client-iperf.log

[2026-06-20T01:55:40Z] INFO: iperf-sends: sent 10 packets, received 10 responses
[2026-06-20T01:55:40Z] INFO: iperf-sends: PASS (sent >= 5 packets)
[2026-06-20T01:55:40Z] INFO: JUnit XML written to: /tmp/test-results/tier3-iperf-sends.xml
[2026-06-20T01:55:40Z] INFO: iperf-sends test complete

sender-test-client-iperf.log

[2026-06-20T01:54:39Z] INFO: Pre-flight: checking DPDK state and ARP cache...
[2026-06-20T01:54:39Z] INFO: Local IP: 10.0.1.190, Peer IP: 10.0.1.198, Port: 9000
[2026-06-20T01:54:39Z] INFO: /proc/net/arp contents:
IP address       HW type     Flags       HW address            Mask     Device
10.0.1.21        0x1         0x2         02:b3:be:2b:62:0f     *        ens5
10.0.1.198       0x1         0x2         02:be:03:ad:bb:95     *        ens5
10.0.1.133       0x1         0x2         02:bd:05:4b:a6:4d     *        ens5
10.0.1.1         0x1         0x2         02:38:88:da:02:87     *        ens5
10.0.1.190       0x1         0x2         02:48:2a:5f:c8:41     *        ens5
[2026-06-20T01:54:39Z] INFO: DPDK runtime state:
No /var/run/dpdk/ directory
[2026-06-20T01:54:39Z] INFO: vfio-pci bindings:
0000:00:06.0
bind
module
new_id
remove_id
uevent
unbind
[2026-06-20T01:54:39Z] INFO: Test binary: /opt/dpdk-stdlib/target/release/test-client
-rwxr-xr-x. 2 root root 1933272 Jun 20 01:41 /opt/dpdk-stdlib/target/release/test-client
[2026-06-20T01:54:39Z] INFO: Launching test-client: /opt/dpdk-stdlib/target/release/test-client --target 10.0.1.198 --port 9000 --bind-ip 10.0.1.190 --count 10 --delay 200
[2026-06-20T01:54:41Z] INFO: Test client output: UDP Test Client
Target: 10.0.1.198:9000
Bind address: 10.0.1.190:0
Message: 'dpdk-to-kernel-test-payload'
Count: 10
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.190:32768 (MAC: 02:48:2a:5f:c8:41)
Backend: dpdk
Sending packets...
Sent 30 bytes: 'dpdk-to-kernel-test-payload #1'
Received 30 bytes from 10.0.1.198:9000: 'dpdk-to-kernel-test-payload #1'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #2'
Received 30 bytes from 10.0.1.198:9000: 'dpdk-to-kernel-test-payload #2'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #3'
Received 30 bytes from 10.0.1.198:9000: 'dpdk-to-kernel-test-payload #3'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #4'
Received 30 bytes from 10.0.1.198:9000: 'dpdk-to-kernel-test-payload #4'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #5'
Received 30 bytes from 10.0.1.198:9000: 'dpdk-to-kernel-test-payload #5'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #6'
Received 30 bytes from 10.0.1.198:9000: 'dpdk-to-kernel-test-payload #6'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #7'
Received 30 bytes from 10.0.1.198:9000: 'dpdk-to-kernel-test-payload #7'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #8'
Received 30 bytes from 10.0.1.198:9000: 'dpdk-to-kernel-test-payload #8'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #9'
Received 30 bytes from 10.0.1.198:9000: 'dpdk-to-kernel-test-payload #9'
Sent 31 bytes: 'dpdk-to-kernel-test-payload #10'
Received 31 bytes from 10.0.1.198:9000: 'dpdk-to-kernel-test-payload #10'
Test complete
[2026-06-20T01:54:41Z] INFO: our-app-sends: sent 10 packets, received 10 responses
[2026-06-20T01:54:41Z] INFO: our-app-sends: PASS (sent >= 5 packets)
[2026-06-20T01:54:41Z] INFO: JUnit XML written to: /tmp/test-results/tier3-our-app-sends.xml
[2026-06-20T01:54:41Z] INFO: our-app-sends test complete
⚠️ SSM Command Failures (receiver-ssm-failure.log) 
=== Polling timeout after 30s ===
Status: InProgress
Instance: i-01a623c3bb29bb016 (receiver)
Command ID: 97f212b1-9264-47b2-83c2-92a0755bd58d

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-01a623c3bb29bb016 (receiver)
Command ID: e7418487-5955-4300-b376-6294d299f1a8

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-01a623c3bb29bb016 (receiver)
Command ID: de1d72f2-c855-4154-8e98-44482c0b58d1

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-01a623c3bb29bb016 (receiver)
Command ID: ab6da0ee-dbd3-4d57-9ea7-1e01f19c037f

=== STDOUT ===


=== STDERR ===
⚠️ SSM Command Failures (sender-ssm-failure.log) 
=== Polling timeout 120s (SSM timeout: 90s, command: cd /opt/dpdk-stdlib && bash scripts/integration-tests/configure-eni.sh --action unbind && bash scripts/integration-tests/configure-eni.sh --action assign-ip --ip 10.0.1.190) ===
Status: InProgress
Instance: i-0002c2d63f412360d (sender)
Command ID: 05515ab0-c1b4-48b8-b1f0-53dca5307122

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-0002c2d63f412360d (sender)
Command ID: ee988bea-d33e-45c5-b130-8f22d525bfba

=== STDOUT ===


=== STDERR ===
Network & PCI State

receiver-network-interfaces.log

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:12:94:c3:cb:99 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-0ecebff9d651adb52
    altname device-number-0.0
    inet 10.0.1.253/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 2159sec preferred_lft 2159sec
    inet6 fe80::12:94ff:fec3:cb99/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

sender-network-interfaces.log

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:d3:a1:9f:0d:8d brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-0aff32715c13e8efe
    altname device-number-0.0
    inet 10.0.1.66/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 2183sec preferred_lft 2183sec
    inet6 fe80::d3:a1ff:fe9f:d8d/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

receiver-networking-diag-baseline.txt

=== NETWORKING DIAGNOSTICS ===
timestamp: 2026-06-20T01:43:31Z
hostname: ip-10-0-1-253.ec2.internal
kernel: 6.18.33-63.124.amzn2023.x86_64

=== DPDK PORT STATUS ===

Network devices using DPDK-compatible driver
============================================
0000:00:06.0 'Elastic Network Adapter (ENA) ec20' drv=vfio-pci unused=ena

Network devices using kernel driver
===================================
0000:00:05.0 'Elastic Network Adapter (ENA) ec20' if=ens5 drv=ena unused=vfio-pci *Active*

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'DMA' devices detected
=========================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

No 'Regex' devices detected
===========================

=== IP ADDRESSES ===
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:12:94:c3:cb:99 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-0ecebff9d651adb52
    altname device-number-0.0
    inet 10.0.1.253/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 3183sec preferred_lft 3183sec
    inet6 fe80::12:94ff:fec3:cb99/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

=== ARP TABLE ===
10.0.1.1 dev ens5 lladdr 02:38:88:da:02:87 REACHABLE 
10.0.1.133 dev ens5 lladdr 02:bd:05:4b:a6:4d STALE 
10.0.1.21 dev ens5 lladdr 02:b3:be:2b:62:0f REACHABLE 

=== ROUTE TABLE ===
default via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.253 metric 512 
10.0.0.2 via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.253 metric 512 
10.0.1.0/24 dev ens5 proto kernel scope link src 10.0.1.253 metric 512 
10.0.1.1 dev ens5 proto dhcp scope link src 10.0.1.253 metric 512 

=== IMDS: ENI INFORMATION ===
ENI MACs found: 02:12:94:c3:cb:99/ 02:be:03:ad:bb:95/ 

--- ENI: 02:12:94:c3:cb:99/ ---
  device-number: 0
  local-ipv4s: 10.0.1.253
  subnet-id: subnet-08a1ea097b3ef8fb9
  subnet-cidr: 10.0.1.0/24

--- ENI: 02:be:03:ad:bb:95/ ---
  device-number: 1
  local-ipv4s: 10.0.1.198
  subnet-id: subnet-08a1ea097b3ef8fb9
  subnet-cidr: 10.0.1.0/24


=== GATEWAY ARP TEST ===
Gateway IP: 10.0.1.1
Gateway ARP entry:
10.0.1.1 dev ens5 lladdr 02:38:88:da:02:87 REACHABLE 

arping result:
ARPING 10.0.1.1 from 10.0.1.253 ens5
Unicast reply from 10.0.1.1 [02:38:88:DA:02:87]  0.533ms
Sent 1 probes (1 broadcast(s))
Received 1 response(s)

=== HUGEPAGE STATUS ===
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:     51200 kB
HugePages_Total:    1024
HugePages_Free:     1024
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         2097152 kB

=== VFIO STATUS ===
total 0
drwxr-xr-x.  2 root root       80 Jun 20 01:43 .
drwxr-xr-x. 14 root root     3100 Jun 20 01:42 ..
crw-------.  1 root root 243,   0 Jun 20 01:43 noiommu-0
crw-rw-rw-.  1 root root  10, 196 Jun 20 01:36 vfio

noiommu mode:
Y

=== DPDK SHARED MEMORY ===
no /var/run/dpdk/ directory (clean state)

=== DPDK-RELATED DMESG (last 30 lines) ===
[    0.052894] printk: legacy console [ttyS0] enabled
[    0.053943] x2apic enabled
[    0.058195] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto
[    0.058306] x86/fpu: Enabled xstate features 0x2ff, context size is 2568 bytes, using 'compacted' format.
[    0.068042] audit: type=2000 audit(1781919389.926:1): state=initialized audit_enabled=0 res=1
[    0.087638] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
[    0.158243] ACPI: Interpreter enabled
[    0.158243] ACPI: Enabled 2 GPEs in block 00 to 0F
[    0.167249] pci 0000:00:05.0: enabling Extended Tags
[    0.226606] SGI XFS with ACLs, security attributes, quota, no debug enabled
[    0.250112] ACPI: \_SB_.LNKD: Enabled at IRQ 11
[    0.254266] IPI shorthand broadcast: enabled
[    2.185471] systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
[    2.222370] systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
[    2.289391] VFIO - User Level meta-driver version: 0.3
[    2.648454] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.16.1g
[    2.663454] ena 0000:00:05.0: ENA device version: 0.10
[    2.664161] ena 0000:00:05.0: ENA controller version: 0.0.1 implementation version 1
[    2.764719] ena 0000:00:05.0: ENA Large LLQ is disabled
[    2.777036] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem c0500000, mac addr 02:12:94:c3:cb:99
[    2.795202] ena 0000:00:05.0 ens5: renamed from eth0
[  371.705602] pci 0000:00:06.0: enabling Extended Tags
[  371.709505] ena 0000:00:06.0: enabling device (0000 -> 0002)
[  371.721706] ena 0000:00:06.0: ENA device version: 0.10
[  371.722463] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  371.821684] ena 0000:00:06.0: ENA Large LLQ is disabled
[  371.833549] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:be:03:ad:bb:95
[  371.840467] ena 0000:00:06.0 ens6: renamed from eth0
[  409.158706] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  409.160143] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device

=== DPDK-RELATED PROCESSES ===
no DPDK processes running

=== END DIAGNOSTICS ===

sender-networking-diag-baseline.txt

=== NETWORKING DIAGNOSTICS ===
timestamp: 2026-06-20T01:43:24Z
hostname: ip-10-0-1-66.ec2.internal
kernel: 6.18.33-63.124.amzn2023.x86_64

=== DPDK PORT STATUS ===

Network devices using DPDK-compatible driver
============================================
0000:00:06.0 'Elastic Network Adapter (ENA) ec20' drv=vfio-pci unused=ena

Network devices using kernel driver
===================================
0000:00:05.0 'Elastic Network Adapter (ENA) ec20' if=ens5 drv=ena unused=vfio-pci *Active*

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'DMA' devices detected
=========================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

No 'Regex' devices detected
===========================

=== IP ADDRESSES ===
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:d3:a1:9f:0d:8d brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-0aff32715c13e8efe
    altname device-number-0.0
    inet 10.0.1.66/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 3193sec preferred_lft 3193sec
    inet6 fe80::d3:a1ff:fe9f:d8d/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

=== ARP TABLE ===
10.0.1.21 dev ens5 lladdr 02:b3:be:2b:62:0f REACHABLE 
10.0.1.133 dev ens5 lladdr 02:bd:05:4b:a6:4d STALE 
10.0.1.1 dev ens5 lladdr 02:38:88:da:02:87 REACHABLE 

=== ROUTE TABLE ===
default via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.66 metric 512 
10.0.0.2 via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.66 metric 512 
10.0.1.0/24 dev ens5 proto kernel scope link src 10.0.1.66 metric 512 
10.0.1.1 dev ens5 proto dhcp scope link src 10.0.1.66 metric 512 

=== IMDS: ENI INFORMATION ===
ENI MACs found: 02:48:2a:5f:c8:41/ 02:d3:a1:9f:0d:8d/ 

--- ENI: 02:48:2a:5f:c8:41/ ---
  device-number: 1
  local-ipv4s: 10.0.1.190
  subnet-id: subnet-08a1ea097b3ef8fb9
  subnet-cidr: 10.0.1.0/24

--- ENI: 02:d3:a1:9f:0d:8d/ ---
  device-number: 0
  local-ipv4s: 10.0.1.66
  subnet-id: subnet-08a1ea097b3ef8fb9
  subnet-cidr: 10.0.1.0/24


=== GATEWAY ARP TEST ===
Gateway IP: 10.0.1.1
Gateway ARP entry:
10.0.1.1 dev ens5 lladdr 02:38:88:da:02:87 REACHABLE 

arping result:
ARPING 10.0.1.1 from 10.0.1.66 ens5
Unicast reply from 10.0.1.1 [02:38:88:DA:02:87]  0.540ms
Sent 1 probes (1 broadcast(s))
Received 1 response(s)

=== HUGEPAGE STATUS ===
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:     51200 kB
HugePages_Total:    1024
HugePages_Free:     1024
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         2097152 kB

=== VFIO STATUS ===
total 0
drwxr-xr-x.  2 root root       80 Jun 20 01:43 .
drwxr-xr-x. 14 root root     3100 Jun 20 01:42 ..
crw-------.  1 root root 243,   0 Jun 20 01:43 noiommu-0
crw-rw-rw-.  1 root root  10, 196 Jun 20 01:36 vfio

noiommu mode:
Y

=== DPDK SHARED MEMORY ===
no /var/run/dpdk/ directory (clean state)

=== DPDK-RELATED DMESG (last 30 lines) ===
[    0.051985] printk: legacy console [ttyS0] enabled
[    0.053019] x2apic enabled
[    0.057249] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto
[    0.057358] x86/fpu: Enabled xstate features 0x2ff, context size is 2568 bytes, using 'compacted' format.
[    0.067083] audit: type=2000 audit(1781919391.303:1): state=initialized audit_enabled=0 res=1
[    0.086688] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
[    0.157064] ACPI: Interpreter enabled
[    0.157064] ACPI: Enabled 2 GPEs in block 00 to 0F
[    0.165985] pci 0000:00:05.0: enabling Extended Tags
[    0.228428] SGI XFS with ACLs, security attributes, quota, no debug enabled
[    0.237018] ACPI: \_SB_.LNKD: Enabled at IRQ 11
[    0.244919] IPI shorthand broadcast: enabled
[    3.546420] systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
[    3.582823] systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
[    3.641324] VFIO - User Level meta-driver version: 0.3
[    4.043382] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.16.1g
[    4.054652] ena 0000:00:05.0: ENA device version: 0.10
[    4.055427] ena 0000:00:05.0: ENA controller version: 0.0.1 implementation version 1
[    4.154722] ena 0000:00:05.0: ENA Large LLQ is disabled
[    4.167892] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem c0500000, mac addr 02:d3:a1:9f:0d:8d
[    4.228213] ena 0000:00:05.0 ens5: renamed from eth0
[  371.281778] pci 0000:00:06.0: enabling Extended Tags
[  371.285557] ena 0000:00:06.0: enabling device (0000 -> 0002)
[  371.298242] ena 0000:00:06.0: ENA device version: 0.10
[  371.299030] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  371.400200] ena 0000:00:06.0: ENA Large LLQ is disabled
[  371.412410] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:48:2a:5f:c8:41
[  371.419981] ena 0000:00:06.0 ens6: renamed from eth0
[  401.355342] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  401.356757] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device

=== DPDK-RELATED PROCESSES ===
no DPDK processes running

=== END DIAGNOSTICS ===
⚠️ Crash Diagnostics

receiver-dmesg-crashes.log

[    0.067436] pid_max: default: 32768 minimum: 301
[    0.171122] iommu: Default domain type: Translated
[    0.176996] NetLabel:  unlabeled traffic allowed by default
[    0.202591] PCI: CLS 0 bytes, default 64
[    0.269024] nvme nvme0: 2/0/0 default/read/poll queues
[    0.448996] systemd[1]: systemd 252.23-11.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    0.449117] systemd[1]: No hostname configured, using default hostname.
[    0.511440] systemd[1]: Queued start job for default target initrd.target.
[    1.684628] systemd[1]: systemd 252.23-11.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[  444.824285] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:8192)
[  714.662001] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:8628)
[ 1312.986988] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:9788)

sender-dmesg-crashes.log

[    0.066502] pid_max: default: 32768 minimum: 301
[    0.169804] iommu: Default domain type: Translated
[    0.175627] NetLabel:  unlabeled traffic allowed by default
[    0.202676] PCI: CLS 0 bytes, default 64
[    0.271132] nvme nvme0: 2/0/0 default/read/poll queues
[    0.429701] systemd[1]: systemd 252.23-11.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    0.429826] systemd[1]: No hostname configured, using default hostname.
[    0.515686] systemd[1]: Queued start job for default target initrd.target.
[    2.978615] systemd[1]: systemd 252.23-11.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[ 1131.504461] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:9479)
Kernel Console (dmesg)

receiver-console-output.log (PCI/driver events only)

[  444.822722] vfio-pci 0000:00:06.0: reset done
[  444.824285] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:8192)
[  444.825726] vfio-pci 0000:00:06.0: resetting
[  445.042613] vfio-pci 0000:00:06.0: reset done
[  523.450238] vfio-pci 0000:00:06.0: Removing from iommu group 0
[  524.460427] ena 0000:00:06.0: ENA device version: 0.10
[  524.461205] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  524.560407] ena 0000:00:06.0: ENA Large LLQ is disabled
[  524.572333] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:be:03:ad:bb:95
[  524.580626] ena 0000:00:06.0 ens6: renamed from eth0
[  544.014394] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  544.015643] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[  714.441683] vfio-pci 0000:00:06.0: resetting
[  714.660439] vfio-pci 0000:00:06.0: reset done
[  714.662001] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:8628)
[  714.663461] vfio-pci 0000:00:06.0: resetting
[  714.880233] vfio-pci 0000:00:06.0: reset done
[  787.753450] vfio-pci 0000:00:06.0: Removing from iommu group 0
[  788.766231] ena 0000:00:06.0: ENA device version: 0.10
[  788.767011] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  788.865702] ena 0000:00:06.0: ENA Large LLQ is disabled
[  788.878496] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:be:03:ad:bb:95
[  788.889650] ena 0000:00:06.0 ens6: renamed from eth0 (while UP)
[ 1229.817997] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1229.819437] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1312.762072] vfio-pci 0000:00:06.0: resetting
[ 1312.985412] vfio-pci 0000:00:06.0: reset done
[ 1312.986988] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:9788)
[ 1312.988441] vfio-pci 0000:00:06.0: resetting
[ 1313.205309] vfio-pci 0000:00:06.0: reset done

sender-console-output.log (PCI/driver events only)

[ 1131.505893] vfio-pci 0000:00:06.0: resetting
[ 1131.722780] vfio-pci 0000:00:06.0: reset done
[ 1191.847301] vfio-pci 0000:00:06.0: Removing from iommu group 0
[ 1192.860348] ena 0000:00:06.0: ENA device version: 0.10
[ 1192.861126] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[ 1192.960554] ena 0000:00:06.0: ENA Large LLQ is disabled
[ 1192.972561] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:48:2a:5f:c8:41
[ 1192.983989] ena 0000:00:06.0 ens6: renamed from eth0 (while UP)
[ 1221.856959] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1221.858335] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1327.243698] vfio-pci 0000:00:06.0: resetting
[ 1327.460583] vfio-pci 0000:00:06.0: reset done
[ 1327.462146] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:9919)
[ 1327.463734] vfio-pci 0000:00:06.0: resetting
[ 1327.680448] vfio-pci 0000:00:06.0: reset done
[ 1327.967552] vfio-pci 0000:00:06.0: resetting
[ 1328.190443] vfio-pci 0000:00:06.0: reset done
[ 1328.192009] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:9939)
[ 1328.193562] vfio-pci 0000:00:06.0: resetting
[ 1328.410403] vfio-pci 0000:00:06.0: reset done
[ 1329.699890] vfio-pci 0000:00:06.0: resetting
[ 1329.920449] vfio-pci 0000:00:06.0: reset done
[ 1329.921993] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:9962)
[ 1329.923604] vfio-pci 0000:00:06.0: resetting
[ 1330.140401] vfio-pci 0000:00:06.0: reset done
[ 1331.429553] vfio-pci 0000:00:06.0: resetting
[ 1331.640403] vfio-pci 0000:00:06.0: reset done
[ 1331.641955] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:9985)
[ 1331.643509] vfio-pci 0000:00:06.0: resetting
[ 1331.860389] vfio-pci 0000:00:06.0: reset done

@gspivey

gspivey commented Jun 20, 2026

Copy link
Copy Markdown
Owner Author

CI Summary

All actionable CI checks are green:

  • ✅ build — passed
  • ✅ validate-cdk — passed
  • ✅ synthetic-perf — passed
  • ✅ Integration Tests (x86) — 12/12 tests, 0 failures
  • ⚠️ Integration Tests (Graviton) — tier3 transient failure (sender DPDK bind race, unrelated to this scripts-only PR)

Blocker: workflow scope

Cannot proceed to performance test phase because .github/workflows/perf-tests-tcp.yml could not be pushed — the PAT lacks workflow scope. Once the PAT is updated with workflow permission, push the workflow file and trigger with:

gh workflow run perf-tests-tcp.yml --ref agent/tcp-perf-ci

The runner script (scripts/run-perf-tests-tcp.sh) is pushed and validated.

@github-actions

github-actions Bot commented Jun 25, 2026

Copy link
Copy Markdown

[CI] Stage: Deploy

Infrastructure ready.

  • Sender: i-006a4561bd109ab72 (DPDK ENI: 10.0.1.177)
  • Receiver: i-00f2cd3ad2d9fe489 (DPDK ENI: 10.0.1.150)
  • Both instances SSM-ready.

@github-actions

github-actions Bot commented Jun 25, 2026

Copy link
Copy Markdown

[CI] Stage: Summary

All tests PASSED.

ARP seeding: kernel /proc/net/arp (automatic)

  • tier1-dpdk-echo: 6 tests, 0 failures
  • tier2-kernel-interop: 4 tests, 0 failures
  • tier3-iperf-interop: 1 tests, 0 failures
  • tier3-iperf-interop: 1 tests, 0 failures

@github-actions

github-actions Bot commented Jun 25, 2026

Copy link
Copy Markdown

✅ Integration Tests Passed — Graviton (run)

Branch: 97/merge | Commit: c9ee553a

Test Results

  • tier1-dpdk-echo.xml: 6 tests, 0 failures
  • tier2-kernel-interop.xml: 4 tests, 0 failures
  • tier3-iperf-sends.xml: 1 tests, 0 failures
  • tier3-our-app-sends.xml: 1 tests, 0 failures

Application Logs (last 20 lines)

receiver-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.150:9000 (MAC: 02:86:7c:c9:fc:bb)
echo listening on 10.0.1.150:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.177:9000 (MAC: 02:b9:c4:02:ce:81)
echo listening on 10.0.1.177:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-test-client.log

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
DPDK bind failed (Port init failed: Invalid port ID: 0), falling back to tokio
Backend: tokio
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.150:9000: 'arp-probe #1'
Test complete
[2026-06-25T12:03:24Z] INFO: ARP resolution succeeded (got response from peer)
[2026-06-25T12:03:24Z] INFO: Test: udp_send_receive
[2026-06-25T12:03:25Z] INFO: UDP send/receive succeeded
[2026-06-25T12:03:25Z] INFO: Test: echo_roundtrip
[2026-06-25T12:03:26Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-06-25T12:03:26Z] INFO: Test: payload_integrity
[2026-06-25T12:03:26Z] INFO: Response received, checking payload match...
[2026-06-25T12:03:26Z] INFO: Payload integrity verified (found in response)
[2026-06-25T12:03:26Z] INFO: JUnit XML written to: /tmp/test-results/tier2-kernel-interop.xml
[2026-06-25T12:03:26Z] INFO: Tier 2 sender tests complete. Results: /tmp/test-results/tier2-kernel-interop.xml

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@gspivey