TCP over DPDK: smoke tiers + sustained-exchange engine fix + CI summary reporting#111
Open
gspivey wants to merge 3 commits into
Open
TCP over DPDK: smoke tiers + sustained-exchange engine fix + CI summary reporting#111gspivey wants to merge 3 commits into
gspivey wants to merge 3 commits into
Conversation
connect->echo->close smoke coverage for the DPDK TCP stack, modeled on the existing UDP Tier 1/2 integration tiers (two-host, JUnit, SSM-driven): - apps/tcp-kernel-client: pure std::net reference client (the 'not our TCP stack' peer). Prints TCP_KERNEL_OK/FAIL, exit-coded. - tier1-tcp-echo.sh: DPDK<->DPDK, sync (tcp-echo) + async (tokio-tcp-echo) servers via --server-binary; bidir echo + 20x multi round-trip. - tier2-tcp-echo.sh: kernel client -> DPDK tcp-echo server — the standard-stack interop that exposed the codec padding bug (#110); mirrors the passing UDP Tier-2 kernel->DPDK direction. - run-integration-tests.sh: run_tier1_tcp / _async / run_tier2_tcp registered in the default CI set (+ --tier tcp1/tcp1a/tcp2); CLEANUP_CMD extended to kill the TCP binaries so a stale process can't hold the DPDK EAL primary lock; --tier validation also accepts 4 (was dispatched but rejected). - CDK: DPDK<->DPDK allTcp self-ingress (else Tier-1 TCP handshakes are dropped). Verified locally: bash -n, cargo build, markers match binary output. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Synthetic Performance Results — Graviton (run)Commit: ✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01) Synthetic UDP Performance ResultsMeasures framework overhead: sync IPv4 Baseline
IPv6
IPv6 vs IPv4 Comparison (sync path)
IPv4 avg sync/async ratio: 0.9x, worst: 1.0x | IPv6 vs IPv4 worst ratio: 1.31x (OK)
|
Synthetic Performance Results (run)Commit: ✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01) Synthetic UDP Performance ResultsMeasures framework overhead: sync IPv4 Baseline
IPv6
IPv6 vs IPv4 Comparison (sync path)
IPv4 avg sync/async ratio: 0.9x, worst: 1.1x | IPv6 vs IPv4 worst ratio: 1.32x (OK)
|
[CI] Stage: DeployInfrastructure ready.
|
[CI] Stage: DeployInfrastructure ready.
|
[CI] Stage: SummarySome tests FAILED (exit code: 1). ARP seeding: kernel /proc/net/arp (automatic)
|
1 similar comment
[CI] Stage: SummarySome tests FAILED (exit code: 1). ARP seeding: kernel /proc/net/arp (automatic)
|
❌ Integration Tests Failed (Run 28580887516)Branch: Test Results
Application Logs (last 20 lines)sender-echo-server.log sender-test-client.log sender-test-client-iperf.log Full Application Logs (last 200 lines each)sender-echo-server.logsender-test-client.logsender-test-client-iperf.log
|
❌ Integration Tests Failed — Graviton (run)Branch: Test Results
Application Logs (last 20 lines)sender-echo-server.log sender-test-client.log |
This was referenced Jul 2, 2026
The PR summary comment previously showed only per-tier 'N tests, M failures' — to see WHICH testcase failed, WHY, or whether it was a real code issue vs an SSM/ENI infra flake, you had to download the JUnit artifact or dig through collapsed log <details>. generate_markdown_summary now emits a self-contained test-results/summary.md (posted via the existing post_pr_comment) with: - a verdict separating REAL failures from INFRA flakes, - a per-tier per-testcase table (result + time + reason), - a Real-failures section with detail excerpts, - an Infra-flakes section (SSM/ENI setup, labeled task #10). Classification uses the JUnit failure type: real tier failures are type=AssertionError (junit_add_failure); synthetic setup failures are type=ExecutionError (generate_failure_xml), with a keyword fallback. Report-only — exit-code behavior unchanged. Scripts-only, no workflow change. Verified locally against sample pass/real-fail/infra-fail XMLs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…xchange) handle_established's ACK path pruned the retransmit queue but never drained the acknowledged bytes from the front of send_buf, nor cleared has_unacked_data. After the first send+ACK, send_buf still held the stale acked bytes while already_sent_offset (derived from the now-empty retransmit queue) reset to 0 — so the next small write was misclassified as unsent behind Nagle (has_unacked_data==true, len<MSS) and never transmitted. The connection stalled after the first exchange. This is the sustained-exchange (bidir_multi) hang the TCP smoke tiers isolated on real hardware (#111 Graviton: single round-trip PASS, 20-on-one-connection HANG). Fix: on cumulative ACK, drain acked bytes off the front of send_buf, rebase the surviving retransmit-entry offsets, and clear has_unacked_data when nothing is in flight (snd_una==snd_nxt). Offline regression tests/loopback_stall_repro.rs wires client<->server engines through in-memory queues and drives 20 sequential 64B echoes on one connection: FAILS before (stalls at exchange 2), PASSES after. 221 unit + all property/ integration tests green. Follow-up: the same send_buf drain is missing in handle_fin_wait_1 / handle_close_wait (teardown states — a send_buf leak, not a stall). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
gspivey
changed the title
Synthetic Performance Results — Graviton (run)Commit: ✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01) Synthetic UDP Performance ResultsMeasures framework overhead: sync IPv4 Baseline
IPv6
IPv6 vs IPv4 Comparison (sync path)
IPv4 avg sync/async ratio: 0.9x, worst: 1.0x | IPv6 vs IPv4 worst ratio: 1.30x (OK)
|
Synthetic Performance Results (run)Commit: ✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01) Synthetic UDP Performance ResultsMeasures framework overhead: sync IPv4 Baseline
IPv6
IPv6 vs IPv4 Comparison (sync path)
IPv4 avg sync/async ratio: 0.9x, worst: 1.0x | IPv6 vs IPv4 worst ratio: 1.31x (OK)
|
[CI] Stage: DeployInfrastructure ready.
|
[CI] Stage: DeployInfrastructure ready.
|
Integration Tests — ❌ 1 real failure ·
|
| Tier | Test | Result | Time | Reason |
|---|---|---|---|---|
| tier1-dpdk-echo | arp_resolution | ✅ pass | 0.965s | |
| tier1-dpdk-echo | udp_send_receive | ✅ pass | 1.704s | |
| tier1-dpdk-echo | echo_roundtrip | ✅ pass | 1.493s | |
| tier1-dpdk-echo | payload_integrity | ✅ pass | 0.705s | |
| tier1-dpdk-echo | jumbo_diagnostics | ✅ pass | 0.017s | |
| tier1-dpdk-echo | jumbo_echo_8000 | ✅ pass | 1.696s | |
| tier1-tcp-echo-async | execution | 0.000s | ENI bind failed on receiver instance | |
| tier1-tcp-echo | bidir_echo | ✅ pass | 0.700s | |
| tier1-tcp-echo | bidir_multi | ❌ real | 60.219s | client exited non-zero |
| tier2-kernel-interop | arp_resolution | ✅ pass | 0.032s | |
| tier2-kernel-interop | udp_send_receive | ✅ pass | 1.033s | |
| tier2-kernel-interop | echo_roundtrip | ✅ pass | 0.837s | |
| tier2-kernel-interop | payload_integrity | ✅ pass | 0.035s | |
| tier2-tcp-echo | execution | 0.000s | ENI bind failed on receiver instance | |
| tier3-iperf-interop | our_app_sends | ✅ pass | 2.517s |
❌ Real failures (1) — code/test issues
- tier1-tcp-echo / bidir_multi — client exited non-zero
details
TCP Test Client
Target: 10.0.1.8:9000
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'apos;PA'apos;
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Mode: bidir (echo verification, 64B payload)
⚠️ Infra flakes (2) — SSM/ENI setup, not code (task #10)
- tier1-tcp-echo-async / execution — ENI bind failed on receiver instance
- tier2-tcp-echo / execution — ENI bind failed on receiver instance
Integration Tests — ❌ 1 real failure ·
|
| Tier | Test | Result | Time | Reason |
|---|---|---|---|---|
| tier1-dpdk-echo | arp_resolution | ✅ pass | 2.047s | |
| tier1-dpdk-echo | udp_send_receive | ✅ pass | 1.723s | |
| tier1-dpdk-echo | echo_roundtrip | ✅ pass | 1.532s | |
| tier1-dpdk-echo | payload_integrity | ✅ pass | 0.735s | |
| tier1-dpdk-echo | jumbo_diagnostics | ✅ pass | 0.021s | |
| tier1-dpdk-echo | jumbo_echo_8000 | ✅ pass | 1.721s | |
| tier1-tcp-echo-async | execution | 0.000s | ENI bind failed on receiver instance | |
| tier1-tcp-echo | bidir_echo | ✅ pass | 0.728s | |
| tier1-tcp-echo | bidir_multi | ❌ real | 60.218s | client exited non-zero |
| tier2-kernel-interop | arp_resolution | ✅ pass | 0.059s | |
| tier2-kernel-interop | udp_send_receive | ✅ pass | 1.063s | |
| tier2-kernel-interop | echo_roundtrip | ✅ pass | 0.865s | |
| tier2-kernel-interop | payload_integrity | ✅ pass | 0.063s | |
| tier2-tcp-echo | execution | 0.000s | ENI bind failed on receiver instance | |
| tier3-iperf-interop | our_app_sends | ✅ pass | 2.535s |
❌ Real failures (1) — code/test issues
- tier1-tcp-echo / bidir_multi — client exited non-zero
details
TCP Test Client
Target: 10.0.1.208:9000
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'apos;PA'apos;
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Mode: bidir (echo verification, 64B payload)
⚠️ Infra flakes (2) — SSM/ENI setup, not code (task #10)
- tier1-tcp-echo-async / execution — ENI bind failed on receiver instance
- tier2-tcp-echo / execution — ENI bind failed on receiver instance
❌ Integration Tests Failed — Graviton (run)Branch: Test Results
Application Logs (last 20 lines)sender-echo-server.log sender-test-client.log |
❌ Integration Tests Failed (Run 28587091099)Branch: Test Results
Application Logs (last 20 lines)sender-echo-server.log sender-test-client.log sender-test-client-iperf.log Full Application Logs (last 200 lines each)sender-echo-server.logsender-test-client.logsender-test-client-iperf.log
|
Owner
Author
|
We previously changes the order of the tests so we would not have to rebind. These are not "flaky" tests at this point they are just failing. To solve this start with a host without interfaces bound to DPDk perform all tests without the bind the nics and perform all the tests that need it removing the bind/re-bind flow |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One CR for the TCP-over-the-wire work (was split across #111/#112/#113 — consolidated here; all squash into development anyway).
Commits
tcp-kernel-clientreference peer, runner registration, CDK DPDK↔DPDK TCP rule.bidir_multi) hang the tiers isolated on real HW.handle_establishednever drained acked bytes fromsend_buf/ clearedhas_unacked_data, so Nagle withheld every write after the first exchange. Offline reprotests/loopback_stall_repro.rs(fails before, passes after); 221+ tests green.post_pr_comment(script-only).Status
Verified locally (bash -n, cargo build/test, adversarial review of the tiers). On hardware (Graviton), the tiers proved a single TCP round-trip works (#110) and isolated the sustained hang → commit 2 fixes it. Note: the SSM/ENI flake (task #10) polluted the run (cancelled x86, flaked the extra ENI re-binds) — separate infra issue.
🤖 Generated with Claude Code