fix(perf): TRex ASTF mode + SSM TimeoutSeconds>=30 + kill stale TCP DUTs#103
Merged
Conversation
Fixes the three harness bugs from run 28509354005 (the DPDK TCP engine
itself worked — tcp-echo bound + listened on the real NIC):
1. TRex ran in STL mode but the TCP benchmark speaks ASTF -> 'RPC
configuration mismatch'. Add start_trex_server <stl|astf> + an
ensure_trex_mode helper; UDP configs ensure STL, TCP configs ensure
ASTF; initial mode is derived from the --configs list.
2. SSM send-command rejects TimeoutSeconds < 30; two calls passed 15
(DPDK-state cleanup + instance-type probe). Clamp to >=30 in
ssm_run_command so every caller is safe.
3. dut_stop_all_apps / dut_bind_* pkill patterns missed the TCP binaries
(tcp-echo/tokio-tcp-echo/plain-tcp-echo), so a stale tcp-echo kept the
DPDK primary-process lock and the next config's EAL init failed
('Cannot create lock ... another primary process running'). Replace
the ad-hoc patterns with a shared DUT_APP_ERE covering all DUT
binaries + clear /var/run/dpdk on stop.
Scripts-only; bash -n clean. Re-dispatch perf-tests.yml to verify.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add scripts/perf-tests/test/test_run_perf_tests.sh — pure-bash, no AWS/TRex,
runs in <1s by sourcing the harness (main is now source-guarded) and stubbing
the aws CLI. Guards the three bug classes from run 28509354005:
- SSM TimeoutSeconds clamp (>=30)
- DUT_APP_ERE matches every DUT binary incl. tcp-echo/tokio-tcp-echo/
plain-tcp-echo, and does NOT match t-rex-64/sshd
- TRex stl/astf mode selection (trex_mode_for_configs) + ensure_trex_mode
restart-on-change
Harness: source-guard main(); extract trex_mode_for_configs() so the mode
logic is unit-testable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Synthetic Performance Results (run)Commit: ✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01) Synthetic UDP Performance ResultsMeasures framework overhead: sync IPv4 Baseline
IPv6
IPv6 vs IPv4 Comparison (sync path)
IPv4 avg sync/async ratio: 0.9x, worst: 1.0x | IPv6 vs IPv4 worst ratio: 1.32x (OK)
|
Synthetic Performance Results — Graviton (run)Commit: ✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01) Synthetic UDP Performance ResultsMeasures framework overhead: sync IPv4 Baseline
IPv6
IPv6 vs IPv4 Comparison (sync path)
IPv4 avg sync/async ratio: 0.9x, worst: 1.0x | IPv6 vs IPv4 worst ratio: 1.33x (OK)
|
[CI] Stage: DeployInfrastructure ready.
|
[CI] Stage: DeployInfrastructure ready.
|
…Y cleanup - DUT_RUST_BINS/DUT_OTHER_BINS arrays are the single source of truth; DUT_APP_ERE composes from them via the IFS='|' join. Add a binary in one place. - DUT_KILL_GRACE_SECS names the magic '10'; the wait list is expanded locally via seq so the remote command stays POSIX-portable (dash-safe). - dut_kill_snippet() DRYs the three near-identical cleanup blocks (dut_bind_dpdk/dut_bind_kernel/dut_stop_all_apps). - Tests now derive coverage from the arrays and assert the snippet composition (23 passed/0 failed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
[CI] Stage: SummaryAll tests PASSED. ARP seeding: kernel /proc/net/arp (automatic)
|
1 similar comment
[CI] Stage: SummaryAll tests PASSED. ARP seeding: kernel /proc/net/arp (automatic)
|
✅ Integration Tests Passed (Run 28513039208)Branch: Test Results
Application Logs (last 20 lines)receiver-echo-server.log sender-echo-server.log sender-test-client.log receiver-test-client-iperf.log sender-test-client-iperf.log Full Application Logs (last 200 lines each)receiver-echo-server.logsender-echo-server.logsender-test-client.logreceiver-test-client-iperf.logsender-test-client-iperf.log
|
✅ Integration Tests Passed — Graviton (run)Branch: Test Results
Application Logs (last 20 lines)receiver-echo-server.log sender-echo-server.log sender-test-client.log |
Synthetic Performance Results (run)Commit: ✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01) Synthetic UDP Performance ResultsMeasures framework overhead: sync IPv4 Baseline
IPv6
IPv6 vs IPv4 Comparison (sync path)
IPv4 avg sync/async ratio: 0.9x, worst: 1.1x | IPv6 vs IPv4 worst ratio: 1.33x (OK)
|
Synthetic Performance Results — Graviton (run)Commit: ✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01) Synthetic UDP Performance ResultsMeasures framework overhead: sync IPv4 Baseline
IPv6
IPv6 vs IPv4 Comparison (sync path)
IPv4 avg sync/async ratio: 0.9x, worst: 1.0x | IPv6 vs IPv4 worst ratio: 1.35x (OK)
|
[CI] Stage: DeployInfrastructure ready.
|
[CI] Stage: DeployInfrastructure ready.
|
[CI] Stage: SummaryAll tests PASSED. ARP seeding: kernel /proc/net/arp (automatic)
|
1 similar comment
[CI] Stage: SummaryAll tests PASSED. ARP seeding: kernel /proc/net/arp (automatic)
|
✅ Integration Tests Passed — Graviton (run)Branch: Test Results
Application Logs (last 20 lines)receiver-echo-server.log sender-echo-server.log sender-test-client.log |
✅ Integration Tests Passed (Run 28514233861)Branch: Test Results
Application Logs (last 20 lines)receiver-echo-server.log sender-echo-server.log sender-test-client.log receiver-test-client-iperf.log sender-test-client-iperf.log Full Application Logs (last 200 lines each)receiver-echo-server.logsender-echo-server.logsender-test-client.logreceiver-test-client-iperf.logsender-test-client-iperf.log
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the three harness bugs that failed perf run 28509354005. The DPDK TCP engine itself worked —
tcp-echodid a real vfio/net_enaPCI probe (new_real_nic) and listened on10.0.1.10:9000. These are harness-only fixes; scripts-only,bash -nclean.Bugs fixed
-i(STL) but the TCP benchmark connects as ASTF →RPC configuration mismatch - server 'STL', client 'ASTF'. Addedstart_trex_server <stl|astf>+ anensure_trex_modehelper. UDP configs ensure STL; TCP configs ensure ASTF; the initial mode is derived from the--configslist (no mid-run restart for coherent runs). Corrects the wrong "serves both" comment.TimeoutSeconds< 30 — AWS rejects it (ParamValidation); two calls passed15(DPDK-state cleanup + instance-type probe). Clamp to ≥30 inssm_run_commandso every caller is safe.dut_stop_all_apps/dut_bind_*pkill patterns missedtcp-echo/tokio-tcp-echo/plain-tcp-echo, so the previoustcp-echosurvived and the next config's EAL init failed (Cannot create lock ... another primary process running). Replaced the ad-hoc patterns with a sharedDUT_APP_EREcovering all DUT binaries and clear/var/run/dpdkon stop.Verify
Re-dispatch:
gh workflow run perf-tests.yml --ref development -f configs=rust-dpdk-tcp,tokio-dpdk-tcp,plain-rust-tcp(after merge) → expect non-zero CPS/throughput inperf-report.json.🤖 Generated with Claude Code