[CE-047] cargo test runs single-threaded in CI — async timing tests may be flaky under contention

[Jaydbrown]

Problem

The CI workflow runs cargo test --all-features with the default thread count for the test harness. For a payment engine with async Tokio code, all tests use #[tokio::test] which creates a single-threaded Tokio runtime by default (current-thread executor). This means tests don't validate concurrent behavior.

Current CI:

# .github/workflows/ci.yml
- name: Test
  run: cargo test --all-features

Default cargo test uses multiple OS threads for parallelism between test functions, but each #[tokio::test] function uses a single-threaded Tokio runtime internally.

Exact location: src/retry.rs tests:

#[tokio::test]
async fn succeeds_on_first_attempt() { ... }

#[tokio::test]
async fn succeeds_on_third_attempt() { ... }

All use default #[tokio::test] which maps to #[tokio::test(flavor = "current_thread")].

Root Cause

#[tokio::test] defaults to current_thread flavor. This is fine for simple unit tests but misses concurrency bugs in the batch processor (src/batch.rs) which uses JoinSet to spawn multiple tasks simultaneously.

Impact

• Concurrency bugs in process_batch() (race conditions between JoinSet tasks) are never exercised in CI
• The retry() function's behavior under concurrent load is untested
• cargo test with --test-threads=N controls OS-level parallelism but not Tokio runtime threading

Fix

Fix 1 — Use multi-thread Tokio runtime in concurrent tests:

// src/batch.rs tests (or tests/integration_test.rs):
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn test_batch_processes_concurrently() {
    let engine = Arc::new(create_test_engine_with_mock_adapter());
    let items: Vec<BatchItem> = (0..10).map(|i| BatchItem {
        sender: format!("G{:055}", i),
        recipient: format!("G{:055}", i + 1),
        amount: 100,
        token: "XLM".into(),
        urgency: Urgency::Standard,
    }).collect();

    let result = process_batch(engine, items).await.unwrap();
    assert_eq!(result.total, 10);
    assert_eq!(result.succeeded, 10);
}

Fix 2 — Add --test-threads flag in CI for parallel test execution:

# .github/workflows/ci.yml
- name: Test
  run: cargo test --all-features -- --test-threads=4
  # Runs up to 4 test functions concurrently

Fix 3 — Increase CI CPU count for meaningful parallelism:

# .github/workflows/ci.yml
jobs:
  ci:
    runs-on: ubuntu-latest-4-cores   # Use a larger runner
    # or:
    runs-on: ubuntu-latest
    # Accept that ubuntu-latest has 2 CPUs and design tests accordingly

Steps to Verify

1. Add #[tokio::test(flavor = "multi_thread", worker_threads = 4)] to batch tests
2. Introduce a deliberate race condition in process_batch() — confirm the multi-thread test catches it, the single-thread test misses it
3. cargo test -- --test-threads=4 — confirm tests still pass under parallelism
4. Check for Rust cargo test output showing tests running in parallel: test result: ok. N passed; M measured; O filtered out

[YfengJ]

I would like to apply for this issue if it is still available.

If assigned, I would keep this focused on test concurrency coverage: add or update a batch/retry test to use #[tokio::test(flavor = "multi_thread", worker_threads = 4)], update CI to run tests with explicit --test-threads, and verify the suite still passes under parallel execution without changing runtime behavior. I would avoid larger runner changes unless the maintainers specifically want that.

I will wait for assignment or explicit maintainer confirmation before starting implementation or opening a PR.