Skip to content

testing.md

Latest commit

 

History

History
432 lines (332 loc) · 23.6 KB

testing.md

File metadata and controls

432 lines (332 loc) · 23.6 KB
Raw

Testing

missive tests should be runnable from a clean checkout without contacting third-party agents by default. Use scripts/quality-gate.sh as the standard local validation entry point before commits and pull requests.

CI validation

GitHub Actions runs the Linux default quality gate on push and pull_request through .github/workflows/ci.yml, then runs a Rust workspace matrix on Linux and macOS. The workflow first validates GitHub Actions YAML with actionlint; the Linux quality-gate job runs scripts/quality-gate.sh so CI covers the same shell checks, guardrails, Rust formatting, clippy, tests, doc tests, docs, builds, and cargo-deny supply-chain policy as local validation. The workspace matrix runs cargo check, cargo test, and debug builds on ubuntu-latest and macos-latest so portable CLI/library functionality stays buildable beyond Linux. Windows workspace testing is not part of default CI until the state-path and lock fixtures are made native Windows-safe. Platform-specific shell/service tests are gated: Bash example smoke scripts are not treated as Windows-native tests, Linux systemd and macOS launchd service planning are documented as host-specific, and Windows service installation remains unsupported with a clear diagnostic. Optional coverage smoke is available through manual workflow dispatch or the non-secret MISSIVE_CI_COVERAGE=1 repository variable. See ci.md for the workflow, cache, platform matrix, and secret posture.

Container validation

The Docker development image contains Rust stable, protobuf tooling, SQLite tooling, ShellCheck, Python/YAML validation support, and the default optional cargo helpers used by the quality gate. Build it and run the local quality gate inside the container with:

docker build --pull=false --tag missive-dev:local .
scripts/docker-integration.sh

The wrapper bind-mounts the checkout, uses temporary container paths for Cargo and target output, sets MISSIVE_HOME outside the repository, and runs scripts/quality-gate.sh. Use MISSIVE_AGGRESSIVE_TESTS=1 scripts/quality-gate.sh on the host to include Docker/devcontainer checks in the aggressive path when Docker is available. See container.md for manual docker run and devcontainer workflows.

Property-based tests

Property-based tests use proptest and run as part of the normal workspace test pass. Current properties cover core identifier round trips and rejection, routing-policy parsing from CLI/config names, config rejection of reserved A2A service-parameter headers, metadata merge right-bias plus deterministic key order, CLI value parsers for group weights/tags/metadata, and router selection invariants for round-robin, weighted, and tag-match policies.

For targeted runs:

cargo test -p missive-core --all-features
cargo test -p missive-router --all-features
cargo test -p missive-cli --lib --all-features group::tests

When a property fails, proptest prints the minimized failing input and reproduction details in the test output; set PROPTEST_RNG_SEED=<seed> when a reported RNG seed needs to be replayed exactly. Do not commit transient failure artifacts unless a regression file is intentionally added as a stable seed for a known bug fix.

Fuzzing smoke tests

cargo-fuzz targets live under fuzz/ and are intentionally scoped to untrusted parser/replay boundaries rather than long-running network behaviour. Current targets cover TOML config parsing, A2A protocol JSON parsing, event replay reconstruction, and stdio/HTTP/file-drop adapter frame parsing.

Compile all fuzz targets on the stable toolchain:

cargo fuzz build --sanitizer none

Run short targeted smoke checks:

cargo fuzz run config_parse --sanitizer none -- -runs=100
cargo fuzz run a2a_json_parse --sanitizer none -- -runs=100
cargo fuzz run event_replay --sanitizer none -- -runs=100
cargo fuzz run cli_frame_parse --sanitizer none -- -runs=100

Aggressive quality-gate mode automatically discovers the same targets when cargo-fuzz is installed and bounds each run with MISSIVE_FUZZ_SECONDS. It defaults to MISSIVE_FUZZ_SANITIZER=none for stable-toolchain compatibility; use a compatible nightly setup and MISSIVE_FUZZ_SANITIZER=address for sanitizer-backed campaigns:

MISSIVE_AGGRESSIVE_TESTS=1 MISSIVE_FUZZ_SECONDS=3 scripts/quality-gate.sh

Generated fuzz/corpus/, fuzz/artifacts/, fuzz/crashes/, fuzz/coverage/, and fuzz/target/ output is ignored and guarded against accidental commits. Commit only intentional, reviewed regression seeds when a future bug fix explicitly requires stable corpus data.

Mutation and failure-injection tests

Targeted failure-injection tests now exercise critical communication-control paths without relying on external services:

  • missive-store verifies transactions roll back both newly inserted rows and updates to preexisting rows after injected SQLite constraint failures.
  • missive-router rejects injected invalid candidate weights, duplicate normalized requirements, and missing preferred-agent references before planning a route.
  • missive-cli auth/redaction tests reject token-looking --bearer-token-env values without echoing them.
  • task_command injects malformed remote cancellation responses while proving outbound auth headers are sent, stderr is redacted, and no partial task row is persisted.
  • reduce_command injects a failing local reducer pipeline, records a missive.reduce.failed event, and redacts command stderr in both output and the event journal.

The bounded mutation smoke entry point is:

scripts/mutation-smoke.sh

By default it runs cargo-mutants in --check mode over the store repository, router planner, auth/output redaction, task wait/cancel, and collective command source files. The same script is used by aggressive quality-gate mode when cargo-mutants is installed:

MISSIVE_AGGRESSIVE_TESTS=1 MISSIVE_MUTANTS_TIMEOUT=20 scripts/quality-gate.sh

Useful controls:

Variable Default Purpose
MISSIVE_MUTANTS_MODE check Use check, list, or run. run executes tests against selected mutants.
MISSIVE_MUTANTS_FILES critical source list Space-separated file globs passed as repeated --file filters.
MISSIVE_MUTANTS_RE unset Regex filter for specific mutants from cargo mutants --list output.
MISSIVE_MUTANTS_SHARD 1/12 Bounds the default smoke slice. Use wider shards for campaigns.
MISSIVE_MUTANTS_TIMEOUT 30 Per-command timeout in seconds.
MISSIVE_MUTANTS_JOBS 1 Parallel mutant jobs.
MISSIVE_MUTANTS_BASELINE skip Baseline strategy; the quality gate has already run the normal tests.
MISSIVE_MUTANTS_KEEP_OUTPUT 0 Set to 1 to retain the temporary mutants output directory for inspection.

Longer local campaigns should keep artifacts out of the repository. Examples:

# List planned mutants in the critical smoke set.
MISSIVE_MUTANTS_MODE=list scripts/mutation-smoke.sh

# Run tests for one small router mutant slice.
MISSIVE_MUTANTS_MODE=run \
  MISSIVE_MUTANTS_FILES='crates/missive-router/src/lib.rs' \
  MISSIVE_MUTANTS_SHARD=1/24 \
  MISSIVE_MUTANTS_TIMEOUT=60 \
  scripts/mutation-smoke.sh

# Run a broader campaign outside the smoke wrapper and keep generated output ignored.
cargo mutants --workspace --all-features --output mutants.out --timeout 120

mutants.out/ and temporary missive-mutants.* directories are generated runtime artifacts; do not commit them.

Supply-chain checks and SBOM smoke

The reviewed dependency policy lives in ../deny.toml. Run it locally with:

cargo deny --locked check

Useful companion checks are:

cargo audit
cargo machete
cargo tree -d --all-features

A local CycloneDX JSON SBOM can be generated and smoke-validated with JSON tools:

scripts/generate-sbom.sh --output /tmp/missive-sbom.cdx.json
python3 -m json.tool /tmp/missive-sbom.cdx.json >/dev/null

Generated SBOMs and reports are ignored/guarded artifacts. See supply-chain.md for allowed licenses, duplicate-version exceptions, advisory handling, and the dependency update workflow.

Benchmarks and performance budgets

Criterion benchmarks live in crates/missive-cli/benches/performance.rs and cover config load/redaction, store operations, router planning, event replay, A2A streaming event parsing, and local group collective gather/reduce paths. Compile all benchmark targets without measuring:

cargo bench --workspace --all-features --no-run

Run the benchmark suite locally:

cargo bench -p missive-cli --bench performance --all-features

For a fast CI-style smoke run of the custom Criterion harness:

cargo bench -p missive-cli --bench performance --all-features -- --test

The current budgets are intentionally soft investigation thresholds rather than default quality-gate failures. See performance.md for the benchmark list, baseline/compare workflow, generated artifact hygiene, and initial budget table.

Observability tests

crates/missive-observe unit tests build scoped tracing dispatchers with an in-memory writer so RUST_LOG-style filters, human/JSON log rendering, and secret redaction are deterministic without mutating the process-global subscriber. CLI tests cover mapping --trace, --verbose, RUST_LOG, and MISSIVE_LOG_FORMAT=json into the shared observe config plus cli.command span fields in JSON stderr logs. completion_manpage_command tests generate bash/zsh/fish/PowerShell completion scripts, snapshot stable prefixes, assert completion/manpage generation bypasses configuration loading, and verify JSON envelopes for generated content. doctor_command tests run with isolated MISSIVE_HOME directories and cover no-config reports, valid populated config plus migrated SQLite state, invalid config as a structured failed check, unmigrated database detection, mock reachable/unreachable A2A endpoint probes, gateway running/unavailable status probes, deterministic PATH tool lookup, and redaction of secret-like config/auth metadata. logs_command tests assert actionable empty-source JSON, profile-file JSON/NDJSON rendering, bounded --limit behavior, and token/cookie/API-key redaction; events_command tests cover event list/export/replay plus bounded events tail follow/timeout behavior and secret-like free-text payload redaction. A2A, store, and adapter unit tests use scoped dispatchers to verify representative a2a.request, store.operation, and adapter.event spans without relying on one process-global subscriber in parallel test runs.

Local mock A2A server fixtures

Reusable A2A integration fixtures live in the dev-support crate:

crates/missive-test-support

The main fixture is missive_test_support::MockA2aServer. It starts a local 127.0.0.1 HTTP server on an ephemeral port and serves:

  • public Agent Card discovery at /.well-known/agent-card.json
  • HTTP+JSON A2A endpoints under /a2a
  • JSON-RPC A2A endpoints at /rpc
  • SendMessage direct-message or task responses
  • SendStreamingMessage SSE streams, including JSON-RPC-wrapped stream events
  • GetTask, ListTasks, and CancelTask task routes with controllable task-state queues
  • push notification config create/get/list/delete fixture endpoints
  • optional auth-header requirements
  • optional VERSION_NOT_SUPPORTED responses for unsupported A2A-Version values
  • optional malformed Agent Card, send, task, stream, and JSON-RPC responses
  • optional SendMessage response delay for timeout tests

Example integration test setup:

use missive_test_support::{MockA2aServer, send_message_response_message};

let server = MockA2aServer::start();
server.handle().set_send_response(send_message_response_message(
    "msg-fixture-response",
    "ctx-fixture-response",
    "fixture response",
));

let base_url = server.base_url();
let http_interface = server.http_json_interface_url();
let rpc_interface = server.json_rpc_interface_url();
let requests = server.requests();

For task polling tests, enqueue deterministic state transitions before invoking the client:

server.handle().enqueue_task_states(
    "task-1",
    "ctx-1",
    ["TASK_STATE_WORKING", "TASK_STATE_COMPLETED"],
);

For streaming tests, replace the SSE event list:

use missive_test_support::{artifact_update_event, status_update_event};

server.handle().set_stream_events(vec![
    status_update_event("task-1", "ctx-1", "TASK_STATE_WORKING", Some("thinking")),
    artifact_update_event("task-1", "ctx-1", "artifact-1", "answer", true, true),
    status_update_event("task-1", "ctx-1", "TASK_STATE_COMPLETED", None),
]);

For push config tests, MockA2aServer serves both HTTP+JSON /a2a/tasks/{taskId}/pushNotificationConfigs routes and JSON-RPC Create/Get/List/DeleteTaskPushNotificationConfig methods. Use server.handle().insert_push_config(...) or run the CLI push create command to seed fixture state.

For auth and protocol-version error paths, configure the builder:

let server = MockA2aServer::builder()
    .require_auth_header("Authorization", "Bearer fixture-value")
    .supported_protocol_versions(["2.0"])
    .start();

The fixture records request method, path, lowercase headers, and UTF-8 body for assertions. It supports task resubscription through POST/GET /a2a/tasks/{id}:subscribe and JSON-RPC SubscribeToTask, returning deterministic SSE events from the same stream-event queue used by message:stream. It is intentionally deterministic and local-only; tests must not use it as a proxy to external services.

A2A conformance fixture suite

Protocol-versioned conformance examples live under:

tests/fixtures/a2a/1.0/

The directory name tracks the A2A major/minor protocol version. The suite covers Agent Cards, messages, tasks, artifacts, streaming events, push notification configs, JSON-RPC envelopes, HTTP error bodies, and normalized CLI golden outputs under tests/fixtures/a2a/1.0/cli/.

Targeted conformance checks are:

cargo test -p missive-a2a --test protocol_fixtures --all-features
cargo test -p missive-cli --test a2a_conformance_fixtures --all-features

To intentionally refresh CLI golden outputs after a public output contract change, run:

MISSIVE_UPDATE_GOLDENS=1 cargo test -p missive-cli --test a2a_conformance_fixtures --all-features

Only commit the normalized golden JSON, not local ports, generated message IDs, timestamps, runtime databases, or captured external traffic. The fixture README contains the update process for future A2A protocol versions.

Upstream a2a-rs interoperability check

The opt-in upstream compatibility script runs missive against the pinned a2aproject/a2a-rs hello-world example agent:

scripts/interop-a2a-rs.sh

It clones the pinned upstream revision into a temporary directory, starts the example server on its fixed local ports, creates an isolated MISSIVE_HOME, and validates Agent Card discovery, send, stream, remote task list, and push configuration when the example advertises push support. The current pinned hello-world Agent Card advertises pushNotifications=false, so push_config is reported as an upstream_example_limitation skip rather than a missive failure. See interoperability.md for the pinned revision, configuration variables, result files, and failure classifications.

Command example smoke tests

Runnable shell examples live under examples/. The canonical entry point is:

examples/run-smoke.sh

The scripts create temporary MISSIVE_HOME state, start or reuse local mock A2A servers, and exercise agent registry, send, stream, task, context, group, route, gateway, and the three-agent broadcast/barrier/gather/reduce collective workflow. The default quality gate covers them through the CLI integration test:

cargo test -p missive-cli --test example_smoke --all-features

The test injects an already-built missive binary plus in-process MockA2aServer instances, including three distinct servers for the multi-agent collective demo, so CI does not invoke nested Cargo builds or contact external services.

Running the fixture tests

Targeted checks for the reusable mock server and CLI integration are:

cargo test -p missive-test-support --all-targets
cargo test -p missive-cli --test example_smoke --all-features
cargo test -p missive-cli --test mock_a2a_server_fixture --all-features
cargo test -p missive-cli --test push_command --all-features
cargo test -p missive-cli --test group_command --all-features
cargo test -p missive-cli --test bcast_command --all-features
cargo test -p missive-cli --test barrier_command --all-features
cargo test -p missive-cli --test gather_command --all-features
cargo test -p missive-cli --test reduce_command --all-features
cargo test -p missive-cli --test job_command --all-features
cargo test -p missive-cli --test adapter_file_drop_command --all-features
cargo test -p missive-cli --test gateway_command --all-features
cargo test -p missive-cli --test webhook_command --all-features

The default quality gate also runs these tests and the conformance suite through the workspace test pass:

scripts/quality-gate.sh

Extending fixtures

The group command integration test uses an isolated MISSIVE_HOME and local agent registry rows to cover group create/list/show, member add/remove, duplicate rank handling, rename membership preservation, delete cascades, missing reference validation, and human output. The broadcast collective integration test uses isolated state plus reusable mock A2A servers to cover concurrent success, partial failure, timeout through delayed SendMessage responses, per-member task/message persistence, request shape, and missive.bcast.* event rows. The barrier collective integration test covers consuming bcast_result JSON from stdin, remote GetTask polling to terminal completion, local terminal failure/cancellation exit codes, quorum with --failure-policy continue, requested non-terminal states, timeout handling, task persistence, and missive.barrier.* event rows. The gather collective integration test covers rank-ordered local output collection, missing task representation, JSON/NDJSON output, safe artifact export without accidental overwrite, and missive.gather.* event rows. The reduce collective integration test covers local deterministic reduction with provenance, mocked reducer-agent prompting via A2A SendMessage, persisted local reduced-output messages, missive.reduce.* event rows, and the no-gathered-input validation failure. The route explain integration test uses isolated local registry/group rows to cover weighted, tag-match, capability-match, and round-robin dry-run explanations, human/JSON output, candidate-source validation, and invalid routing policy config failures. The capability-selection integration test uses reusable mock A2A servers to cover agent capabilities cache fetch/reuse, group capabilities aggregation, route explain --refresh-capabilities Agent Card refresh, matching by skill label/input mode/output mode/streaming/push support, missing cached capability data diagnostics, and weight tie-breaking; crates/missive-router unit tests cover every built-in policy's deterministic decision path plus capability-mode matching. The background job integration test covers job start/list/show/cancel JSON output, raw-request omission from job views, gateway execution of a queued send job through the reusable mock A2A server, persisted gateway_jobs result state, missive.gateway.job.* events, and job cancel --remote issuing A2A CancelTask. The gateway command integration test spawns the missive binary, waits for the local /healthz endpoint, checks /status component JSON over loopback HTTP, verifies graceful --timeout shutdown, checks NDJSON lifecycle output, and inspects persisted missive.gateway.* event journal rows. The same test file covers the opt-in HTTP adapter by starting the gateway with --http-adapter, checking /adapter/http/healthz, posting authenticated and unauthenticated missive.http.v1 frames over loopback HTTP, asserting invalid request rejection, verifying graceful shutdown, and proving the configured token is not present in stdout or event journal rows. It also covers service management dry runs: generated Linux systemd unit content, planned systemctl commands, refusal to embed secret-looking environment variables, and the safety requirement that --system installs provide an explicit MISSIVE_HOME. The missive-gateway crate tests also exercise service file generation for systemd and launchd, HTTP adapter config/auth/rate-limit helpers, subscription resume by seeding an in-flight task plus a persisted task_subscription job, serving local mock SubscribeToTask SSE updates, verifying terminal cleanup, checking bounded retry/backoff metadata for malformed streams, and background job helpers for public job-kind recognition plus restart pickup of expired running jobs. Gateway session unit tests additionally use fixed clocks to cover daily, idle, and combined reset policy boundaries without wall-clock sleeps. Gateway busy-input unit tests cover queue, interrupt, steer, unsupported-steer fallback to queue/interrupt, queue-depth limits, and the no-active-operation start path without needing live daemon adapters. crates/missive-adapters unit tests cover deriving adapter definitions from config, duplicate/missing registry factories, disabled adapter handling, cross-adapter event rejection, the built-in stdio frame parser/writer, stdio invalid-frame diagnostics, stdio streaming output frame sequencing, file-drop schema validation, atomic ready-file handoff, file-drop registry creation, and fake/stdio/file-drop adapters that map identity, emit lifecycle/inbound-message events, accept outbound updates, and record acknowledgements. HTTP adapter unit tests cover missive.http.v1 frame validation, adapter registry creation, and inbound-message mapping. External chat stub unit tests cover the static Discord/Slack/Telegram/Matrix/Email roadmap metadata, feature-gated factory registration in all-features builds, no-default feature compilation with no external factories enabled, deterministic identity mapping, and explicit configuration errors for live runtime operations. crates/missive-cli also has adapter_stdio_command integration coverage for valid long-running task frames, invalid frame recovery, and wrapped streaming NDJSON output through the reusable local A2A mock server, plus adapter_file_drop_command coverage for temp-directory handoff, ignored partial files, malformed-file error archival, and background job request files. crates/missive-gateway also has a fake adapter event-sink test proving adapter inbound messages can enter the daemon event bus and update runtime output. The webhook command integration test similarly spawns the binary, waits for local /healthz, posts unauthorized, malformed, and valid A2A StreamResponse payloads over loopback HTTP, verifies graceful --max-events shutdown, checks NDJSON output, and inspects the persisted event journal. Neither test contacts external tunnel providers or third-party services.

When future work adds more adapter workers, additional collectives, or compatibility suites, prefer extending crates/missive-test-support instead of adding another one-off TCP mock inside a test file. Keep fixture changes focused on protocol surfaces needed by the change:

  1. add a helper or route to MockA2aServer/MockA2aHandle
  2. cover it with a local test in crates/missive-test-support
  3. add a CLI or crate integration test that consumes the helper
  4. document any new endpoint shape or limitation here

Do not commit runtime databases, logs, sockets, pid files, captured external traffic, real credentials, or private service URLs as fixture data.