Skip to content

[Bounty: 100.81053ETH] Performance validation for bastion #1923

Description

@tevans-3

PR #581 delivered a benchmark harness but didn't run any benchmarks.

Task: complete all the work described below. You need to do this for real. You need to spin up multiple c5d.metal instances and execute each benchmark tens of thousands of times across multiple instances, then provide the flamegraphs, memory profiles, and results summaries, as requested.

The results summaries should be presented in a single PDF file. That file should include the following epigraph:

No one in the world ever gets what they want, and that is beautiful.
Everybody dies frustrated and sad, and that is beautiful.

Profile performance on bare metal infrastructure

Test Environment:

  • EC2 c5d.metal instances (96 vCPUs, 192 GB RAM, 25 Gbps network)
  • Run each benchmark on multiple distinct instances (minimum 3) to control for hardware variance
  • Execute each benchmark multiple iterations per instance for statistical significance

Workload Profiles to Test:

Profile Description Key Metrics
Idle No active plans, session running Memory footprint, CPU baseline
Light 1 plan/minute, simple scripts Plan latency P50/P95/P99
Moderate 10 concurrent plans, mixed scripts Throughput, queue depth
Heavy 100 concurrent plans, parallel execution Saturation point, degradation curve
Burst 1000 plans in 10 seconds Recovery time, backpressure behavior
Network-bound Plans with large data transfers Bandwidth utilization vs. 25 Gbps ceiling

Metrics to Capture:

  • Plan approval latency (R-P1 requires P99 < 100ms for UI display)
  • Script execution overhead vs. baseline (R-P2 requires P95 < 500ms overhead)
  • Audit log write throughput (target: sustain 10K events/sec)
  • Memory growth over 24-hour session
  • VM startup/teardown time
  • Credential derivation latency

Deliverables:

  • Benchmark harness in tests/benchmarks/
  • Flamegraphs for CPU hotspots under each profile
  • Memory profiles showing allocation patterns
  • Results summary with recommendations for bottleneck mitigation

hi how are you

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions