feat: add competitor benchmark framework for WAC performance

[rubenfiszel]

Summary

Adds a reproducible benchmark framework that compares Windmill's workflow-as-code performance against 6 competitors: Temporal, Inngest, Restate, Kestra, Prefect, and Apache Airflow.

Each platform runs an equivalent 3-step sequential workflow with inline step execution. The harness measures cold start, single-execution latency (median + P95), throughput under concurrency, and per-step orchestration overhead.

Results (on local dev machine, 100 latency samples, 200 throughput batch)

Competitor	Median Latency	P95	Throughput	Step Overhead
Restate	4.1ms	5.0ms	2,057/s	1.4ms
Windmill	103ms	105ms	152/s	34ms
Temporal	119ms	283ms	39/s	40ms
Kestra	177ms	326ms	68/s	59ms
Inngest	260ms	261ms	90/s	87ms
Airflow	3,792ms	4,277ms	7.4/s	1,264ms
Prefect	9,679ms	12,631ms	1.3/s	3,226ms

Windmill uses step() (inline execution, no child jobs) — architecturally equivalent to what Restate ctx.run(), Inngest step.run(), and others do. With 10 workers and SLEEP_QUEUE=50, Windmill achieves the lowest latency and highest throughput among queue-based systems.

Changes

• benchmarks/competitors/ — Full benchmark framework (Deno TypeScript)
  • types.ts — CompetitorAdapter interface and result types
  • lib/ — Docker lifecycle, timing, statistics, result serialization
  • competitor_harness.ts — Main orchestrator (setup → deploy → test → teardown)
  • competitor_suite.ts — CLI entrypoint
  • competitor_graphs.ts — SVG bar chart generation
• 7 competitor adapters, each with Docker Compose + workflow definition:
  • windmill/ — WAC step() with 10 workers, SLEEP_QUEUE=50
  • temporal/ — TypeScript SDK with proxyActivities(), 10 concurrent tasks
  • inngest/ — step.run() with 10 queue workers
  • restate/ — ctx.run() (in-process)
  • kestra/ — io.kestra.plugin.core.debug.Return tasks
  • prefect/ — Python @task + @flow with .serve()
  • airflow/ — PythonOperator DAG with LocalExecutor
• .github/workflows/benchmark-competitors.yml — Weekly CI on ubicloud-standard-8

Test plan

• Run deno run -A benchmarks/competitors/competitor_suite.ts --competitors windmill --latency-samples 10 --throughput-batch 20 --output-dir ./results
• Run full suite: deno run -A benchmarks/competitors/competitor_suite.ts
• Verify JSON results in ./results/
• Generate graphs: deno run -A --allow-import benchmarks/competitors/competitor_graphs.ts -c benchmarks/competitors/competitor_graphs_config.json

---

Generated with Claude Code

---

Summary by cubic

Add a reproducible benchmark framework to compare Windmill WAC performance against Temporal, Inngest, Restate, Kestra, Prefect, and Airflow. Includes a Deno-based harness, per-competitor Docker setups, a CLI, SVG chart generation, weekly CI, and new docs for continuation.

• New Features
  • Benchmark harness runs an equal 3-step inline workflow across systems and measures cold start, single-run latency (median/P95), throughput, and per-step overhead.
  • Adapters and Docker Compose stacks for Windmill, Temporal, Inngest, Restate, Kestra, Prefect, and Airflow; results saved as JSON with optional SVG charts.
  • CLI competitor_suite.ts to run all or selected competitors with configurable sample and batch sizes.
  • Weekly GitHub Action (.github/workflows/benchmark-competitors.yml) scheduled for Mondays 06:00 UTC on ubicloud-standard-8.
  • backend/write_latest_ee_ref.sh adds --main to pin the EE ref to the latest main.
  • Docs: benchmarks/competitors/README.md for setup and usage, and CONTEXT.md with architecture, decisions, gotchas, and next steps.

^{Written for commit adbf472. Summary will update on new commits.}

[cloudflare-workers-and-pages]

Deploying windmill with    Cloudflare Pages

Latest commit:	adbf472
Status:	✅  Deploy successful!
Preview URL:	https://570a343a.windmill.pages.dev
Branch Preview URL:	https://wac-competitor-research.windmill.pages.dev

View logs