perf: add reproducible private and group messaging benchmarks

.github/workflows/ci.yml

@@ -58,6 +58,7 @@ jobs:
 
       - name: Run self-contained unit tests
         run: |
+          go test ./simulator/benchmark
           go test ./services/commonservices/...
           go test ./commons/metrics/...
           go test ./services/pushmanager/services/httputil

.gitignore

@@ -11,3 +11,4 @@ launcher/uploads/
 CLAUDE.md
 .claude/
 .promotion-metrics/
+/benchmark-results/

BENCHMARKS.md

@@ -0,0 +1,146 @@
+# Reproducible benchmarks
+
+JuggleIM's benchmark harness measures connection establishment separately from steady-state message delivery. It favors reproducibility and explicit limitations over headline numbers.
+
+The harness runs against an isolated JuggleIM deployment, creates synthetic users through the signed server API, connects real WebSocket clients using the production protocol, and records a machine-readable JSON report. It must never target a public or production deployment.
+
+## Workloads
+
+### Private chat
+
+Every connected client is a sender. Clients form a ring: client `N` sends to client `N+1`, and the final client sends to the first. The configured rate is the aggregate publish rate across the ring.
+
+### Group chat
+
+All clients join one synthetic group. A configurable subset of clients publishes to the group at the aggregate target rate. The report distinguishes accepted publishes from observed recipient deliveries, making fan-out visible without treating it as sender throughput.
+
+Both workloads use stored, counted text messages by default. Each JSON payload contains a run ID, phase, sequence number, nanosecond send timestamp, and padding to reach the configured minimum size.
+
+## Metrics
+
+The JSON report contains:
+
+- connection attempts, successes, failures, error codes, and connection latency;
+- publish-to-ACK throughput and P50/P95/P99 latency;
+- publish-to-recipient-callback throughput and P50/P95/P99 latency;
+- workload settings, including clients, senders, message rate, payload size, warm-up, duration, and delivery grace;
+- server commit, dirty-tree state, OS, architecture, Go version, CPU model, logical CPU count, memory, database label, and deployment label;
+- explicit limitations that must accompany any published result.
+
+Warm-up traffic is excluded from all message measurements. Connection establishment is completed before warm-up and is never mixed into steady-state latency.
+
+A small **[verified local smoke run](./docs/benchmarks/results/README.md)** and its machine-readable private-chat and group-chat reports are checked into the repository. They validate the full harness; they are explicitly not production capacity claims.
+
+## One-command local run
+
+Requirements:
+
+- Docker with Compose;
+- Go version from [`go.mod`](./go.mod);
+- `curl`, `jq`, `openssl`, and `shasum`.
+
+Run both baseline workloads:
+
+```bash
+scripts/run-benchmark.sh all
+```
+
+The script builds and starts the local Compose stack, creates an isolated application, runs private and group workloads, and writes JSON files under `benchmark-results/`. It leaves the stack running so logs, profiles, and database state can be inspected. Stop it without deleting the database volume:
+
+```bash
+docker compose down
+```
+
+The default baseline is intentionally modest:
+
+| Setting | Default |
+| --- | ---: |
+| Connected clients | 50 |
+| Aggregate publish rate | 200 messages/second |
+| Group senders | 2 |
+| Warm-up | 10 seconds |
+| Measurement | 30 seconds |
+| Delivery grace | 3 seconds |
+| Minimum payload | 256 bytes |
+
+Override settings with environment variables:
+
+```bash
+BENCH_CLIENTS=500 \
+BENCH_RATE=1000 \
+BENCH_WARMUP=30s \
+BENCH_DURATION=2m \
+BENCH_PAYLOAD_BYTES=1024 \
+scripts/run-benchmark.sh all
+```
+
+Useful variables are `BENCH_CLIENTS`, `BENCH_RATE`, `BENCH_GROUP_SENDERS`, `BENCH_WARMUP`, `BENCH_DURATION`, `BENCH_DELIVERY_GRACE`, `BENCH_PAYLOAD_BYTES`, and `BENCH_OUTPUT_DIR`.
+
+To avoid conflicting with local development services, the runner uses host ports `19001` (API), `19002` (navigator), `19003` (WebSocket), `18090` (admin), `16060` (pprof), and `13306` (MySQL). Override them with the matching `*_HOST_PORT` variables. Containers still use JuggleIM's standard ports over the internal Compose network.
+
+## Running the CLI against an isolated environment
+
+The CLI reads the application secret only from `JIM_BENCH_APP_SECRET`, keeping it out of command arguments and result files:
+
+```bash
+export JIM_BENCH_APP_KEY='benchmark-app'
+export JIM_BENCH_APP_SECRET='replace-with-isolated-app-secret'
+
+go run ./cmd/jimbench \
+  --scenario private \
+  --clients 100 \
+  --rate 500 \
+  --warmup 30s \
+  --duration 2m \
+  --payload-bytes 256
+```
+
+Loopback targets are enforced by default. `--allow-non-loopback` exists only for a dedicated, isolated benchmark environment. Never use it with shared staging, public, or production infrastructure.
+
+## Reproduction checklist
+
+Before publishing a result:
+
+1. Use a clean, named server commit and include the generated JSON files.
+2. Run the load generator on a separate host for serious capacity measurements.
+3. Record server and database hardware separately in `--environment` and `--database` labels.
+4. State container CPU/memory limits, MySQL configuration, network topology, TLS status, and storage type.
+5. Run at least three trials after warm-up and publish every trial, not only the best one.
+6. Report errors and saturation behavior alongside throughput.
+7. Keep private-chat and group fan-out results separate.
+8. Use server-side CPU, memory, network, disk, database, and pprof data to explain bottlenecks.
+
+Example for a dedicated environment:
+
+```bash
+go run ./cmd/jimbench \
+  --scenario group \
+  --ws-url wss://benchmark-im.example.internal \
+  --api-url https://benchmark-api.example.internal/apigateway \
+  --allow-non-loopback \
+  --clients 1000 \
+  --group-senders 10 \
+  --rate 1000 \
+  --warmup 1m \
+  --duration 5m \
+  --environment 'server: 8 vCPU/16 GiB, load generator: 8 vCPU/16 GiB, 10 GbE, TLS enabled' \
+  --database 'MySQL 8.0.42, 8 vCPU/32 GiB, local NVMe, configuration linked in report notes'
+```
+
+## Interpreting results
+
+- ACK throughput is the rate at which JuggleIM accepts publishes; it is not the same as recipient deliveries.
+- Delivery throughput counts callbacks observed by connected clients. Group fan-out can therefore be much larger than publish throughput.
+- ACK latency includes client serialization, WebSocket transport, server processing, and the ACK return path.
+- Delivery latency includes the full sender-to-recipient path but assumes synchronized timing because the load generator hosts all benchmark clients.
+- A local Compose result is a development baseline, not a production capacity claim.
+
+Do not compare JuggleIM with another project unless both systems use equivalent delivery semantics, persistence, acknowledgements, payloads, fan-out, hardware, databases, and test duration.
+
+## Development checks
+
+```bash
+go test ./simulator/benchmark ./cmd/jimbench
+go test -race ./simulator/benchmark
+go vet ./simulator/benchmark ./cmd/jimbench
+```

README.md

@@ -100,6 +100,8 @@ JuggleIM runs as a modular Go service: HTTP and WebSocket gateways route request
 
 Read the **[architecture guide](./docs/architecture.md)** for component boundaries, data ownership, private and group message flows, security boundaries, and deployment constraints. [简体中文版](./docs/architecture_zh.md)
 
+Evaluate performance with the **[reproducible benchmark harness](./BENCHMARKS.md)**, which reports connection setup separately from private-chat and group-chat ACK and delivery throughput, including P50/P95/P99 latency and machine-readable results.
+
 ## 🚀 Quick Start with Docker
 
 Run a complete local JuggleIM stack with MySQL and the admin console:

README_zh.md

@@ -102,6 +102,8 @@ JuggleIM 以模块化 Go 服务运行：HTTP 和 WebSocket 网关通过内部 Ac
 
 完整的组件边界、数据职责、私聊与群聊链路、安全边界和部署约束，请阅读 **[架构文档](./docs/architecture_zh.md)**。[English version](./docs/architecture.md)
 
+可通过 **[可复现性能测试工具](./BENCHMARKS.md)** 分别评估连接建立、私聊和群聊的 ACK 与端到端投递吞吐，并生成包含 P50/P95/P99 延迟及运行环境的机器可读结果。
+
 ## 🚀 使用 Docker 快速开始
 
 通过 Docker Compose 一次启动 MySQL、JuggleIM 服务和管理后台：

cmd/jimbench/main.go

@@ -0,0 +1,81 @@
+package main
+
+import (
+	"context"
+	"flag"
+	"fmt"
+	"os"
+	"os/signal"
+	"syscall"
+
+	"im-server/simulator/benchmark"
+)
+
+func main() {
+	config := benchmark.DefaultConfig()
+	var scenario string
+	flag.StringVar(&scenario, "scenario", string(config.Scenario), "workload scenario: private or group")
+	flag.StringVar(&config.WSURL, "ws-url", config.WSURL, "IM WebSocket base URL")
+	flag.StringVar(&config.APIURL, "api-url", config.APIURL, "signed server API base URL")
+	flag.StringVar(&config.AppKey, "app-key", os.Getenv("JIM_BENCH_APP_KEY"), "benchmark application key")
+	flag.IntVar(&config.Clients, "clients", config.Clients, "number of connected clients")
+	flag.IntVar(&config.GroupSenders, "group-senders", config.GroupSenders, "number of group-message senders")
+	flag.IntVar(&config.Rate, "rate", config.Rate, "aggregate target messages per second")
+	flag.DurationVar(&config.Warmup, "warmup", config.Warmup, "warm-up duration excluded from measurements")
+	flag.DurationVar(&config.Duration, "duration", config.Duration, "measurement duration")
+	flag.DurationVar(&config.DeliveryGrace, "delivery-grace", config.DeliveryGrace, "time to collect late deliveries after sending stops")
+	flag.IntVar(&config.PayloadBytes, "payload-bytes", config.PayloadBytes, "minimum JSON message payload size")
+	flag.IntVar(&config.SetupConcurrency, "setup-concurrency", config.SetupConcurrency, "concurrent user registration requests")
+	flag.IntVar(&config.ConnectConcurrency, "connect-concurrency", config.ConnectConcurrency, "concurrent WebSocket connection attempts")
+	flag.BoolVar(&config.StoreMessages, "store-messages", config.StoreMessages, "set the stored-message flag")
+	flag.BoolVar(&config.CountMessages, "count-messages", config.CountMessages, "set the counted-message flag")
+	flag.BoolVar(&config.AllowNonLoopback, "allow-non-loopback", false, "allow an isolated non-loopback target (never use against production)")
+	flag.StringVar(&config.OutputPath, "output", "", "JSON result path (default: benchmark-results/<run>-<scenario>.json)")
+	flag.StringVar(&config.EnvironmentLabel, "environment", envOr("JIM_BENCH_ENVIRONMENT", config.EnvironmentLabel), "human-readable deployment/configuration label")
+	flag.StringVar(&config.DatabaseLabel, "database", envOr("JIM_BENCH_DATABASE", config.DatabaseLabel), "database engine and configuration label")
+	flag.Parse()
+
+	config.Scenario = benchmark.Scenario(scenario)
+	config.AppSecret = os.Getenv("JIM_BENCH_APP_SECRET")
+	config.ServerCommit = os.Getenv("JIM_BENCH_SERVER_COMMIT")
+	runner, err := benchmark.NewRunner(config)
+	if err != nil {
+		exitf("invalid benchmark configuration: %v", err)
+	}
+	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
+	defer stop()
+
+	fmt.Printf("Preparing %s benchmark: clients=%d rate=%d/s warmup=%s duration=%s payload>=%d bytes\n",
+		config.Scenario, config.Clients, config.Rate, config.Warmup, config.Duration, config.PayloadBytes)
+	report, err := runner.Run(ctx)
+	if err != nil {
+		exitf("benchmark failed: %v", err)
+	}
+	path, err := benchmark.WriteReport(config.OutputPath, report)
+	if err != nil {
+		exitf("save benchmark report: %v", err)
+	}
+
+	fmt.Printf("Benchmark complete.\n")
+	fmt.Printf("  Connections: %d/%d, P95 %.2f ms\n", report.Connections.Succeeded, report.Connections.Attempted, report.Connections.Latency.P95MS)
+	fmt.Printf("  ACKs: %d succeeded, %d failed, %.2f/s, P95 %.2f ms, P99 %.2f ms\n",
+		report.MessageAcknowledgements.Succeeded, report.MessageAcknowledgements.Failed,
+		report.MessageAcknowledgements.ThroughputPerSec, report.MessageAcknowledgements.Latency.P95MS,
+		report.MessageAcknowledgements.Latency.P99MS)
+	fmt.Printf("  Deliveries: %d observed, %.2f/s, P95 %.2f ms, P99 %.2f ms\n",
+		report.Deliveries.Succeeded, report.Deliveries.ThroughputPerSec,
+		report.Deliveries.Latency.P95MS, report.Deliveries.Latency.P99MS)
+	fmt.Printf("  Result: %s\n", path)
+}
+
+func envOr(name, fallback string) string {
+	if value := os.Getenv(name); value != "" {
+		return value
+	}
+	return fallback
+}
+
+func exitf(format string, args ...any) {
+	fmt.Fprintf(os.Stderr, format+"\n", args...)
+	os.Exit(1)
+}

docs/benchmarks/results/README.md

@@ -0,0 +1,17 @@
+# Verified local smoke run
+
+These files prove that the complete benchmark path works against the repository's Docker Compose stack. They are development smoke results, not production capacity claims: the server and load generator shared one Apple M2 host, the measurement lasted only 10 seconds, and the working tree contained the new benchmark harness.
+
+Run date: 2026-07-20 UTC. Server base commit: `b1f4ff52c3e5e6da2184f6b8ccb441fe04f13e21`. Database: the repository's default MySQL 8.0 Compose service. Workload: 20 WebSocket clients, stored and counted 256-byte messages, 3-second warm-up, 10-second measurement, 2-second delivery grace, target 50 publishes/second.
+
+| Scenario | Connections | Successful ACKs | ACK rate | ACK P95 / P99 | Observed deliveries | Delivery rate | Delivery P95 / P99 |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| Private ring | 20/20 | 499/499 | 49.9/s | 2.37 / 17.09 ms | 499 | 49.9/s | 8.47 / 36.75 ms |
+| 20-member group, 2 senders | 20/20 | 499/499 | 49.9/s | 4.54 / 30.19 ms | 9,492 | 949.2/s | 35.29 / 115.88 ms |
+
+Machine-readable reports:
+
+- [`local-smoke-20260720-private.json`](./local-smoke-20260720-private.json)
+- [`local-smoke-20260720-group.json`](./local-smoke-20260720-group.json)
+
+For publishable capacity results, follow the clean-commit, separate-load-generator, multi-trial, resource-monitoring, and full-environment checklist in [`BENCHMARKS.md`](../../../BENCHMARKS.md).

docs/benchmarks/results/local-smoke-20260720-group.json

@@ -0,0 +1 @@
+{"schema_version":1,"run_id":"1784536641-70553608","generated_at":"2026-07-20T08:37:36.993036Z","started_at":"2026-07-20T08:37:21.734704Z","measurement_started_at":"2026-07-20T08:37:24.860001Z","measurement_ended_at":"2026-07-20T08:37:34.860001Z","workload":{"scenario":"group","clients":20,"group_senders":2,"target_messages_per_second":50,"warmup":"3s","duration":"10s","delivery_grace":"2s","payload_bytes":256,"store_messages":true,"count_messages":true,"websocket_url":"ws://127.0.0.1:19003","api_url":"http://127.0.0.1:19001/apigateway","environment_label":"Local Docker Compose; server and load generator share the host; no explicit container CPU or memory limits; compose_sha256=b85237996448ead20f1c45e20211f254918268a152247520ba7b73881ae13cda","database_label":"MySQL 8.0 Docker image; utf8mb4_0900_ai_ci; default Compose configuration"},"environment":{"server_commit":"b1f4ff52c3e5e6da2184f6b8ccb441fe04f13e21","working_tree_dirty":true,"os":"darwin","architecture":"amd64","go_version":"go1.25.12","cpu_model":"Apple M2","logical_cpus":8,"total_memory_bytes":17179869184},"connections":{"attempted":20,"succeeded":20,"failed":0,"throughput_per_second":0,"latency":{"count":20,"min_ms":17.579166,"p50_ms":19.765167,"p95_ms":26.728375,"p99_ms":27.584125,"max_ms":27.584125,"mean_ms":21.951995}},"message_acknowledgements":{"attempted":499,"succeeded":499,"failed":0,"throughput_per_second":49.9,"latency":{"count":499,"min_ms":0.361709,"p50_ms":0.892416,"p95_ms":4.544583,"p99_ms":30.185166,"max_ms":70.9605,"mean_ms":2.129866}},"deliveries":{"attempted":9492,"succeeded":9492,"failed":0,"throughput_per_second":949.2,"latency":{"count":9492,"min_ms":3.394,"p50_ms":7.319,"p95_ms":35.286,"p99_ms":115.884,"max_ms":158.39,"mean_ms":12.0033}},"limitations":["ACK latency uses the load generator's monotonic clock; delivery latency uses wall-clock timestamps within the same load-generator process.","Acknowledgement latency measures client publish to server ACK; delivery latency measures client publish to recipient callback.","The load generator and local Docker stack may share hardware unless the environment label says otherwise.","Results from different hardware or workload settings are not directly comparable."]}

docs/benchmarks/results/local-smoke-20260720-private.json

@@ -0,0 +1 @@
+{"schema_version":1,"run_id":"1784536624-998a7415","generated_at":"2026-07-20T08:37:19.728472Z","started_at":"2026-07-20T08:37:04.251776Z","measurement_started_at":"2026-07-20T08:37:07.392392Z","measurement_ended_at":"2026-07-20T08:37:17.392392Z","workload":{"scenario":"private","clients":20,"target_messages_per_second":50,"warmup":"3s","duration":"10s","delivery_grace":"2s","payload_bytes":256,"store_messages":true,"count_messages":true,"websocket_url":"ws://127.0.0.1:19003","api_url":"http://127.0.0.1:19001/apigateway","environment_label":"Local Docker Compose; server and load generator share the host; no explicit container CPU or memory limits; compose_sha256=b85237996448ead20f1c45e20211f254918268a152247520ba7b73881ae13cda","database_label":"MySQL 8.0 Docker image; utf8mb4_0900_ai_ci; default Compose configuration"},"environment":{"server_commit":"b1f4ff52c3e5e6da2184f6b8ccb441fe04f13e21","working_tree_dirty":true,"os":"darwin","architecture":"amd64","go_version":"go1.25.12","cpu_model":"Apple M2","logical_cpus":8,"total_memory_bytes":17179869184},"connections":{"attempted":20,"succeeded":20,"failed":0,"throughput_per_second":0,"latency":{"count":20,"min_ms":24.325417,"p50_ms":30.048125,"p95_ms":52.907875,"p99_ms":52.968208,"max_ms":52.968208,"mean_ms":32.666724}},"message_acknowledgements":{"attempted":499,"succeeded":499,"failed":0,"throughput_per_second":49.9,"latency":{"count":499,"min_ms":0.591375,"p50_ms":1.1375,"p95_ms":2.36675,"p99_ms":17.09125,"max_ms":81.083375,"mean_ms":1.742864}},"deliveries":{"attempted":499,"succeeded":499,"failed":0,"throughput_per_second":49.9,"latency":{"count":499,"min_ms":2.13,"p50_ms":5.008,"p95_ms":8.47,"p99_ms":36.751,"max_ms":128.973,"mean_ms":6.127274}},"limitations":["ACK latency uses the load generator's monotonic clock; delivery latency uses wall-clock timestamps within the same load-generator process.","Acknowledgement latency measures client publish to server ACK; delivery latency measures client publish to recipient callback.","The load generator and local Docker stack may share hardware unless the environment label says otherwise.","Results from different hardware or workload settings are not directly comparable."]}