Vanguard

Real-time multiplayer quiz platform. A host creates a session, participants join from any device, and everyone receives live question broadcasts, answer feedback, and leaderboard updates over WebSocket.

Live: tryit.selnastol.ru

---

Architecture

Frontend (React)
    │
    ├── REST ──► auth / quiz / session services
    └── WS ───► real-time service (Go)
                        │
               ┌────────┴────────┐
           RabbitMQ           Redis 7
        (event bus)     (state + pub/sub)

Services: auth, quiz, session handle HTTP and persist to PostgreSQL. The real-time service is stateless — all session state lives in Redis so any number of instances can run behind a load balancer.

Real-Time Service

When a host advances to the next question, the full delivery path is:

session service
  └─ AMQP publish → question.{id}.start
        └─ ConsumeQuestionStart (per-session consumer)
              └─ cache.PublishBroadcast → Redis Pub/Sub channel
                    └─ SubscribeBroadcast goroutine (one per instance per active session)
                          └─ BroadcastToSession → N×send channels → N×writePump goroutines → WebSocket clients

Per-connection goroutine model — two goroutines per WebSocket connection:

• handleRead — blocks on conn.ReadMessage, processes incoming events, defers cleanup via closeOnce
• writePump — sole writer on the connection (gorilla/websocket requires single-writer), drains a buffered send channel (cap 256)

closeOnce wraps close(send) to prevent a double-close panic when graceful shutdown and handleRead's deferred teardown race on the same connection.

Reconnect detection — when a client reconnects, the handler queries Redis for their answer history and current question index. If state exists, a catch-up frame is sent before the client is registered; the client never sees a blank screen on page reload.

---

Reliability

RabbitMQ reconnect — WatchAndReconnect runs in background, monitors the connection via NotifyClose, and reconnects with exponential backoff (1s → 2s → 4s, capped at 32s). After reconnect, all active session consumers are restored from a snapshot taken before the drop.

Graceful shutdown on SIGTERM/SIGINT — ordered teardown:

1. ConnectionRegistry.CloseAll() — sends WebSocket close frames to all connected clients
2. Cancels all active per-session RabbitMQ consumer tags
3. Waits up to 30s (matches Kubernetes default grace period) before force-exit

Manual ack on all RabbitMQ consumers — messages are not acknowledged until fully processed. On consumer crash, the broker requeues; no question events are silently lost.

---

Performance

Tests run against real infrastructure via testcontainers — no mocks. Measured end-to-end: from amqp.Channel.Publish to the WebSocket frame arriving at the client.

Broadcast latency

Players	p50	p99	Delivered
500	—	14.5 ms	500/500
1 000	7 ms	11.6 ms	1000/1000
2 000	29 ms	33.7 ms	1976/2000 (98.8%)

Multi-instance fan-out

300/300 clients across 3 independent instances received the broadcast. The publishing instance had no knowledge of the other two — Redis Pub/Sub handled fan-out transparently.

Goroutine leak

0 goroutines leaked at 2 000 concurrent connections. runtime.NumGoroutine delta = exactly 2×N (handleRead + writePump per connection). Verified with -race flag at N=500.

Capacity characteristics

The per-session bottleneck is BroadcastToSession: an O(N) sequential channel-write loop followed by N goroutine wakeups on the Go scheduler. Latency is stable up to ~1 000 players (p99 ≈ 12 ms); above 2 000 the scheduler shows pressure (p99 ≈ 34 ms, ~1% delivery loss under testcontainers load).

Multiple concurrent sessions are well-isolated — each broadcasts only over its own connections, so 10 sessions × 100 players has materially lower tail latency than 1 session × 1 000 players. Estimated total per-instance capacity before scheduler saturation: 10 000–15 000 concurrent connections.

To push the per-session limit past ~2 000: replace the sequential SendMessage loop with a goroutine pool, and shard ConnectionRegistry by session hash to remove cross-session mutex contention.

---

Integration Tests

Requires Docker. testcontainers starts Redis and RabbitMQ automatically.

cd backend-go
go test -v -tags=integration -race -timeout 300s ./real_time/tests/

File	What it covers
integration_test.go	Full game flow: join → question → answer → leaderboard → session end
regression_test.go	Targeted tests for previously fixed bugs
metrics_test.go	Goroutine leak (N=100), latency p99 (N=50), multi-instance fan-out (2 instances)
highload_test.go	Goroutine leak (N=500, -race), latency (N=500), 3-instance × 100 players
stress_test.go	Scaling curve: 2 000 connection leak, latency at 1 000 and 2 000 players

---

Running

Development

cp .env.dev.example .env
docker compose --env-file .env -f docker-compose.yaml -f docker-compose.dev.yaml up -d

Frontend: http://localhost:3000 · Grafana: http://localhost:3001

Production

cp .env.prod.example .env
chmod +x setup.sh && ./setup.sh          # SSL certificates — first deploy only
docker compose --env-file .env -f docker-compose.yaml -f docker-compose.prod.yaml up -d

---

Stack

Real-time service	Go 1.24, gorilla/websocket
Message bus	RabbitMQ 3 — amqp091-go, manual ack, exponential backoff reconnect
State & broadcast	Redis 7 — Pub/Sub fan-out, answer/question state
Frontend	React
Database	PostgreSQL 15
Observability	Prometheus, Grafana, Loki, Promtail
Integration tests	testcontainers-go