docs: add developer guides for recent subsystems

CONTRIBUTING.md

@@ -1,3 +1,7 @@
+## Developer documentation
+
+In-repo guides for contributors and operators live in [`docs/`](docs/README.md): configuration, runners, executors, workflows, API pagination, and HA cluster operations.
+
 ## Pull Requests
 
 When creating a pull-request you should:

README.md

@@ -92,6 +92,7 @@ For more installation options, visit our [Installation page](https://semaphoreui
 
 * [User Guide](https://docs.semaphoreui.com)
 * [API Reference](https://semaphoreui.com/api-docs)
+* [Developer docs](docs/README.md) — configuration, runners, workflows, and operational guides (in-repo)
 * [Postman Collection](https://www.postman.com/semaphoreui)
 
 ## Awesome Semaphore

api-docs.yml

@@ -3449,10 +3449,23 @@ paths:
     get:
       tags:
         - task
-      summary: Get last 200 Tasks related to current project
+      summary: Get recent tasks for the current project (keyset pagination)
+      parameters:
+        - name: count
+          in: query
+          type: integer
+          description: Page size (default and max 200). Legacy alias `limit` is also accepted.
+        - name: before
+          in: query
+          type: integer
+          description: Cursor — return tasks with id strictly less than this value (older page).
       responses:
         200:
-          description: Array of tasks in chronological order
+          description: Array of tasks ordered by id descending (newest first). Check X-Has-Next header for more pages.
+          headers:
+            X-Has-Next:
+              type: string
+              description: "true if older tasks exist beyond this page; false otherwise"
           schema:
             type: array
             items:

docs/README.md

@@ -0,0 +1,14 @@
+# Developer documentation
+
+Internal guides for contributors and operators. User-facing product docs live at [docs.semaphoreui.com](https://docs.semaphoreui.com).
+
+| Guide | Audience | Covers |
+|-------|----------|--------|
+| [Configuration](configuration.md) | Developers, operators | `config.json` / `config.yaml`, env vars, JSON Schema |
+| [Runners and tags](runners-and-tags.md) | Developers, operators | Remote runners, tag routing, webhooks, fleet timeouts |
+| [Runner executors](runner-executors.md) | Operators | Local, Docker, and Kubernetes task execution on runners |
+| [Workflows](workflows.md) | Developers (Pro) | DAG workflows, approvals, artifacts, API overview |
+| [Tasks API pagination](tasks-api-pagination.md) | Developers, API consumers | Keyset pagination for project task history |
+| [Cluster dashboard](cluster-dashboard.md) | Operators (HA) | Admin cluster API, task state inspection, recovery |
+
+Implementation plans for upcoming work are under [`AGENTS/plans/`](../AGENTS/plans/).

docs/cluster-dashboard.md

@@ -0,0 +1,109 @@
+# Cluster dashboard (HA)
+
+The cluster dashboard is an **admin-only** UI and API for inspecting high-availability (HA) deployments and the shared task state backend. It requires the enterprise HA feature (`features.high_availability`).
+
+## When it applies
+
+| `ha.enabled` | Dashboard |
+|--------------|-----------|
+| `false` | UI shows HA disabled; `GET /api/cluster` returns `{"ha_enabled": false}` only |
+| `true` | Full node list, Redis stats, task snapshot, maintenance clear |
+
+Configure HA in the server config:
+
+```yaml
+ha:
+  enabled: true
+  node_id: semaphore-1   # optional; auto-generated if empty
+  redis:
+    addr: redis.example.com:6379
+    pass: "<secret>"
+```
+
+`util.HAEnabled()` is true when `ha` is set and `ha.enabled` is true.
+
+## Admin API
+
+All routes require an authenticated **admin** session (same as other `/api/...` admin routes).
+
+### `GET /api/cluster`
+
+Returns cluster status:
+
+- `ha_enabled` (boolean) — always present
+- `node_id` (string) — this instance, when HA config exists
+- `nodes` (array) — peer nodes, heartbeats, versions (when HA overlay is active)
+- `redis` (object) — connection, memory, key groups (when inspector available)
+
+When HA is enabled but the cluster inspector is unavailable, the handler responds with **503** and a short error message. When HA is disabled, the response is **200** with only `ha_enabled: false` (no error).
+
+### `GET /api/cluster/tasks`
+
+Returns a **task state snapshot** from the task pool store:
+
+| Field | Meaning |
+|-------|---------|
+| `queue` | Tasks waiting to start |
+| `running` | Tasks currently executing |
+| `active_by_project` | Per-project active task records |
+| `aliases` | Alias string → task ID |
+| `claims` | Task IDs claimed for distributed coordination |
+
+Works in non-HA mode too (in-memory store); fields may be empty arrays/objects if the store does not implement introspection.
+
+### `DELETE /api/cluster/tasks`
+
+Maintenance: clear selected record groups from the backend (Redis in HA). Body:
+
+```json
+{
+  "scope": {
+    "queue": true,
+    "running": false,
+    "active": false,
+    "aliases": false,
+    "claims": false,
+    "runtime_fields": false
+  }
+}
+```
+
+At least one scope flag must be `true`. Use only when recovering from a stuck cluster state (orphaned queue entries, stale claims). Clearing **running** or **active** while real tasks execute can cause inconsistent behavior.
+
+The UI exposes the same scope checkboxes under **Clear tasks from Redis** (enabled only when `ha_enabled` is true).
+
+## UI entry
+
+**Admin → Cluster dashboard** (`web/src/views/Cluster.vue`):
+
+- Node table and Redis memory chart when HA is active
+- Live task tables from `/api/cluster/tasks`
+- Upgrade prompt when `features.high_availability` is false
+
+## Architecture sketch
+
+```mermaid
+flowchart LR
+  subgraph nodes [Semaphore nodes]
+    N1[Node A]
+    N2[Node B]
+  end
+  Redis[(Redis task state)]
+  N1 --> Redis
+  N2 --> Redis
+  Admin[Admin UI] --> API["/api/cluster*"]
+  API --> N1
+```
+
+`TaskStateStore` implementations may expose `TaskStateInspector` for snapshots and `ClearTasks`. See `services/tasks/task_state_store.go`.
+
+## OpenAPI
+
+Cluster endpoints are documented in `api-docs.yml` under the `cluster` tag (may be commented until Dredd hooks cover them). Regenerate the public Swagger bundle when enabling them in CI.
+
+## Related code
+
+- `api/cluster.go` — handlers
+- `api/router.go` — route registration
+- `pro_interfaces` — `ClusterInspector` for nodes/Redis
+- `services/tasks/task_state_store.go` — snapshot and clear types

docs/configuration.md

@@ -0,0 +1,113 @@
+# Configuration
+
+Semaphore reads settings from a config file, then applies environment-variable overrides and built-in defaults. The canonical field list is maintained in [`config.schema.yaml`](../config.schema.yaml) (JSON Schema draft 2020-12), generated from `util.ConfigType` in Go.
+
+## File format and discovery
+
+Supported formats: **JSON** (`.json`) and **YAML** (`.yaml`, `.yml`). Keys use `snake_case` and match the `json` struct tags in `util/config.go`.
+
+### Search order
+
+When `--config` is not passed and `SEMAPHORE_CONFIG_PATH` is unset, the server looks for the first existing file among:
+
+1. `./config.json`, `./config.yaml`, `./config.yml` (current working directory)
+2. `/usr/local/etc/semaphore/config.{json,yaml,yml}`
+3. `/etc/semaphore/config.{json,yaml,yml}`
+
+Explicit path:
+
+```bash
+./bin/semaphore server --config /etc/semaphore/config.yaml
+# or
+export SEMAPHORE_CONFIG_PATH=/etc/semaphore/config.yaml
+```
+
+Interactive setup (`semaphore setup`) still writes `config.json` by default; YAML is fully supported for hand-written or GitOps-managed installs.
+
+### Load order
+
+`util.ConfigInit` applies settings in this order (later steps win):
+
+1. Config file (if present and not disabled with `--no-config`)
+2. Environment variables (`SEMAPHORE_*`, see `env:` tags on struct fields)
+3. Defaults from struct `default:` tags
+
+Sensitive values can be loaded from companion files (for example `runner.token_file`, `subscription.key_file`) after the main file is parsed.
+
+## Schema validation
+
+Use `config.schema.yaml` in your editor (YAML language server with JSON Schema) or in CI to validate configs before deploy. The schema `$id` is `https://semaphoreui.com/schemas/config.schema.json`.
+
+To regenerate the schema after changing `util.ConfigType`, follow [`.claude/skills/semaphore-config-schema/SKILL.md`](../.claude/skills/semaphore-config-schema/SKILL.md).
+
+## Common options (quick reference)
+
+| Area | Keys | Notes |
+|------|------|-------|
+| Database | `dialect`, `mysql` / `postgres` / `sqlite` | BoltDB was removed in 2.19; use `sqlite` for embedded DB |
+| HTTP | `port`, `interface`, `web_host` | `web_host` is the public URL used in links and emails |
+| TLS | `tls.enabled`, `tls.cert_file`, `tls.key_file` | Optional HTTP→HTTPS redirect via `tls.http_redirect_addr` **or** `tls.http_redirect_port` (mutually exclusive) |
+| Auth | `mfa.totp`, `mfa.email` | Former top-level `auth` was renamed to `mfa` |
+| Runners | `use_remote_runner`, `runner_registration_token`, `runner`, `runners` | `runner` configures a runner process; `runners` configures server-side fleet timeouts |
+| HA | `ha.enabled`, `ha.node_id`, `ha.redis` | Requires enterprise overlay; see [Cluster dashboard](cluster-dashboard.md) |
+| Concurrency | `max_parallel_tasks` | Server-wide cap; per-runner limit is `runner.max_parallel_tasks` |
+
+Environment variable names mirror keys: `port` → `SEMAPHORE_PORT`, nested fields use underscores (`SEMAPHORE_TLS_ENABLED`, `SEMAPHORE_HA_REDIS_ADDR`). Fields tagged `sensitive` are cleared from the process environment after load so secrets do not leak to child processes.
+
+## Examples
+
+### Minimal development (SQLite)
+
+```yaml
+dialect: sqlite
+sqlite:
+  host: /tmp/semaphore.db
+port: ":3000"
+tmp_path: /tmp/semaphore
+cookie_hash: <base64-32-bytes>
+cookie_encryption: <base64-32-bytes>
+access_key_encryption: <base64-32-bytes>
+```
+
+Generate secrets with `semaphore setup` or `openssl rand -base64 32`.
+
+### TLS with HTTP redirect
+
+```yaml
+tls:
+  enabled: true
+  cert_file: /etc/semaphore/tls.crt
+  key_file: /etc/semaphore/tls.key
+  http_redirect_port: 8080
+```
+
+A second listener on port `8080` redirects clients to HTTPS. Use `http_redirect_addr` instead when you need a non-default bind address (for example `:8080` or `127.0.0.1:8080`).
+
+### Remote runner (server side)
+
+```yaml
+use_remote_runner: true
+runner_registration_token: "<admin-generated-token>"
+runners:
+  offline_timeout_sec: 120
+  task_fail_timeout_sec: 420
+  reconcile_interval_sec: 30
+```
+
+Runners register with that token; task routing uses project/global runners and optional tags (see [Runners and tags](runners-and-tags.md)).
+
+## Troubleshooting
+
+| Symptom | Check |
+|---------|--------|
+| Server exits on start | Run with explicit `--config`; validate against `config.schema.yaml` |
+| Wrong database | `dialect` and the matching `mysql`/`postgres`/`sqlite` block |
+| Broken login cookies after config change | `cookie_hash` / `cookie_encryption` must stay stable or all sessions invalidate |
+| Runner never picks up jobs | `use_remote_runner`, runner `active`, tag match on template/inventory |
+| HA features missing in UI | `ha.enabled` and enterprise subscription; cluster API returns `ha_enabled: false` when disabled |
+
+## Related code
+
+- `util/config.go`, `util/config_auth.go` — struct definitions and loading
+- `util/config_test.go` — YAML/JSON load tests
+- `cli/cmd/root.go` — `--config`, `--no-config` flags