From d02a224b7350ad7f2051c0bfb56745a2f2111358 Mon Sep 17 00:00:00 2001 From: Wisward Date: Mon, 1 Jun 2026 15:34:11 -0400 Subject: [PATCH 1/3] docs: record Milestone A completion + deployment phase progress - Add repo-memory tracking the 7-phase deployment progress (Milestone A done: phases 1-4 merged, image published to GHCR; phases 5-7 remain). - Mark Milestone A complete in the deployment plan. --- .claude/memory/MEMORY.md | 1 + .../project-deployment-phase-progress.md | 22 +++++++++++++++++++ ...-06-01-agent-cloud-deployment-alignment.md | 2 ++ 3 files changed, 25 insertions(+) create mode 100644 .claude/memory/project-deployment-phase-progress.md diff --git a/.claude/memory/MEMORY.md b/.claude/memory/MEMORY.md index b84a5fd..ac696f8 100644 --- a/.claude/memory/MEMORY.md +++ b/.claude/memory/MEMORY.md @@ -3,3 +3,4 @@ - [No co-authors in commits](feedback-no-coauthors.md) — never add Co-Authored-By trailers to commit messages - [agent-cloud deployment](project-agent-cloud-deployment.md) — WisBot deploys via agent-cloud as a pulled image; secrets in OpenBao, site values in site-config - [CodeRabbit PR workflow](feedback-coderabbit-pr-workflow.md) — all merges go through a PR; merge only when CodeRabbit fully passes; resolve findings + re-review first +- [Deployment phase progress](project-deployment-phase-progress.md) — Milestone A (phases 1–4) merged + image published; phases 5–7 (agent-cloud + site-config) remain diff --git a/.claude/memory/project-deployment-phase-progress.md b/.claude/memory/project-deployment-phase-progress.md new file mode 100644 index 0000000..c35c1f1 --- /dev/null +++ b/.claude/memory/project-deployment-phase-progress.md @@ -0,0 +1,22 @@ +--- +name: project-deployment-phase-progress +description: Status of the WisBot → agent-cloud deployment alignment (7-phase plan) +metadata: + node_type: memory + type: project +--- + +Tracking the phased migration in [[project-agent-cloud-deployment]]. Full plan: `docs/plans/2026-06-01-agent-cloud-deployment-alignment.md`. Workflow per phase = one branch → PR → CodeRabbit → merge (see [[feedback-coderabbit-pr-workflow]]). + +**Milestone A — WisBot repo: COMPLETE (merged to main as of 2026-06-01)** +- Phase 1 (#9) — config externalized to env vars (`WISBOT_GUILD_ID`, DB/recordings paths, etc.); `Config.Load` reads env → `.env` → default. +- Phase 2 (#10) — HTTP `/health` endpoint (`HttpListener`, `WISBOT_HEALTH_HOST`/`PORT`). +- Phase 3 (#11) — multi-stage Dockerfile; Linux voice via apt `libopus0` (libsodium/SQLite natives come from NuGet); `docker-build.yml` validates build on PRs. +- Phase 4 (#12) — `build-and-publish.yml` pushes to `ghcr.io/uhstray-io/wisbot` on main/tags; legacy self-hosted deploy workflows removed (`deploy-o11y.yml` kept). **First GHCR publish succeeded.** + +**Remaining — agent-cloud + site-config repos:** +- Phase 5 — `agents/wisbot/` deploy dir (compose pulls the image, deploy.sh container-only, `wisbot.env.j2`) + `platform/tests/test_service_wisbot.bats`. *(agent-cloud, AI-agent tier — note: agents live under `agents/`, NOT `platform/services/`.)* +- Phase 6 — composable `deploy-wisbot.yml` (NetBox-style: sparse-checkout → manage-secrets → run-deploy → verify-health) + Semaphore template. +- Phase 7 — site-config inventory entry + seed `secret/services/wisbot` in OpenBao + go-live. **BLOCKED on site-config access + the real guild ID / WisAI Ollama endpoint.** + +**How to apply:** Pick up at Phase 5 in the agent-cloud repo (`/Users/jacobhaig/Documents/GitHub/agent-cloud`). Watch the shared CodeRabbit rate limit — space PRs out. No runtime OpenBao AppRole needed for WisBot (deploy-time token only). WisBot needs no Caddy route (internal `/health`). diff --git a/docs/plans/2026-06-01-agent-cloud-deployment-alignment.md b/docs/plans/2026-06-01-agent-cloud-deployment-alignment.md index c35b94f..1cc20b9 100644 --- a/docs/plans/2026-06-01-agent-cloud-deployment-alignment.md +++ b/docs/plans/2026-06-01-agent-cloud-deployment-alignment.md @@ -166,6 +166,8 @@ The agent-cloud PR is gated by `.github/workflows/lint-and-test.yml` + pre-commi ## 7. Phased Execution +> **Status (2026-06-01):** Milestone A complete — Phases 1 (#9), 2 (#10), 3 (#11), 4 (#12) all merged to `main`; image published to `ghcr.io/uhstray-io/wisbot`. Remaining: Phases 5–6 (agent-cloud repo), Phase 7 (site-config, blocked on access). + **Workflow per phase (mandatory):** branch from `main` → implement → `dotnet build` green → push → open PR → **wait for CodeRabbit + CI** → address findings → confirm checks pass → merge → delete branch. One phase = one branch = one PR. No phase starts until the prior phase is merged (phases are ordered to avoid conflicts). No `Co-Authored-By` trailers (per repo memory). Phases are ordered **lowest-risk / prerequisite first**, so each PR is small, independently reviewable, and leaves `main` shippable. From 2522d187b5c44506beb6fb738ddcd3450a205765 Mon Sep 17 00:00:00 2001 From: Wisward Date: Tue, 2 Jun 2026 00:08:16 -0400 Subject: [PATCH 2/3] docs: refresh phase progress + add Phase 8 (file relay) spec - Fix machine-specific absolute path in the progress memory (CodeRabbit #13). - Update progress: Milestone B (Phase 5 #43 merged, Phase 6 #45 merged); Phase 7 + new Phase 8 remain. - Add Phase 8 spec to the plan: /upload -> unguessable link -> web upload (<=500MB) -> same link downloads, 30-day retention. Decisions: MinIO + DB metadata, built into WisBot via ASP.NET Core/Kestrel, trust-the-link access, public Caddy route. Sub-phases 8a/8b/8c. - Fix appendix secret path -> secret/services/wisbot. --- .../project-deployment-phase-progress.md | 11 ++-- ...-06-01-agent-cloud-deployment-alignment.md | 50 ++++++++++++++++++- 2 files changed, 55 insertions(+), 6 deletions(-) diff --git a/.claude/memory/project-deployment-phase-progress.md b/.claude/memory/project-deployment-phase-progress.md index c35c1f1..df44efe 100644 --- a/.claude/memory/project-deployment-phase-progress.md +++ b/.claude/memory/project-deployment-phase-progress.md @@ -14,9 +14,12 @@ Tracking the phased migration in [[project-agent-cloud-deployment]]. Full plan: - Phase 3 (#11) — multi-stage Dockerfile; Linux voice via apt `libopus0` (libsodium/SQLite natives come from NuGet); `docker-build.yml` validates build on PRs. - Phase 4 (#12) — `build-and-publish.yml` pushes to `ghcr.io/uhstray-io/wisbot` on main/tags; legacy self-hosted deploy workflows removed (`deploy-o11y.yml` kept). **First GHCR publish succeeded.** -**Remaining — agent-cloud + site-config repos:** -- Phase 5 — `agents/wisbot/` deploy dir (compose pulls the image, deploy.sh container-only, `wisbot.env.j2`) + `platform/tests/test_service_wisbot.bats`. *(agent-cloud, AI-agent tier — note: agents live under `agents/`, NOT `platform/services/`.)* -- Phase 6 — composable `deploy-wisbot.yml` (NetBox-style: sparse-checkout → manage-secrets → run-deploy → verify-health) + Semaphore template. +**Milestone B — agent-cloud repo:** +- Phase 5 (agent-cloud #43) — **MERGED.** `agents/wisbot/` deploy dir (compose pulls the image, deploy.sh container-only, `wisbot.env.j2`) + `platform/tests/test_service_wisbot.bats`. *(AI-agent tier — agents live under `agents/`, NOT `platform/services/`.)* +- Phase 6 (agent-cloud #45) — **in review.** `deploy-wisbot.yml` (clone + manage-secrets → deploy.sh → verify-health), `clean-deploy-wisbot.yml`, Semaphore templates, `validate-all` block. Added a backward-compatible `_templates_src` override to `manage-secrets.yml` so `agents/` services can template env files. *(The composable task files in the AUTOMATION-COMPOSABILITY doc — sparse-checkout/run-deploy/verify-health — don't exist yet; real playbooks inline git+shell+uri, which deploy-wisbot mirrors.)* + +**Remaining:** - Phase 7 — site-config inventory entry + seed `secret/services/wisbot` in OpenBao + go-live. **BLOCKED on site-config access + the real guild ID / WisAI Ollama endpoint.** +- Phase 8 — **File relay service** (new, planned): `/upload` → unguessable link → web upload (≤500MB) → same link downloads; bypasses Discord's ~8MB limit; 30-day retention. Decisions: **MinIO + DB metadata** for storage, **built into WisBot via ASP.NET Core/Kestrel** (Docker base `runtime`→`aspnet`), **trust-the-link** access, **public Caddy route + subdomain**. See the plan doc's Phase 8 section. -**How to apply:** Pick up at Phase 5 in the agent-cloud repo (`/Users/jacobhaig/Documents/GitHub/agent-cloud`). Watch the shared CodeRabbit rate limit — space PRs out. No runtime OpenBao AppRole needed for WisBot (deploy-time token only). WisBot needs no Caddy route (internal `/health`). +**How to apply:** Phases 5–6 live in the local agent-cloud clone; build Phase-6-style work in an isolated git worktree to avoid colliding with concurrent work, and stage specific files (never `git add -A`). Watch the shared CodeRabbit rate limit — space PRs out. No runtime OpenBao AppRole needed for WisBot (deploy-time token only). The bot's `/health` is internal (no Caddy), but Phase 8's upload site needs a public Caddy route. diff --git a/docs/plans/2026-06-01-agent-cloud-deployment-alignment.md b/docs/plans/2026-06-01-agent-cloud-deployment-alignment.md index 1cc20b9..e72ba5d 100644 --- a/docs/plans/2026-06-01-agent-cloud-deployment-alignment.md +++ b/docs/plans/2026-06-01-agent-cloud-deployment-alignment.md @@ -194,7 +194,13 @@ Phases are ordered **lowest-risk / prerequisite first**, so each PR is small, in |---|-------|--------|-----------|-------|---------------| | **7** | Inventory + OpenBao seed + go-live | `feat/wisbot-inventory` (site-config) | WS6 | Inventory entry; seed `secret/services/wisbot`; provision/allocate VM; Semaphore branch deploy → validate → merge → redeploy from `main`. | `validate-all.yml` + `validate-secrets.yml` pass; container healthy; no secrets/IPs in public diffs | -**Cross-phase dependencies:** P2 is independent of P1 but ordered after for clean review; P3 depends on P1+P2 (Dockerfile bakes in env config + health); P4 depends on P3 (needs the Dockerfile); P5–P6 depend on P4 (image must be published to pull); P7 depends on P6. +### File relay service (Milestone D) + +| # | Phase | Branch(es) | Scope | Exit / verify | +|---|-------|-----------|-------|---------------| +| **8** | File relay (Discord upload-limit bypass) | `feat/file-relay-*` (multi) | `/upload` → unguessable link → web upload (≤500MB) → same link downloads, 30-day retention. **Full spec in §9.** | upload + download round-trip works through the public URL; >500MB rejected; expired files purged | + +**Cross-phase dependencies:** P2 is independent of P1 but ordered after for clean review; P3 depends on P1+P2 (Dockerfile bakes in env config + health); P4 depends on P3 (needs the Dockerfile); P5–P6 depend on P4 (image must be published to pull); P7 depends on P6. **P8 depends on A–C (a deployed bot + image pipeline) and on the platform MinIO + a public Caddy route.** ## 8. Per-phase verification gate @@ -202,10 +208,50 @@ Every phase must, before merge: (1) build/lint green locally, (2) pass **all** C --- +## 9. Phase 8 — File relay service (Discord upload-limit bypass) + +**Goal:** Let users share files larger than Discord's ~8 MB limit. `/upload` in Discord returns an unguessable link; the user opens it in a browser and uploads one file (≤500 MB); the **same link** then serves that file for download to anyone who has it (so they can pass it to friends). Files are retained **30 days**, then auto-deleted. + +### Decisions (locked 2026-06-02) +1. **Storage:** **MinIO object store + DB metadata** — file bytes live in a MinIO bucket (`wisbot-uploads/`), the DB holds only metadata. (Not blobs-in-DB — 500 MB blobs are an anti-pattern for SQLite/Postgres.) +2. **Service shape:** **built into WisBot** via **ASP.NET Core / Kestrel** — the bot owns the `/upload` command and the web pages. This replaces the Phase 2 `HttpListener` health server with Kestrel and flips the Docker base `dotnet/runtime` → `dotnet/aspnet`. +3. **Access:** **trust-the-link** — the unguessable ID is the credential; no login. The file is labeled with the Discord user who ran `/upload`. +4. **Exposure:** **public** via a **Caddy route + subdomain** (e.g. `up.uhstray.io`) with TLS — unlike the bot's internal `/health`. + +### Flow / design +- `/upload` → generate a cryptographically-random, URL-safe ID (≥128-bit); insert a `pending` row `{id, owner_user_id, owner_username, created_at}`; reply with `https:///u/`. +- `GET /u/{id}` → if `pending`: render the upload form; if `ready`: render a download landing page (filename, size, download button). +- `POST /u/{id}` (multipart, streamed) → **stream** to MinIO `wisbot-uploads/{id}` enforcing the **500 MB cap on the stream** (reject larger without buffering); store `{filename, content_type, size_bytes, uploaded_at, expires_at = uploaded_at + 30d}`, set `status=ready`. One file per ID (reject re-upload once ready). +- `GET /u/{id}/download` → stream from MinIO with **`Content-Disposition: attachment`** + stored content-type (never inline — avoids XSS from user files). +- **Retention loop** (background, like `ReminderService`): periodically delete MinIO objects + rows where `expires_at <= now`, and stale `pending` links never uploaded (e.g. >48 h). + +### DB schema (new `uploads` table) +`id TEXT PK · owner_user_id · owner_username · filename · content_type · size_bytes · status (pending|ready) · created_at · uploaded_at · expires_at`. SQLite works for metadata; revisit Postgres with the broader prod-HA migration. + +### Config (env, via the same Ansible-templated `.env`) +MinIO endpoint + access/secret keys (from OpenBao), bucket name, public base URL/subdomain, max size (default 500 MB), retention days (default 30), and the web listen port. + +### Security / limits +Unguessable ID; `Content-Disposition: attachment`; server-side stream size enforcement; one file per ID; light rate-limit on `/upload`; consider a per-user active-link cap to bound storage abuse (30-day retention bounds total growth). + +### Sub-phases (each its own branch/PR) +- **8a (WisBot repo):** swap health server to ASP.NET Core/Kestrel; add `/upload` command, upload/download endpoints, `uploads` DB schema, MinIO client, streamed 500 MB enforcement. Docker base → `aspnet`. +- **8b (WisBot repo):** 30-day retention cleanup loop (+ stale-pending purge). +- **8c (agent-cloud + site-config):** provision MinIO bucket + creds (OpenBao `secret/services/wisbot`), add the **Caddy route + subdomain**, extend `wisbot.env.j2` with MinIO/upload vars, publish a new image, redeploy. + +### Open items to confirm before 8a +- Subdomain/hostname for the public URL (needs DNS + Caddy). +- Whether MinIO creds live under `secret/services/wisbot` or a dedicated path; bucket lifecycle (MinIO-side 30-day expiry as a backstop to the app's cleanup?). +- Per-user link/quota caps (abuse bounding). + +--- + ## Appendix — files to add/change **WisBot repo:** `Dockerfile`, `.dockerignore`, `Wisbot.csproj` (voice libs), `Config.cs`, `Bot.cs`, `Database.cs`, `Services/VoiceRecorder.cs`, `Program.cs` (+ new health listener), `.env.example`, `CLAUDE.md`, `.github/workflows/build-and-publish.yml` (new), remove `deployment_prod.yml` + `deployment_refresh_prod.yml`. **agent-cloud repo:** `agents/wisbot/deployment/{compose.yml,deploy.sh,update.sh,validate.sh,templates/wisbot.env.j2,CLAUDE.md,README.md,.env.example}`, `agents/wisbot/context/{architecture,prompts,skills,use-cases}/.gitkeep`, `platform/playbooks/deploy-wisbot.yml`, `platform/playbooks/clean-deploy-wisbot.yml`, `platform/tests/test_service_wisbot.bats`, edits to `validate-all.yml`/`validate-secrets.yml`/`semaphore/templates.yml`, root `README.md`/`CLAUDE.md` updates. *(No `apply-policy-wisbot.yml` — no runtime AppRole.)* -**site-config repo:** `inventory/production.yml` entry; OpenBao `secret/services/discord` seed. +**site-config repo:** `inventory/production.yml` entry; OpenBao `secret/services/wisbot` seed (`discord_token`). + +**Phase 8 (file relay) adds:** WisBot repo — ASP.NET Core/Kestrel web (upload/download endpoints), `/upload` command, `uploads` DB schema, MinIO client, retention loop, Docker base → `aspnet`. agent-cloud — MinIO bucket + creds, Caddy route + subdomain, `wisbot.env.j2` MinIO/upload vars. From 123aaf4fe5dd5be3b0aac5a46f56ac98b9f278d8 Mon Sep 17 00:00:00 2001 From: Wisward Date: Tue, 2 Jun 2026 00:13:38 -0400 Subject: [PATCH 3/3] docs: mark Phase 6 merged in progress memory (CodeRabbit) --- .claude/memory/project-deployment-phase-progress.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.claude/memory/project-deployment-phase-progress.md b/.claude/memory/project-deployment-phase-progress.md index df44efe..abe73e8 100644 --- a/.claude/memory/project-deployment-phase-progress.md +++ b/.claude/memory/project-deployment-phase-progress.md @@ -16,7 +16,7 @@ Tracking the phased migration in [[project-agent-cloud-deployment]]. Full plan: **Milestone B — agent-cloud repo:** - Phase 5 (agent-cloud #43) — **MERGED.** `agents/wisbot/` deploy dir (compose pulls the image, deploy.sh container-only, `wisbot.env.j2`) + `platform/tests/test_service_wisbot.bats`. *(AI-agent tier — agents live under `agents/`, NOT `platform/services/`.)* -- Phase 6 (agent-cloud #45) — **in review.** `deploy-wisbot.yml` (clone + manage-secrets → deploy.sh → verify-health), `clean-deploy-wisbot.yml`, Semaphore templates, `validate-all` block. Added a backward-compatible `_templates_src` override to `manage-secrets.yml` so `agents/` services can template env files. *(The composable task files in the AUTOMATION-COMPOSABILITY doc — sparse-checkout/run-deploy/verify-health — don't exist yet; real playbooks inline git+shell+uri, which deploy-wisbot mirrors.)* +- Phase 6 (agent-cloud #45) — **MERGED.** `deploy-wisbot.yml` (clone + manage-secrets → deploy.sh → verify-health), `clean-deploy-wisbot.yml`, Semaphore templates, `validate-all` block. Added a backward-compatible `_templates_src` override to `manage-secrets.yml` so `agents/` services can template env files. *(The composable task files in the AUTOMATION-COMPOSABILITY doc — sparse-checkout/run-deploy/verify-health — don't exist yet; real playbooks inline git+shell+uri, which deploy-wisbot mirrors.)* **Remaining:** - Phase 7 — site-config inventory entry + seed `secret/services/wisbot` in OpenBao + go-live. **BLOCKED on site-config access + the real guild ID / WisAI Ollama endpoint.**