Skip to content

docs: migration-safety for NocoDB/n8n plan + backlog status accuracy #47

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JacobHaig merged 1 commit into from
Jun 2, 2026
Merged

docs: migration-safety for NocoDB/n8n plan + backlog status accuracy #47

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -208,10 +208,10 @@ Semaphore templates are managed as code in `platform/semaphore/templates.yml`.
- **Semaphore pipeline** — 25+ task templates, SSH key auth
- **NetBox deployed** — full stack with Diode discovery pipeline, orb-agent with OpenBao vault integration, 32 IPs + pfSense device discovered
- **Composable automation** — manage-secrets, manage-diode-credentials, manage-approle, deploy-orb-agent all working
- **pfSense sync** — runs as an orb-agent worker on a 15-minute cadence (no separate playbook); `platform/services/netbox/deployment/lib/pfsense-sync.py`

### In Progress
- NocoDB and n8n deployment via composable pattern
- pfSense sync as independent scheduled workflow (every 15 min)
- NocoDB and n8n deployment via composable pattern — both are deployed today via the **legacy** `deploy.sh` path (bash-generated secrets in an on-VM `secrets/` dir). Migration to the composable pattern is planned in `plan/development/nocodb-n8n-composable-migration.md` but **execution is HELD**: it's an in-place migration of live services with stateful secrets (n8n `N8N_ENCRYPTION_KEY`, NocoDB JWT, Postgres passwords) that must be pre-seeded into OpenBao before cutover or they'd be regenerated — needs live OpenBao access first (see that plan's "Migration Safety").
Comment on lines 213 to +214
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Jun 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Add a blank line before ### In Progress to satisfy markdownlint MD022.

Insert one empty line between the last bullet in “Completed” and the ### In Progress heading.

🧰 Tools
🪛 markdownlint-cli2 (0.22.1)

[warning] 213-213: Headings should be surrounded by blank lines
Expected: 1; Actual: 0; Below

(MD022, blanks-around-headings)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@CLAUDE.md` around lines 213 - 214, Add a single blank line immediately before
the "### In Progress" heading in CLAUDE.md (i.e., insert an empty line between
the last bullet of the "Completed" section and the line that starts with "### In
Progress") to satisfy markdownlint rule MD022; ensure you only add one newline
and do not alter surrounding text or bullets.

- Dedicated orb-agent AppRole — provisioning is now code-managed via `provision-orb-agent-approle.yml` (creates the scoped policy + AppRole from `orb-agent.hcl`, stores creds at `secret/services/approles/orb-agent`); pending a run against live OpenBao to replace the manually-created credentials

### Planned
Expand Down
28 changes: 27 additions & 1 deletion plan/development/nocodb-n8n-composable-migration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: NocoDB & n8n Composable Deployment Migration
authors: ["@jestyr27"]
date: 2026-05-10
status: Draft
status: Draft — EXECUTION HELD (see "Migration Safety" below)
tags: [nocodb, n8n, composable-deployment, ansible, openbao]
---

Expand All @@ -14,6 +14,31 @@ tags: [nocodb, n8n, composable-deployment, ansible, openbao]

NocoDB and n8n currently use legacy deploy scripts that generate secrets in bash and store them in `secrets/` directories on disk. This violates the composable deployment pattern where OpenBao is the single source of truth for credentials.

## Migration Safety — preserving existing stateful secrets (READ FIRST)

> **Status: execution HELD as of 2026-06-02.** These services are **already deployed and in use**, so this is an *in-place migration of live state*, not a greenfield deploy. Several secrets are **stateful** — regenerating them on cutover causes data loss or breakage, not just a rotation:
>
> | Secret | If regenerated on a live service |
> |--------|----------------------------------|
> | **n8n `N8N_ENCRYPTION_KEY`** | **All stored workflow credentials become permanently undecryptable.** Unrecoverable. |
> | NocoDB `NC_AUTH_JWT_SECRET` | All existing sessions/JWTs invalidated (users logged out; any JWT-derived state broken). |
> | Postgres passwords (nocodb pg; n8n admin + non-root) | The running Postgres was *initialised* with the current password — a new one fails auth against the existing data volume. |
>
> The task plan below uses `_secret_definitions: type: random`, which `manage-secrets.yml` only generates **when the key is absent** at its OpenBao path. That is correct for a clean deploy, but on an in-place migration it will regenerate every secret that isn't already present *at the exact path/key-name `manage-secrets.yml` reads* (`secret/services/<svc>`, keyed by the `_secret_definitions` `name`). The legacy `deploy.sh` stored these under a VM-local `secrets/` dir and under different OpenBao keys — so a naive first run **would** regenerate them.
>
> **Therefore execution is held until we can read the live OpenBao + VM secret state** and complete the prerequisite below. Do NOT run Task 1+ against a live nocodb/n8n host before that.

### Task 0 (prerequisite, on the live cluster): capture and pre-seed existing secrets

Before the first composable deploy of an existing service:

1. On the live VM, read the current values from `config/<svc>.env` (and/or the legacy `secrets/` dir): for n8n — `N8N_ENCRYPTION_KEY`, `POSTGRES_PASSWORD`, `POSTGRES_NON_ROOT_PASSWORD`; for nocodb — the `NC_DB`/`POSTGRES_PASSWORD` and `NC_AUTH_JWT_SECRET`.
2. Write each into OpenBao at the **exact** path + key the new `_secret_definitions` expect (`secret/data/services/<svc>` with keys `postgres_password`, `jwt_secret`, `encryption_key`, `user_password`, …). `tasks/seed-discovery-credentials.yml` is the existing pattern for idempotent copy-into-OpenBao.
3. Verify with `check-secrets.yml` that every `type: random` definition resolves to a **pre-existing** value, so the first `manage-secrets.yml` run *fetches* rather than *generates*.
4. Only then run `deploy-<svc>.yml`. Diff the rendered `.env` against the current one before `compose up` to confirm no stateful value changed.

(For a genuinely fresh host with no data, Task 0 is a no-op and `type: random` correctly generates.)

## Design Principles

1. **OpenBao as source of truth** -- All credentials generated, stored, and fetched via OpenBao
Expand Down Expand Up @@ -1156,3 +1181,4 @@ Total: 7 files created, 5 files modified, 9 commits.
| Date | Change |
| --- | --- |
| 2026-05-10 | Initial creation. 9-task migration plan for NocoDB and n8n composable deployments. |
| 2026-06-02 | Added "Migration Safety" + Task 0 (pre-seed existing stateful secrets) after an audit found the `type: random` definitions would regenerate the live n8n encryption key / NocoDB JWT / Postgres passwords on cutover. Status set to EXECUTION HELD pending live OpenBao access to complete Task 0. |
Loading