feat(crypto)!: derive every at-rest key from one ENCRYPTION_KEY

.claude/skills/security-audit/SKILL.md

@@ -95,6 +95,9 @@ Before reviewing, list the feature's actual surface — do not guess from the na
 
 - Randomness for anything security-bearing: `crypto/rand`, **and check the error return**.
 - Encryption at rest: `pkg/cryptoutil` (AES-256-GCM, fresh nonce per encrypt). Do not hand-roll.
+  A new use derives its own subkey from `ENCRYPTION_KEY` (`DeriveKey`, HKDF) rather than
+  reusing another purpose's key or adding a third secret. Ciphertext in a table column binds
+  to its row with `EncryptWithAAD` so a blob cannot be moved between rows or owners.
 - Tokens are stored as SHA-256 hashes, are single-use, and expire. Passwords use `pkg/hash`
   (bcrypt cost 12).
 - New config secret? Validate it at startup and **fail closed** (`cmd/api/secrets.go` is the

.env.example

@@ -2,7 +2,8 @@
 # NinerLog API — Environment Variables
 # =============================================================================
 # Copy this file to .env and adjust values for your environment.
-# All variables have sensible defaults for local development.
+# Everything has a sensible local-development default except ENCRYPTION_KEY,
+# which has to be generated — the API will not start without it.
 
 # -----------------------------------------------------------------------------
 # Database
@@ -24,6 +25,25 @@ REFRESH_SECRET=change-this-refresh-secret-in-production
 JWT_EXPIRES_IN=15m
 REFRESH_EXPIRES_IN=7d
 
+# -----------------------------------------------------------------------------
+# Encryption at rest — REQUIRED, CHANGE THIS IN PRODUCTION
+# -----------------------------------------------------------------------------
+# One 32-byte key, base64-encoded: openssl rand -base64 32
+#
+# The API will not start without it. Everything the server encrypts at rest
+# derives its own subkey from this one secret (HKDF): licence/credential files,
+# 2FA secrets, cloud backup credentials. No two of them share key bytes, and
+# none of them is this key.
+#
+# Treat it exactly like the database password, and KEEP A COPY. Data sealed with
+# it cannot be recovered without it — not from a database dump, not by us. There
+# is no reset.
+#
+# TOTP_ENCRYPTION_KEY and BACKUP_CREDENTIALS_KEY are gone. The server refuses to
+# start while either is still set, because data sealed with them cannot be read
+# under the new scheme; see docs/UPGRADING.md before removing them.
+ENCRYPTION_KEY=
+
 # -----------------------------------------------------------------------------
 # CORS (comma-separated origins)
 # -----------------------------------------------------------------------------
@@ -48,10 +68,12 @@ MIGRATIONS_PATH=db/migrations
 # Licence / credential reference files
 # -----------------------------------------------------------------------------
 # Photos, scans and PDFs attached to a licence or credential (max 5 MB, 5 per
-# document; JPEG, PNG and PDF only). Set to false to close the feature entirely
-# — uploads *and* downloads then answer 403. Stored files are kept and reappear
-# if re-enabled. DOCUMENT_IMAGES_ENABLED is the previous name for this knob and
-# is still honoured; the new name wins when both are set.
+# document; JPEG, PNG and PDF only). Requires ENCRYPTION_KEY above — stored
+# files are encrypted at rest and there is no plaintext fallback.
+# Set to false to close the feature entirely — uploads *and* downloads then
+# answer 403. Stored files are kept and reappear if re-enabled.
+# DOCUMENT_IMAGES_ENABLED is the previous name for this knob and is still
+# honoured; the new name wins when both are set.
 # DOCUMENT_FILES_ENABLED=false
 #
 # Per-user budget for READING files (listing a document's files and downloading
@@ -60,6 +82,20 @@ MIGRATIONS_PATH=db/migrations
 # shared 'expensive' bucket.
 # FILE_READ_RATE_LIMIT_PER_MINUTE=90
 
+# -----------------------------------------------------------------------------
+# Cloud backups (optional, off by default)
+# -----------------------------------------------------------------------------
+# Scheduled backups of a pilot's data to their own S3, SFTP or WebDAV storage.
+# Destination credentials are encrypted at rest under a subkey of
+# ENCRYPTION_KEY. This used to be switched on by the presence of a separate
+# backup key; with one shared key it needs its own switch, or setting that key
+# would silently start a scheduler and outbound connections.
+# CLOUD_BACKUPS_ENABLED=true
+#
+# The e2e backup targets are containers on a private network, and the SSRF guard
+# blocks private ranges by default. Only for test stacks.
+# BACKUP_ALLOW_PRIVATE_NETWORKS=true
+
 # -----------------------------------------------------------------------------
 # SMTP / Email (optional — emails are logged to stdout when SMTP_HOST is empty)
 # -----------------------------------------------------------------------------

README.md

@@ -103,6 +103,17 @@ test/e2e/                    # End-to-end tests
 
 See `.env.example` for a complete list of configuration options including database connection, JWT secrets, CORS settings, SMTP configuration, and TLS settings.
 
+**Encryption at rest.** `ENCRYPTION_KEY` (32 bytes, base64 — `openssl rand -base64 32`)
+is **required**: the API will not start without it. It protects everything the database
+stores on the application's behalf — licence and credential files, 2FA secrets, cloud
+backup credentials — with each use deriving its own subkey, so one secret covers all of
+them without any two sharing key bytes. Keep a copy alongside the database password:
+sealed data cannot be recovered without it, and there is no reset. Upgrading from a
+release with `TOTP_ENCRYPTION_KEY` or `BACKUP_CREDENTIALS_KEY`? Read
+[docs/UPGRADING.md](docs/UPGRADING.md) first — those are removed, the server refuses to
+start while they are still set, and a migration clears every 2FA enrolment, session and
+backup destination, because none of them can be decrypted any more.
+
 **Single sign-on (optional).** Setting `OIDC_ISSUER` switches the deployment to OIDC
 mode, where an external identity provider owns all accounts and NinerLog's own password,
 registration, 2FA and passkey endpoints are disabled. It is off by default. See

api-spec/openapi.yaml

@@ -6620,10 +6620,15 @@ components:
           properties:
             enabled:
               type: boolean
-              description: When false, every /files endpoint answers 403 — uploads and downloads alike
+              description: >-
+                When false, every /files endpoint answers 403 — uploads and
+                downloads alike. False when the operator switched the feature
+                off, and also when no encryption key is configured: stored
+                files are encrypted at rest and the feature does not run
+                without one.
             maxBytes:
               type: integer
-              description: Maximum size of a single image in bytes
+              description: Maximum size of a single file in bytes, measured before encryption
               example: 5242880
             maxPerDocument:
               type: integer
@@ -8283,10 +8288,14 @@ components:
           description: Whether ADMIN_EMAIL is set
         cloudBackupsConfigured:
           type: boolean
-          description: Whether cloud backups are enabled (BACKUP_CREDENTIALS_KEY is set)
+          description: Whether cloud backups are enabled (CLOUD_BACKUPS_ENABLED=true)
         documentFilesEnabled:
           type: boolean
-          description: Whether licence/credential reference files are enabled (DOCUMENT_FILES_ENABLED is not "false")
+          description: >-
+            Whether licence/credential reference files are enabled
+            (DOCUMENT_FILES_ENABLED is not "false" and ENCRYPTION_KEY is set —
+            stored files are encrypted at rest and the feature does not run
+            without a key)
         unverifiedCleanupEnabled:
           type: boolean
           description: |
@@ -9676,7 +9685,9 @@ components:
     DocumentFilesDisabled:
       description: >-
         Document files are switched off on this server
-        (DOCUMENT_FILES_ENABLED=false). Applies to reading as well as
+        (DOCUMENT_FILES_ENABLED=false, or no ENCRYPTION_KEY is configured —
+        stored files are encrypted at rest and the feature does not run
+        without a key). Applies to reading as well as
         uploading: serving stored files is the bandwidth half of the abuse
         surface the switch exists to close. Already-stored files are retained
         and become reachable again if the operator re-enables the feature.

cmd/api/env_test.go

@@ -44,3 +44,41 @@ func TestEnvIntNarrow(t *testing.T) {
 		})
 	}
 }
+
+// envBool gates subsystems that make outbound connections, so "not the string
+// false" is not good enough: a typo or a "0" must not read as "on".
+func TestEnvBool(t *testing.T) {
+	const key = "NINERLOG_TEST_ENV_BOOL"
+
+	cases := []struct {
+		name string
+		set  bool
+		val  string
+		def  bool
+		want bool
+	}{
+		{name: "unset keeps default", def: false, want: false},
+		{name: "unset keeps a true default", def: true, want: true},
+		{name: "empty keeps default", set: true, val: "", def: false, want: false},
+		{name: "true enables", set: true, val: "true", def: false, want: true},
+		{name: "1 enables", set: true, val: "1", def: false, want: true},
+		{name: "false disables", set: true, val: "false", def: true, want: false},
+		{name: "0 disables", set: true, val: "0", def: true, want: false},
+
+		// The difference from envBoolWithLegacy, which would read both of these
+		// as "on" because they are not the exact string "false".
+		{name: "no keeps default", set: true, val: "no", def: false, want: false},
+		{name: "typo keeps default", set: true, val: "ture", def: false, want: false},
+	}
+
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			if tc.set {
+				t.Setenv(key, tc.val)
+			}
+			if got := envBool(key, tc.def); got != tc.want {
+				t.Errorf("envBool(%q=%q, def=%v) = %v, want %v", key, tc.val, tc.def, got, tc.want)
+			}
+		})
+	}
+}

cmd/api/main.go

@@ -85,10 +85,6 @@ func envIntNarrow(key string, def int) int {
 	return int(v)
 }
 
-// envDuration reads a Go duration (e.g. "24h", "60s") from the environment,
-// keeping the default when the variable is unset, unparseable, or non-positive.
-// Same fail-safe reasoning as envInt: a misconfigured retention window must not
-// silently become zero.
 // envBoolWithLegacy reads a boolean feature switch, honouring a previous name
 // for the same knob. Only the exact string "false" disables — an unset or
 // unparseable value leaves the default in place, so a typo never silently
@@ -106,6 +102,32 @@ func envBoolWithLegacy(key, legacyKey string, def bool) bool {
 	return def
 }
 
+// envBool reads a boolean switch, keeping the default when the variable is
+// unset or unparseable.
+//
+// Unlike envBoolWithLegacy — which exists for opt-out knobs, where only the
+// exact string "false" disables — this parses properly, because it is used for
+// opt-in switches. "0", "no" and a typo must not all read as "on"; a subsystem
+// that starts making outbound connections should do so because someone wrote
+// "true", not because they wrote something that merely was not "false".
+func envBool(key string, def bool) bool {
+	raw := os.Getenv(key)
+	if raw == "" {
+		return def
+	}
+	v, err := strconv.ParseBool(raw)
+	if err != nil {
+		slog.Warn("Ignoring invalid environment value, using default",
+			"key", key, "value", raw, "default", def)
+		return def
+	}
+	return v
+}
+
+// envDuration reads a Go duration (e.g. "24h", "60s") from the environment,
+// keeping the default when the variable is unset, unparseable, or non-positive.
+// Same fail-safe reasoning as envInt: a misconfigured retention window must not
+// silently become zero.
 func envDuration(key string, def time.Duration) time.Duration {
 	raw := os.Getenv(key)
 	if raw == "" {
@@ -225,18 +247,48 @@ func main() {
 	licenseRepo := postgres.NewLicenseRepository(db)
 	flightRepo := postgres.NewFlightRepository(db)
 	flightBaselineRepo := postgres.NewFlightBaselineRepository(db)
-	// TOTP secrets are encrypted at rest when TOTP_ENCRYPTION_KEY (base64,
-	// 32 bytes) is set. Without it, secrets are stored as plaintext; warn so
-	// operators enable encryption in production.
-	var totpAEAD *cryptoutil.AEAD
-	if totpKey := os.Getenv("TOTP_ENCRYPTION_KEY"); totpKey != "" {
-		totpAEAD, err = cryptoutil.NewFromBase64(totpKey)
-		if err != nil {
-			fatal("invalid TOTP_ENCRYPTION_KEY", "error", err)
+	// ENCRYPTION_KEY is the single operator-facing secret behind every piece of
+	// data this server encrypts at rest. Each use derives its own subkey from
+	// it (HKDF, see cryptoutil.DeriveKey), so one key in the environment
+	// protects several independent things without any two of them sharing key
+	// bytes — recovering the subkey that reads licence scans reveals nothing
+	// about the one that reads 2FA secrets, and neither reveals the master.
+	//
+	// It is required, not optional. Every previous arrangement here had a
+	// degraded mode where a missing key meant "store it in the clear anyway",
+	// and a warning nobody reads is not a security control. One key, mandatory,
+	// no plaintext path.
+	//
+	// It is never generated or defaulted: a key the server invents is a key it
+	// cannot remember across a restart, and losing the key loses everything
+	// sealed under it. Generate one with `openssl rand -base64 32` and keep it
+	// wherever the database password lives — to anyone holding a stolen backup
+	// the two are worth exactly the same.
+	masterKey, err := cryptoutil.DecodeKey(os.Getenv("ENCRYPTION_KEY"))
+	if err != nil {
+		fatal("ENCRYPTION_KEY is required and must be 32 random bytes, base64-encoded",
+			"error", err, "hint", "generate one with `openssl rand -base64 32`")
+	}
+
+	// The per-purpose key variables this replaced are refused rather than
+	// ignored. Silently disregarding one would leave an operator believing
+	// their 2FA secrets or backup credentials are still readable when the
+	// server can no longer decrypt them, and they would find out from a locked
+	// out pilot. Failing at startup puts the problem where it can be fixed.
+	for _, removed := range []struct{ name, effect string }{
+		{"TOTP_ENCRYPTION_KEY", "2FA secrets sealed with it cannot be read; affected users must re-enrol"},
+		{"BACKUP_CREDENTIALS_KEY", "backup destination credentials sealed with it cannot be read; those destinations must be re-created"},
+	} {
+		if os.Getenv(removed.name) != "" {
+			fatal(removed.name+" is no longer supported — all keys now derive from ENCRYPTION_KEY",
+				"effect", removed.effect,
+				"hint", "unset "+removed.name+" once the affected data has been dealt with; see docs/UPGRADING.md")
 		}
-		slog.Info("TOTP secrets encrypted at rest")
-	} else {
-		slog.Warn("TOTP_ENCRYPTION_KEY not set — 2FA secrets are stored unencrypted")
+	}
+
+	totpAEAD, err := cryptoutil.DeriveAEAD(masterKey, cryptoutil.PurposeTOTPSecrets)
+	if err != nil {
+		fatal("could not derive the TOTP encryption key", "error", err)
 	}
 
 	// Initialize services. The two-factor service is built first: the auth
@@ -399,21 +451,36 @@ func main() {
 	// feature grew beyond images. It is still honoured so an operator who
 	// already switched the feature off does not silently get it switched back
 	// on by an upgrade; the new name wins when both are set.
+	//
+	// Stored files are encrypted at rest under a subkey of ENCRYPTION_KEY.
+	// There is no unencrypted mode: these are scans of identity documents, and
+	// a database dump that hands them over in the clear is exactly what the
+	// encryption exists to prevent.
 	documentFilesEnabled := envBoolWithLegacy("DOCUMENT_FILES_ENABLED", "DOCUMENT_IMAGES_ENABLED", true)
+	documentFileAEAD, err := cryptoutil.DeriveAEAD(masterKey, cryptoutil.PurposeDocumentFile)
+	if err != nil {
+		fatal("could not derive the document file encryption key", "error", err)
+	}
 	documentFileService := service.NewDocumentFileService(
-		postgres.NewDocumentFileRepository(db), licenseRepo, credentialRepo, documentFilesEnabled)
+		postgres.NewDocumentFileRepository(db), licenseRepo, credentialRepo, documentFilesEnabled, documentFileAEAD)
 	apiHandler.SetDocumentFileService(documentFileService)
 
 	startedAt := time.Now()
 	apiHandler.SetStartedAt(startedAt)
 	apiHandler.SetCORSOrigins(corsOrigins)
 
-	// Cloud backup service (optional — enabled only when BACKUP_CREDENTIALS_KEY is set).
+	// Cloud backup service (optional — CLOUD_BACKUPS_ENABLED=true).
+	//
+	// This used to be switched on by the presence of its own key. With every
+	// key now derived from ENCRYPTION_KEY that would mean setting one secret
+	// silently started a scheduler and a set of outbound-connecting providers,
+	// so the subsystem gets an explicit switch instead. It stays off by
+	// default, which is what "no backup key configured" meant before.
 	var backupScheduler *cloudbackup.Scheduler
-	if backupKey := os.Getenv("BACKUP_CREDENTIALS_KEY"); backupKey != "" {
-		aead, err := cryptoutil.NewFromBase64(backupKey)
+	if envBool("CLOUD_BACKUPS_ENABLED", false) {
+		aead, err := cryptoutil.DeriveAEAD(masterKey, cryptoutil.PurposeBackupCredentials)
 		if err != nil {
-			fatal("invalid BACKUP_CREDENTIALS_KEY", "error", err)
+			fatal("could not derive the backup credentials encryption key", "error", err)
 		}
 		backupDestRepo := postgres.NewBackupDestinationRepository(db)
 		backupRunRepo := postgres.NewBackupRunRepository(db)
@@ -443,7 +510,7 @@ func main() {
 		backupScheduler = cloudbackup.NewScheduler(backupSvc, 0, nil)
 		slog.Info("Cloud backups enabled (S3, SFTP, WebDAV providers)")
 	} else {
-		slog.Info("Cloud backups disabled (set BACKUP_CREDENTIALS_KEY to enable)")
+		slog.Info("Cloud backups disabled (set CLOUD_BACKUPS_ENABLED=true to enable)")
 	}
 
 	// Setup router

db/migrations/000039_create_backup_destinations.up.sql

@@ -1,7 +1,8 @@
 -- Cloud backup destinations: per-user S3-compatible (and future provider) targets.
 -- The pilot owns the destination; we store only the minimum credential needed
 -- to upload a single file under a single prefix, encrypted at rest with a
--- server-held AES-256-GCM key (BACKUP_CREDENTIALS_KEY env var).
+-- server-held AES-256-GCM key (derived from the ENCRYPTION_KEY env var; see
+-- migration 60, which introduced the shared derivation).
 CREATE TABLE backup_destinations (
     id                    UUID PRIMARY KEY DEFAULT gen_random_uuid(),
     user_id               UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,

db/migrations/000060_encrypt_document_files.down.sql

@@ -0,0 +1,19 @@
+-- Remove at-rest encryption for licence/credential files.
+--
+-- DESTRUCTIVE, and unavoidably so. Every remaining row holds AES-GCM
+-- ciphertext, and nothing in the database can turn it back into a file — the
+-- key is in the application's environment. Dropping the nonce column on its own
+-- would leave those blobs in `data` for the application to serve to a browser
+-- as if they were JPEGs: not a rollback, a silent corruption of every stored
+-- scan. So the rows go with the column.
+--
+-- To roll back and keep the files, download them through the API first, or
+-- restore a dump taken before the upgrade.
+
+DELETE FROM document_files;
+
+ALTER TABLE document_files DROP CONSTRAINT document_files_data_nonce_size;
+ALTER TABLE document_files DROP COLUMN data_nonce;
+
+COMMENT ON COLUMN document_files.data IS 'Raw file bytes, served only over an authenticated request; PDFs are always served as an attachment';
+COMMENT ON COLUMN document_files.byte_size IS NULL;