feat: per-app MFA guard in the proxy (#1005 slice 4)

book/src/admin.md

@@ -65,6 +65,30 @@ acting username. A **last-admin guard** stops you deleting or demoting
 the only remaining admin (so the portal can't be locked out); the
 `RUSCKER_ADMIN_TOKEN` break-glass login is the other safety net.
 
+### Two-factor authentication for selected apps
+
+In an app's **Access & scale** settings, enable **Require 2FA**
+(`require-mfa`) to require a user-owned authenticator-app code before the
+proxy will select or start that app's container. The same enrolled TOTP
+factor is reused across protected apps; the switch is a per-app step-up
+policy, not a separate enrollment for every app.
+
+On first access, a signed-in user without a factor is guided through setup
+and receives recovery codes. Later access redirects to a challenge when the
+browser has no current trusted-device proof. **MFA validity days** controls
+that cadence (7 days by default, at most 30); `0` means every new login
+session must prove MFA, even if the browser still has a trusted-device
+cookie. API routes do not redirect: they return `401` without a login and
+`403` when MFA is still required.
+
+Users can open **Two-factor authentication** in their account to **forget
+this device** or **forget all trusted devices** without ending their login
+sessions. If a phone or recovery-code set is lost, an Admin can open the
+user's edit page and **Reset 2FA**; this deletes the factor, recovery codes
+and device grants, so the next protected-app visit starts guided enrollment
+again. The `RUSCKER_ADMIN_TOKEN` remains an audited break-glass bypass for
+emergencies and should not be used for routine app access.
+
 ## Screens
 
 The sections below follow the panel's tab order: daily drivers first

crates/ruscker-admin/assets/i18n/en/landing.ftl

@@ -646,7 +646,6 @@ spec-form-require-mfa = Require 2FA
 spec-form-require-mfa-hint = Users without a configured TOTP factor will be guided through enrollment on first access to a protected app.
 spec-form-mfa-validity = Ask again after N days
 spec-form-mfa-validity-hint = Blank = 7 days. Use 0 to require a new proof in every login session, with no remembered device.
-spec-form-mfa-staged-note = 2FA enforcement arrives in an upcoming release; for now, this app is not yet protected.
 spec-form-identity-headers = Send identity headers to the app
 spec-form-identity-headers-hint = Adds X-SP-UserId and X-SP-UserGroups for signed-in users. Off by default; enable only for apps that need and trust this identity.
 spec-form-identity-claims = Additional identity claims

crates/ruscker-admin/assets/i18n/es/landing.ftl

@@ -646,7 +646,6 @@ spec-form-require-mfa = Exigir 2FA
 spec-form-require-mfa-hint = Los usuarios sin un factor TOTP configurado recibirán instrucciones para registrarlo en el primer acceso a una app protegida.
 spec-form-mfa-validity = Volver a solicitar después de N días
 spec-form-mfa-validity-hint = Vacío = 7 días. Usa 0 para exigir una nueva prueba en cada sesión de inicio de sesión, sin dispositivo recordado.
-spec-form-mfa-staged-note = La aplicación de 2FA llegará en una próxima versión; por ahora, esta app aún no está protegida.
 spec-form-identity-headers = Enviar cabeceras de identidad a la app
 spec-form-identity-headers-hint = Añade X-SP-UserId y X-SP-UserGroups para usuarios autenticados. Desactivado por defecto; actívalo solo para apps que necesiten y confíen en esta identidad.
 spec-form-identity-claims = Datos de identidad adicionales

crates/ruscker-admin/assets/i18n/fr/landing.ftl

@@ -646,7 +646,6 @@ spec-form-require-mfa = Exiger la 2FA
 spec-form-require-mfa-hint = Les utilisateurs sans facteur TOTP configuré seront guidés pour l’inscrire lors du premier accès à une app protégée.
 spec-form-mfa-validity = Redemander après N jours
 spec-form-mfa-validity-hint = Vide = 7 jours. Utilisez 0 pour exiger une nouvelle preuve à chaque session de connexion, sans appareil mémorisé.
-spec-form-mfa-staged-note = L’application de la 2FA arrivera dans une prochaine version ; pour l’instant, cette app n’est pas encore protégée.
 spec-form-identity-headers = Envoyer les en-têtes d’identité à l’app
 spec-form-identity-headers-hint = Ajoute X-SP-UserId et X-SP-UserGroups pour les utilisateurs connectés. Désactivé par défaut ; activez uniquement pour les apps qui ont besoin de cette identité et lui font confiance.
 spec-form-identity-claims = Attributs d’identité supplémentaires

crates/ruscker-admin/assets/i18n/pt/landing.ftl

@@ -650,7 +650,6 @@ spec-form-require-mfa = Exigir 2FA
 spec-form-require-mfa-hint = Usuários sem um fator TOTP configurado serão orientados a cadastrá-lo no primeiro acesso a um app protegido.
 spec-form-mfa-validity = Solicitar novamente após N dias
 spec-form-mfa-validity-hint = Em branco = 7 dias. Use 0 para exigir nova prova em cada sessão de login, sem dispositivo lembrado.
-spec-form-mfa-staged-note = A exigência de 2FA chega em uma próxima versão; por enquanto, este app ainda não está protegido.
 spec-form-identity-headers = Enviar cabeçalhos de identidade ao app
 spec-form-identity-headers-hint = Adiciona X-SP-UserId e X-SP-UserGroups para usuários autenticados. Desativado por padrão; ative apenas para apps que precisam e confiam nessa identidade.
 spec-form-identity-claims = Dados adicionais de identidade

crates/ruscker-admin/src/routes/proxy.rs

@@ -170,6 +170,19 @@ const API_PREFIX: &str = "/api/";
 static INFLIGHT: std::sync::LazyLock<dashmap::DashMap<ruscker_core::ReplicaId, u32>> =
     std::sync::LazyLock::new(dashmap::DashMap::new);
 
+/// Break-glass MFA bypasses are deliberately loud, but a browser page load
+/// can generate dozens of proxied subrequests. Deduplicate the persistent
+/// audit row per `(admin session, spec)` while retaining a WARN for every
+/// request. Reserving the entry before the database write also prevents two
+/// concurrent first requests from inserting duplicate rows.
+static MFA_BREAK_GLASS_AUDITS: std::sync::LazyLock<
+    dashmap::DashMap<(String, String), std::time::Instant>,
+> = std::sync::LazyLock::new(dashmap::DashMap::new);
+
+const MFA_BREAK_GLASS_AUDIT_COOLDOWN: std::time::Duration =
+    std::time::Duration::from_secs(15 * 60);
+const MFA_BREAK_GLASS_AUDIT_CAP: usize = 256;
+
 /// RAII guard: bumps a replica's in-flight count on creation and drops
 /// it on `Drop`, covering every return/error path of the forward.
 pub(crate) struct InflightGuard(ruscker_core::ReplicaId);
@@ -435,6 +448,13 @@ async fn forward(
             // is sent to log in; everyone else (and all API clients) get
             // a flat 403 (CORS-wrapped for the `/api/` family).
             if route_prefix == APP_PREFIX && session.0.is_none() {
+                if ws_upgrade.0.is_some() {
+                    return (
+                        StatusCode::UNAUTHORIZED,
+                        "authentication required before WebSocket upgrade\n",
+                    )
+                        .into_response();
+                }
                 // Proxy routes are not wrapped by the chrome's
                 // `prefix_base_path` Location-rewriter, so build the
                 // base-prefixed login URL ourselves (#294) — otherwise a
@@ -461,6 +481,94 @@ async fn forward(
         }
     }
 
+    // 2e. Per-app MFA step-up (#1005). This MUST stay ahead of every
+    //     backend/replica path below: an enrollment/challenge redirect may
+    //     never wake or start a container. Unprotected specs skip this block
+    //     entirely, including all MFA database reads.
+    if spec.effective_require_mfa() {
+        let admin_session = session.0.as_ref();
+        let session_id = cookies
+            .get(crate::auth::COOKIE_NAME)
+            .map(|cookie| cookie.value().to_string());
+
+        if admin_session.is_some_and(|session| {
+            session.role == crate::auth::Role::Admin && session.actor.is_none()
+        }) {
+            let session_id = session_id.as_deref().unwrap_or("missing-session-cookie");
+            tracing::warn!(
+                spec = %spec.id,
+                "break-glass admin session bypassed per-app MFA"
+            );
+            audit_mfa_break_glass_bypass(&state, session_id, &spec.id).await;
+        } else {
+            let Some(admin_session) = admin_session else {
+                if route_prefix == API_PREFIX {
+                    return with_cors(
+                        (
+                            StatusCode::UNAUTHORIZED,
+                            "MFA-protected API requires an authenticated user session\n",
+                        )
+                            .into_response(),
+                        cors_on,
+                    );
+                }
+                if ws_upgrade.0.is_some() {
+                    return (
+                        StatusCode::UNAUTHORIZED,
+                        "MFA proof required before WebSocket upgrade\n",
+                    )
+                        .into_response();
+                }
+                return mfa_app_redirect(&state, &spec.id, &upstream_path, &req, "login");
+            };
+
+            let (Some(username), Some(session_id)) =
+                (admin_session.actor.as_deref(), session_id.as_deref())
+            else {
+                return with_cors(
+                    (
+                        StatusCode::FORBIDDEN,
+                        "MFA proof required for this protected app\n",
+                    )
+                        .into_response(),
+                    cors_on,
+                );
+            };
+            let decision = crate::mfa::evaluate(&state, username, session_id, &cookies, &spec).await;
+            if decision != crate::mfa::MfaDecision::Satisfied {
+                if route_prefix == API_PREFIX {
+                    return with_cors(
+                        (
+                            StatusCode::FORBIDDEN,
+                            "MFA proof required for this API; complete it in the web portal\n",
+                        )
+                            .into_response(),
+                        cors_on,
+                    );
+                }
+                if ws_upgrade.0.is_some() {
+                    return (
+                        StatusCode::UNAUTHORIZED,
+                        "MFA proof required before WebSocket upgrade\n",
+                    )
+                        .into_response();
+                }
+                let destination = match decision {
+                    crate::mfa::MfaDecision::EnrollmentRequired => "enroll",
+                    crate::mfa::MfaDecision::ChallengeRequired => "challenge",
+                    crate::mfa::MfaDecision::Satisfied => unreachable!(),
+                };
+                return mfa_app_redirect(
+                    &state,
+                    &spec.id,
+                    &upstream_path,
+                    &req,
+                    destination,
+                );
+            }
+        }
+    }
+
     // 3. Backend required to proxy.
     if state.backend.is_none() {
         return with_cors(
@@ -825,6 +933,105 @@ async fn forward(
     }
 }
 
+/// Preserve the route's raw percent-encoding and query string for the MFA
+/// round trip. Axum has already matched this request as `/app/{spec}/{rest}`;
+/// the decoded route values provide a safe fallback, and `local_next_path`
+/// applies the same open-redirect guard used by the challenge handlers.
+fn mfa_app_next(spec_id: &str, upstream_path: &str, req: &Request) -> String {
+    let raw = req
+        .uri()
+        .path_and_query()
+        .map(|value| value.as_str())
+        .unwrap_or_else(|| req.uri().path());
+    if let Some(next) = super::local_next_path(Some(raw)) {
+        return next.to_string();
+    }
+
+    let mut fallback = format!("{APP_PREFIX}{spec_id}{upstream_path}");
+    if let Some(query) = req.uri().query() {
+        fallback.push('?');
+        fallback.push_str(query);
+    }
+    super::local_next_path(Some(&fallback))
+        .unwrap_or("/")
+        .to_string()
+}
+
+fn mfa_app_redirect(
+    state: &AppState,
+    spec_id: &str,
+    upstream_path: &str,
+    req: &Request,
+    destination: &str,
+) -> Response {
+    let next = mfa_app_next(spec_id, upstream_path, req);
+    let path = match destination {
+        "login" => format!("{}/admin/login", state.base_path),
+        "enroll" => format!("{}/admin/account/mfa", state.base_path),
+        "challenge" => format!("{}/admin/account/mfa/challenge", state.base_path),
+        _ => unreachable!(),
+    };
+    Redirect::to(&super::with_next_query(&path, &next)).into_response()
+}
+
+async fn audit_mfa_break_glass_bypass(state: &AppState, session_id: &str, spec_id: &str) {
+    let now = std::time::Instant::now();
+    let key = (session_id.to_string(), spec_id.to_string());
+    let should_record = match MFA_BREAK_GLASS_AUDITS.entry(key) {
+        dashmap::mapref::entry::Entry::Occupied(mut entry) => {
+            if now.duration_since(*entry.get()) < MFA_BREAK_GLASS_AUDIT_COOLDOWN {
+                false
+            } else {
+                entry.insert(now);
+                true
+            }
+        }
+        dashmap::mapref::entry::Entry::Vacant(entry) => {
+            entry.insert(now);
+            true
+        }
+    };
+    if !should_record {
+        return;
+    }
+
+    if MFA_BREAK_GLASS_AUDITS.len() > MFA_BREAK_GLASS_AUDIT_CAP {
+        MFA_BREAK_GLASS_AUDITS
+            .retain(|_, at| now.duration_since(*at) < MFA_BREAK_GLASS_AUDIT_COOLDOWN);
+    }
+
+    let Some(db) = state.db.as_ref() else {
+        // No DB to persist to: drop the dedup reservation so this bypass is
+        // re-attempted if a database is later attached, rather than being
+        // silently suppressed for 15 minutes (codex review, #1005).
+        MFA_BREAK_GLASS_AUDITS.remove(&(session_id.to_string(), spec_id.to_string()));
+        tracing::warn!(
+            spec = %spec_id,
+            "cannot persist break-glass MFA bypass audit without a config database"
+        );
+        return;
+    };
+    let target = format!("spec:{spec_id}");
+    if let Err(err) = crate::db::audit::record(
+        db,
+        "token",
+        "mfa.break_glass_bypass",
+        &target,
+        None,
+    )
+    .await
+    {
+        // Roll back the reservation on a transient write failure so the
+        // NEXT bypass retries the audit instead of the key suppressing it
+        // for the whole cooldown, leaving a persistent audit gap even after
+        // the DB recovers (codex review, #1005). Over-auditing (a possible
+        // duplicate row if a concurrent write succeeded) is safe; an audit
+        // gap is not.
+        MFA_BREAK_GLASS_AUDITS.remove(&(session_id.to_string(), spec_id.to_string()));
+        tracing::warn!(error = ?err, spec = %spec_id, "audit break-glass MFA bypass failed");
+    }
+}
+
 /// Wrap a response body so `guard` only drops once the body is fully
 /// consumed/dropped — keeping the replica's in-flight count accurate for
 /// long downloads/streams (#424).

crates/ruscker-admin/templates/admin/spec_form.html

@@ -604,7 +604,6 @@ <h1 class="text-xl font-medium tracking-tight" x-text="form.display_name || form
                  :disabled="!form.require_mfa"
                  class="spec-form-input">
           <div class="spec-form-help">{{ self.t("spec-form-mfa-validity-hint") }}</div>
-          <div class="spec-form-help" style="color:var(--color-lock)">{{ self.t("spec-form-mfa-staged-note") }}</div>
         </div>
       </div>