From 6331699b3b10f2c05ccb925037dcf8ef7be1211d Mon Sep 17 00:00:00 2001 From: mkn Date: Wed, 5 Aug 2026 20:25:25 +0200 Subject: [PATCH] =?UTF-8?q?feat:=20expiry=20warnings,=20per-site=20off-sit?= =?UTF-8?q?e=20backups,=20per-site=20fail2ban,=20verified=20node=E2=86=92m?= =?UTF-8?q?aster=20TLS?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four things the product promised and the software did not do. EXPIRY WARNINGS WERE NEVER SENT. run_action's notify branch only logged; its own comment admitted "real SMTP integration ships with the controller (sub-project 4)", which never happened. Meanwhile the UI labels owner_email "for warning notifications" and profiles expose warning offsets — so a site was auto-suspended with no prior word to the customer. Now it sends, over the same path the billing sweep uses, in Czech, stating the real behaviour read off the surrounding code: suspend on the expiry date, delete after the grace period, with the delete date computed the same `.max(1)` way the scheduler actually queues it. Days remaining come from the clock, not from the offset the row was queued for, so a late tick says what is really left instead of counting down from a stale number. When it CANNOT send — no relay, no owner_email, no expiry — it says so loudly and audits the row as "skipped" rather than logging success. The letter is a pure function with tests for the 30/7/1-day and same-day variants. OFF-SITE BACKUPS WERE NODE-GLOBAL, so "backups to secure off-site storage" could not be sold per client. A hosting can now pin its own target; the resolution order is hosting pin, else node default, so an unset hosting behaves byte-identically to before — a pure addition. A pin naming a target that is missing, disabled, or has no readable secret uploads NOTHING and says so, instead of silently falling back to the node default: a customer paying for off-site backups must not have their data quietly go somewhere else, or nowhere. FAIL2BAN HAD NO PER-SITE SWITCH, so hardening could not be scoped to the customers who pay for it. The per-hosting HTTP brute-force scan is now opt-out (default ON — nothing changes for existing sites), and the [fail2ban] section is finally editable in Settings instead of by hand in TOML. Node-wide sources (sshd/ftp/mail) stay node-wide: they are not attributable to one site. The auto_ban guards are untouched — it still refuses non-public IPs, and the panel login still bans the real TCP peer, never a request header. NODE→MASTER TLS IS NOW VERIFIED. This is the leg that carries the node's per-node secret in the clear on every heartbeat, and it is the leg that CAN be verified: the master normally holds a CA-issued certificate while the worker does not. `[enrollment] verify_tls` becomes tri-state — absent means verify whenever the master URL is https with a DNS hostname, true always verifies, false is the documented escape hatch for a self-signed master. A verification failure is never retried with `-k`; silently downgrading is precisely the outcome an attacker wants, so the agent aborts and logs the fix. Separately, the worker TLS pin is now WRITE-ONCE with refuse-and-warn on change, mirroring the response-signing key: the audit's objection to pin enforcement was never the enforcement, it was that the pin arrived over an unauthenticated channel and was stored last-write-wins. Enforcement stays OFF by default — this makes it safe to turn on, it does not turn it on. 926 tests pass. --- bin/hyperion-agent/src/config.rs | 44 +- bin/hyperion-agent/src/enroll.rs | 431 ++++++++- bin/hyperion-web/src/dispatcher.rs | 154 ++- bin/hyperion-web/src/handlers/hostings.rs | 296 +++++- bin/hyperion-web/src/handlers/settings.rs | 140 ++- bin/hyperion-web/src/lib.rs | 4 + .../templates/hostings_detail.html | 37 +- bin/hyperion-web/templates/install.html | 23 +- bin/hyperion-web/templates/settings.html | 239 ++++- crates/hyperion-core/src/service.rs | 906 ++++++++++++++++-- crates/hyperion-rpc-client/src/remote.rs | 23 +- packaging/agent.toml.example | 37 + packaging/install/install-node.sh | 77 +- 13 files changed, 2253 insertions(+), 158 deletions(-) diff --git a/bin/hyperion-agent/src/config.rs b/bin/hyperion-agent/src/config.rs index b370fd9d..be95ec7b 100644 --- a/bin/hyperion-agent/src/config.rs +++ b/bin/hyperion-agent/src/config.rs @@ -108,15 +108,25 @@ pub struct EnrollmentSection { /// Path where the assigned node_id is persisted after first enrollment. /// Defaults to /etc/hyperion/node-id.json. pub state_file: Option, - /// When `true`, the node verifies the master's TLS cert against - /// the system CA bundle. Defaults to `false` because install- - /// master.sh ships a self-signed cert (no DNS at install time - /// → no LE) and the node has no trust anchor to bootstrap. - /// The bearer token + per-node secret are the auth; TLS here - /// is encryption-in-transit. Set `true` once the master serves - /// a real LE cert. + /// TLS verification for the node→master leg — enrollment, and every + /// heartbeat, which carries this node's plaintext secret. + /// + /// Three states, and the ABSENT one is the point: + /// - key absent ⇒ **auto**: verify whenever `master_url` is + /// `https://` with a DNS hostname, because that is the shape a + /// CA-issued master certificate has. The master is normally the + /// side that HAS a real certificate; the worker is the side that + /// cannot, which is why this direction can be verified today and + /// the master→worker one still leans on cert pinning. + /// - `true` ⇒ always verify, even for an IP-literal master URL. + /// - `false` ⇒ the documented escape hatch for a master serving a + /// self-signed certificate. The invite token and the per-node + /// secret then cross a channel an on-path attacker can read. + /// + /// A failed verification is never retried unverified: the agent + /// aborts and names the fix. See `enroll::decide_verify_tls`. #[serde(default)] - pub verify_tls: bool, + pub verify_tls: Option, } #[derive(Debug, Clone, Deserialize)] @@ -290,6 +300,24 @@ mod tests { assert_eq!(cfg.acme.contact_email, "admin@example.com"); } + /// The tri-state is load-bearing: an agent.toml that never mentions + /// `verify_tls` must be distinguishable from one that deliberately + /// switched verification OFF, because those two get opposite + /// treatment (auto-decide vs. honour the escape hatch). + #[test] + fn verify_tls_absent_is_not_the_same_as_false() { + let absent: Config = + toml::from_str("[enrollment]\nmaster_url = \"https://m.example.cz\"").expect("parse"); + assert_eq!(absent.enrollment.verify_tls, None); + // Not even a whole missing section flips it to a decision. + assert_eq!(Config::default().enrollment.verify_tls, None); + + let off: Config = toml::from_str("[enrollment]\nverify_tls = false").expect("parse"); + assert_eq!(off.enrollment.verify_tls, Some(false)); + let on: Config = toml::from_str("[enrollment]\nverify_tls = true").expect("parse"); + assert_eq!(on.enrollment.verify_tls, Some(true)); + } + #[test] fn partial_toml_overrides_default() { let toml = r#" diff --git a/bin/hyperion-agent/src/enroll.rs b/bin/hyperion-agent/src/enroll.rs index 28980c0e..29d74685 100644 --- a/bin/hyperion-agent/src/enroll.rs +++ b/bin/hyperion-agent/src/enroll.rs @@ -5,14 +5,21 @@ //! receive back `{node_id, master_url}`, persist it, and stop. //! Subsequent boots see the state file and skip enrollment. //! -//! TLS note: the master defaults to a self-signed cert (install- -//! master.sh does NOT provision a real LE cert because at install -//! time the master often has no DNS yet). The node has no trust -//! anchor — chicken-egg — so the enrollment + heartbeat curls use -//! `-k` (skip TLS verification). The bearer token + per-node secret -//! ARE the authentication; TLS here is just encryption-in-transit. -//! Operators with a real LE cert on the master can flip -//! `verify_tls = true` in agent.toml to enforce verification. +//! TLS note: this is the leg that can actually be verified. The master +//! is the side that normally holds a CA-issued certificate (it serves +//! the panel on a real hostname); the worker is the side that cannot, +//! which is why the master→worker direction still leans on cert +//! pinning. And it is the leg worth verifying: every heartbeat carries +//! this node's per-node secret in the clear, so an on-path attacker who +//! can read it can then impersonate the node to the master. +//! +//! So `[enrollment] verify_tls` is a TRI-STATE (see +//! [`decide_verify_tls`]): absent ⇒ verify whenever the master URL is +//! `https://` with a DNS hostname, `true` ⇒ always verify, `false` ⇒ +//! the documented escape hatch for the self-signed master that +//! install-master.sh still ships. A verification failure is never +//! retried with `-k`: silently downgrading is exactly the outcome an +//! attacker wants, so the agent aborts and logs the fix instead. use serde::{Deserialize, Serialize}; use std::path::PathBuf; @@ -23,11 +30,11 @@ pub struct EnrollmentConfig { pub token: String, pub label: String, pub state_file: PathBuf, - /// When `false`, curl uses `-k` (skip TLS verification) — the - /// default because the master usually has a self-signed cert - /// and there's no trust anchor on the node yet. Flip to `true` - /// when the master serves a real LE cert. - pub verify_tls: bool, + /// The operator's `[enrollment] verify_tls`, un-defaulted: `None` + /// when the key is absent. Resolved per-URL by + /// [`decide_verify_tls`] rather than here, because the http→https + /// fallback below can change which URL we are actually talking to. + pub verify_tls: Option, /// Path to the agent.toml so we can blank out `invite_token` /// after a successful enrollment. `None` for tests + the `hctl /// enroll` one-shot path that didn't load a config. The clear @@ -153,6 +160,167 @@ pub async fn enroll_with_retry(cfg: &EnrollmentConfig) -> Result<(), String> { )) } +/// What TLS verification one node→master request gets. Produced by +/// [`decide_verify_tls`] and consumed by every curl on this leg. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum MasterTls { + /// Verify the master's certificate against this node's CA bundle. + Verify, + /// The operator wrote `verify_tls = false` — the documented escape + /// hatch for a master that serves a self-signed certificate. + SkipByOperator, + /// `https://` to a host no public CA could certify (an IP literal, + /// a single-label name), so there is nothing to verify against. + SkipNoAnchor, + /// `http://` — this connection has no TLS at all. + Plaintext, +} + +impl MasterTls { + fn verifies(self) -> bool { + matches!(self, MasterTls::Verify) + } + + /// One line for the log. The operator has to be able to answer "is + /// this node actually verifying anything?" from `journalctl` alone + /// — a bare `verify_tls=false` field would not say *why*. + fn describe(self) -> &'static str { + match self { + MasterTls::Verify => "verifying the master certificate against this node's CA bundle", + MasterTls::SkipByOperator => { + "NOT verifying — [enrollment] verify_tls = false in agent.toml; anyone on the \ + path can read this node's secret" + } + MasterTls::SkipNoAnchor => { + "NOT verifying — the master URL is an IP literal or a single-label name, which \ + no public CA certifies; give the master a hostname + certificate, then set \ + verify_tls = true" + } + MasterTls::Plaintext => { + "NO TLS — master_url is http://, so the invite token and this node's secret \ + cross the network in cleartext; move the master to https://" + } + } + } +} + +/// Resolve `[enrollment] verify_tls` for ONE url. +/// +/// `http://` short-circuits: there is no certificate on that connection +/// to verify, and reporting "verifying" for it would be a lie. Past +/// that the operator's explicit choice wins in both directions — `true` +/// even against an IP literal (they may have a private CA installed), +/// `false` as the escape hatch. Only the ABSENT case is decided here, +/// and it verifies whenever the master URL has the shape a CA-issued +/// certificate can cover. +fn decide_verify_tls(master_url: &str, configured: Option) -> MasterTls { + let url = master_url.trim(); + if !url.starts_with("https://") { + return MasterTls::Plaintext; + } + match configured { + Some(false) => MasterTls::SkipByOperator, + Some(true) => MasterTls::Verify, + None if host_can_hold_a_ca_certificate(host_of(url)) => MasterTls::Verify, + None => MasterTls::SkipNoAnchor, + } +} + +/// Host portion of `https://host[:port][/path]`, brackets stripped off +/// an IPv6 literal. Deliberately not a general URL parser — the only +/// input is the master URL the operator typed into install-node.sh. +fn host_of(url: &str) -> &str { + let rest = url.strip_prefix("https://").unwrap_or(url); + let rest = rest.split('/').next().unwrap_or(""); + if let Some(inner) = rest.strip_prefix('[') { + // `[2a01:...]:9443` — the colons are the address, not a port. + return inner.split(']').next().unwrap_or(inner); + } + match rest.rsplit_once(':') { + Some((h, _)) => h, + None => rest, + } +} + +/// Could a certificate for `host` plausibly chain to something in a +/// trust store? True for a dotted DNS name, false for an IP literal or +/// a single-label name like `master` / `localhost`. +/// +/// Deliberately permissive about the zone: `master.lan` passes, because +/// an operator who put their own CA in `/usr/local/share/ca-certificates` +/// gets a verified channel out of it, and if they didn't, the failure is +/// loud and names the fix rather than quietly falling back to `-k`. +fn host_can_hold_a_ca_certificate(host: &str) -> bool { + if host.is_empty() || host.parse::().is_ok() { + return false; + } + match host.trim_end_matches('.').rsplit_once('.') { + Some((label, tld)) => { + !label.is_empty() && tld.len() >= 2 && tld.chars().all(|c| c.is_ascii_alphabetic()) + } + None => false, + } +} + +/// Does this curl failure mean specifically "the master's certificate +/// did not verify"? A DNS failure or a refused connection must NOT be +/// reported as a certificate problem — the recipes are unrelated and an +/// operator sent to the wrong one loses an afternoon. +/// +/// Exit codes first, substrings as belt-and-braces: distro curl builds +/// disagree on which code a given OpenSSL/GnuTLS error surfaces as. +fn is_tls_verification_failure(err: &str) -> bool { + let e = err.to_ascii_lowercase(); + // Exit-code matches: 60 is CURLE_PEER_FAILED_VERIFICATION, 51 its + // legacy "peer certificate not OK" spelling, 77 an unreadable CA + // bundle (still "we could not verify", still not a network fault). + if e.contains("exit some(60)") || e.contains("exit some(51)") || e.contains("exit some(77)") { + return true; + } + // Substring matches — covers builds whose exit code differs but + // whose message is unambiguous. + e.contains("certificate verify failed") + || e.contains("self-signed certificate") + || e.contains("self signed certificate") + || e.contains("unable to get local issuer certificate") + || e.contains("ssl certificate problem") +} + +/// The message an operator gets when the master's certificate is +/// refused. It names every fix rather than one blessed path, because +/// which is correct depends on facts this node cannot see (does the +/// master have DNS? is the CA private?). +/// +/// It also states what did NOT happen: we did not retry with `-k`. +/// Silently downgrading is the whole reason this leg was unverified for +/// so long, and an operator who assumes we retried would draw exactly +/// the wrong conclusion from a node that then never appears. +fn tls_verification_help(master_url: &str, err: &str) -> String { + format!( + "{err}\n→ the TLS certificate at {master_url} did NOT verify against this node's CA \ + bundle, so the request was ABORTED — NOT retried unverified, which would hand this \ + node's secret to whoever holds the path. Fix one of:\n\ + (a) give the master a CA-issued certificate for that hostname (certbot on the master); \ + or\n\ + (b) trust the master's own CA here: copy it to \ + /usr/local/share/ca-certificates/hyperion-master.crt && sudo update-ca-certificates; \ + or\n\ + (c) accept an unverified channel — set `verify_tls = false` under [enrollment] in \ + /etc/hyperion/agent.toml and restart hyperion-agent. Master→node commands stay \ + Ed25519-signed either way, but this node's heartbeats become readable on the path." + ) +} + +/// Attach the recipe above to a failure that IS a refused certificate, +/// and leave every other failure untouched. +fn annotate_tls_failure(master_url: &str, tls: MasterTls, err: String) -> String { + if tls.verifies() && is_tls_verification_failure(&err) { + tls_verification_help(master_url, &err) + } else { + err + } +} + /// Immediate, no-delay enrollment attempt. Used by `hctl enroll`. /// Auto-tries the http URL as https on transient TLS errors — covers /// the common case where the operator pasted http:// but the master @@ -196,30 +364,44 @@ pub async fn enroll_now(cfg: &EnrollmentConfig) -> Result<(), String> { // (empty reply, "wrong version number") AND the URL is http://, // retry as https:// — that's the very common "master is HTTPS // but operator copy-pasted http:" trap. - tracing::info!(master = %base, "attempting node enrollment"); + // + // The verification decision is therefore resolved per-URL rather + // than once per config: that fallback changes which connection we + // are describing, and an http URL has no certificate to verify + // while its https twin does. + let tls = decide_verify_tls(&base, cfg.verify_tls); + tracing::info!(master = %base, tls = tls.describe(), "attempting node enrollment"); let primary_url = format!("{base}/api/enroll"); - match post_json(&primary_url, &body, cfg.verify_tls).await { - Ok(stdout) => return finish_enrollment(cfg, &stdout).await, + match post_json(&primary_url, &body, tls.verifies()).await { + Ok(stdout) => finish_enrollment(cfg, &stdout).await, Err(e) if should_try_https_fallback(&base, &e) => { - let https = format!("https://{}/api/enroll", &base[7..]); + let https_base = format!("https://{}", &base[7..]); + // The upgrade to https is also an upgrade in what we can + // check, so re-decide against the URL we're about to use. + let tls = decide_verify_tls(&https_base, cfg.verify_tls); tracing::warn!( error = %e, + tls = tls.describe(), "enrollment over {base} failed — retrying with https://" ); - let stdout = post_json(&https, &body, cfg.verify_tls).await?; + let stdout = post_json(&format!("{https_base}/api/enroll"), &body, tls.verifies()) + .await + .map_err(|e| annotate_tls_failure(&https_base, tls, e))?; // Persist the discovered scheme so subsequent heartbeats // skip the fallback dance. let mut adjusted = cfg.clone(); - adjusted.master_url = format!("https://{}", &base[7..]); - return finish_enrollment(&adjusted, &stdout).await; + adjusted.master_url = https_base; + finish_enrollment(&adjusted, &stdout).await } - Err(e) => Err(e), + Err(e) => Err(annotate_tls_failure(&base, tls, e)), } } /// Helper: POST JSON, return stdout on HTTP 2xx or a useful error -/// string. `verify_tls=false` adds `-k` (chicken-egg: until we've -/// enrolled we have no trust anchor for the master's cert). +/// string. `verify_tls=false` adds `-k`. This function does NOT decide +/// that — [`decide_verify_tls`] does, per URL, and a caller that hands +/// it `false` has already established there is nothing to verify or +/// that the operator opted out. /// /// Body is fed via curl's stdin (`--data-binary @-`), NOT via argv. /// The previous `--data ` approach put the invite token (on @@ -414,9 +596,11 @@ async fn clear_invite_token_in_config(path: &std::path::Path) -> Result<(), Stri /// `period_secs` and POSTs {node_id, secret, agent_version} to /// `/api/heartbeat`. Single error → log + retry next tick. /// -/// `verify_tls` mirrors `EnrollmentConfig::verify_tls` — default -/// off so self-signed master certs work. The bearer secret is the -/// auth; TLS is just encryption-in-transit. +/// `verify_tls` mirrors `EnrollmentConfig::verify_tls` — the operator's +/// un-defaulted setting, resolved per tick by [`decide_verify_tls`] +/// against the master URL we persisted at enrollment. This is the leg +/// that matters most: the body below carries the per-node secret in +/// the clear on every single tick. /// /// `resp_pubkey` is our response-signing pubkey, derived at startup /// from the loaded key and passed in rather than read here: it is @@ -426,7 +610,7 @@ async fn clear_invite_token_in_config(path: &std::path::Path) -> Result<(), Stri pub async fn heartbeat_loop( state_file: std::path::PathBuf, period_secs: u64, - verify_tls: bool, + verify_tls: Option, inbound_cert: std::path::PathBuf, resp_pubkey: Option, ) { @@ -441,6 +625,10 @@ pub async fn heartbeat_loop( // `None` when remote_rpc is disabled (no cert) — the master simply // records no pin for this node, which is fine. let mut tls_spki_pin: Option = None; + // Both one-shot: the TLS posture and the certificate-refused recipe + // are identical on every tick, and this loop runs 1440 times a day. + let mut tls_policy_logged = false; + let mut tls_help_logged = false; let period = std::time::Duration::from_secs(period_secs); let mut interval = tokio::time::interval(period); // First tick fires immediately — skip it so we wait one period after @@ -455,6 +643,14 @@ pub async fn heartbeat_loop( if tls_spki_pin.is_none() { tls_spki_pin = hyperion_core::tls_pin::spki_pin_from_cert_file(&inbound_cert).await; } + // The master URL comes from node-id.json (operator-supplied at + // enrollment, never from the response), so this is a decision + // about a trusted string, not one the master can steer. + let tls = decide_verify_tls(&p.master_url, verify_tls); + if !tls_policy_logged { + tls_policy_logged = true; + tracing::info!(master = %p.master_url, tls = tls.describe(), "heartbeat TLS policy"); + } let url = format!("{}/api/heartbeat", p.master_url.trim_end_matches('/')); let body = match serde_json::to_string(&serde_json::json!({ "node_id": p.node_id, @@ -475,7 +671,7 @@ pub async fn heartbeat_loop( use std::process::Stdio; use tokio::io::AsyncWriteExt; let mut args: Vec<&str> = vec!["-fsS", "--max-time", "8"]; - if !verify_tls { + if !tls.verifies() { args.push("-k"); } args.extend([ @@ -553,12 +749,37 @@ pub async fn heartbeat_loop( } } Ok(out) => { - tracing::warn!( - code = ?out.status.code(), - stderr = %String::from_utf8_lossy(&out.stderr).trim(), - master = %p.master_url, - "heartbeat returned non-zero — will retry" - ); + let stderr = String::from_utf8_lossy(&out.stderr).trim().to_string(); + // Same shape should_try_https_fallback matches on, so one + // classifier serves both call sites. + let detail = format!("exit {:?}: {stderr}", out.status.code()); + if tls.verifies() && is_tls_verification_failure(&detail) { + // NOT retried with -k, here or anywhere: a node that + // silently downgrades on a bad certificate is a node an + // attacker only has to break once. It goes stale in the + // panel instead, which is the visible failure we want. + if !tls_help_logged { + tls_help_logged = true; + tracing::error!( + master = %p.master_url, + "SECURITY: heartbeat refused — {}", + tls_verification_help(&p.master_url, &detail) + ); + } else { + tracing::warn!( + master = %p.master_url, + detail = %detail, + "heartbeat still refused — the master certificate does not verify" + ); + } + } else { + tracing::warn!( + code = ?out.status.code(), + stderr = %stderr, + master = %p.master_url, + "heartbeat returned non-zero — will retry" + ); + } } Err(e) => tracing::warn!(error=%e, "heartbeat curl failed"), } @@ -654,6 +875,148 @@ mod tests { assert_eq!(decide_pubkey_pin(Some("KEY_A"), "KEY_B"), PubkeyPin::Refuse); } + /// The default. A master URL with the shape a CA-issued certificate + /// can cover gets verified WITHOUT the operator opting in — this is + /// the leg that carries the node's plaintext secret on every tick. + #[test] + fn verify_tls_defaults_on_for_an_https_hostname() { + for url in [ + "https://master.example.com", + "https://master.example.cz:8443", + "https://panel.hyperion.example.co.uk/", + // A private zone still counts: the operator may have put + // their own CA in the node's trust store, and if not, the + // failure names the fix instead of downgrading. + "https://master.lan", + // Trailing whitespace from a hand-edited toml. + " https://master.example.com ", + ] { + assert_eq!( + decide_verify_tls(url, None), + MasterTls::Verify, + "{url} should verify by default" + ); + } + } + + /// ...and nowhere else. Auto must never claim to verify something + /// it cannot: an IP literal or a single-label name has no publicly + /// certifiable identity, and http:// has no certificate at all. + #[test] + fn verify_tls_auto_declines_where_there_is_no_trust_anchor() { + assert_eq!( + decide_verify_tls("https://203.0.113.9:8443", None), + MasterTls::SkipNoAnchor + ); + assert_eq!( + decide_verify_tls("https://[2001:db8::1]:8443", None), + MasterTls::SkipNoAnchor + ); + assert_eq!( + decide_verify_tls("https://master:8443", None), + MasterTls::SkipNoAnchor + ); + assert_eq!( + decide_verify_tls("https://localhost:8443", None), + MasterTls::SkipNoAnchor + ); + // Numeric last label — not a TLD, so not a certifiable name. + assert_eq!( + decide_verify_tls("https://10.0.0.5", None), + MasterTls::SkipNoAnchor + ); + // http:// is Plaintext, not "skip": there is no certificate on + // that connection, and describing it as skipped verification + // would understate what actually happens to the token. + assert_eq!( + decide_verify_tls("http://master.example.com", None), + MasterTls::Plaintext + ); + // None of these are Verify — the invariant that matters. + for url in [ + "https://203.0.113.9", + "https://master", + "http://master.example.com", + ] { + assert!(!decide_verify_tls(url, None).verifies(), "{url}"); + } + } + + /// The escape hatch, and its opposite. An explicit setting wins in + /// BOTH directions — `false` is what an operator with a self-signed + /// master sets, `true` is what an operator with a private CA sets + /// for an IP-literal master that auto would have declined. + #[test] + fn an_explicit_setting_beats_the_auto_decision() { + assert_eq!( + decide_verify_tls("https://master.example.com", Some(false)), + MasterTls::SkipByOperator, + "the documented escape hatch must survive the new default" + ); + assert_eq!( + decide_verify_tls("https://203.0.113.9:8443", Some(true)), + MasterTls::Verify + ); + // ...except over http://, where there is nothing to verify no + // matter what the file says. + assert_eq!( + decide_verify_tls("http://master.example.com", Some(true)), + MasterTls::Plaintext + ); + } + + /// Only a REFUSED certificate gets the certificate recipe. Sending + /// an operator whose DNS is broken off to install a CA costs them + /// an afternoon. + #[test] + fn only_certificate_failures_get_the_certificate_recipe() { + assert!(is_tls_verification_failure( + "POST https://m.example.com/api/enroll exit Some(60): SSL certificate problem: \ + self-signed certificate" + )); + assert!(is_tls_verification_failure( + "exit Some(77): error setting certificate file" + )); + assert!(is_tls_verification_failure( + "exit Some(35): ssl routines::certificate verify failed" + )); + // Not certificate problems. + assert!(!is_tls_verification_failure( + "exit Some(6): Could not resolve host: master.example.com" + )); + assert!(!is_tls_verification_failure( + "exit Some(7): Failed to connect to master.example.com port 8443" + )); + assert!(!is_tls_verification_failure("exit Some(22): 404 Not Found")); + + // The annotation only fires when we were actually verifying... + let refused = "exit Some(60): self-signed certificate".to_string(); + let helped = annotate_tls_failure("https://m.example.com", MasterTls::Verify, refused); + assert!(helped.contains("verify_tls = false"), "names the opt-out"); + assert!( + helped.contains("NOT retried unverified"), + "must say what did not happen: {helped}" + ); + // ...and never rewrites an unrelated failure. + let dns = "exit Some(6): Could not resolve host".to_string(); + assert_eq!( + annotate_tls_failure("https://m.example.com", MasterTls::Verify, dns.clone()), + dns + ); + // Nor one from a connection we never verified in the first + // place — telling that operator their certificate is bad when + // we passed `-k` would be a fabricated diagnosis. + let same = "exit Some(60): self-signed certificate".to_string(); + assert_eq!( + annotate_tls_failure( + "https://m.example.com", + MasterTls::SkipByOperator, + same.clone() + ), + same + ); + } + #[test] fn https_fallback_triggers_on_tls_signature_errors() { // Exit 1 + "HTTP/0.9" — the case the user actually hit on diff --git a/bin/hyperion-web/src/dispatcher.rs b/bin/hyperion-web/src/dispatcher.rs index dda68614..861b49f3 100644 --- a/bin/hyperion-web/src/dispatcher.rs +++ b/bin/hyperion-web/src/dispatcher.rs @@ -47,6 +47,14 @@ pub enum DispatchError { /// it never carries response content. #[error("node {node_id} response failed authentication: {reason}")] ResponseAuthFailed { node_id: String, reason: String }, + /// Worker TLS certificate pinning is ENFORCED and the master holds + /// no pin for this node, so the connection would be made with no + /// certificate check at all. Same family as `ResponseAuthFailed` + /// and deliberately not `NodeUnreachable`: nothing is wrong with + /// the node's reachability, the master simply refuses to open an + /// unauthenticated channel while the operator has said not to. + #[error("node {node_id} has no TLS certificate pin on file while pinning is enforced")] + CertPinMissing { node_id: String }, #[error("target node {0} is not enrolled")] UnknownNode(String), #[error("target node {0} has no public_ip on record — cannot reach")] @@ -203,16 +211,15 @@ async fn dispatch_remote( // trip") — but "this node has published nothing" is exactly the state // an on-path attacker engineers by stripping resp_pubkey from the // heartbeat, and skipping the read handed that node a hard-coded - // enforce=false. `check_response_auth` needs the real toggle value to - // refuse it. Cert pinning loses nothing either way: with no reported - // pin there is nothing to pass to --pinnedpubkey regardless. + // enforce=false. BOTH checks below need the real toggle value to + // refuse a node in that state — an absent TLS pin is engineered the + // same way, by stripping tls_spki_pin from the heartbeat. let enforce = cluster_enforcement(state).await; - // Block C enforce phase: when the cluster toggle is on AND this node - // has a heartbeat-reported pin, pin it for real (curl --pinnedpubkey). - let pinned_pubkey = match &route.reported_pin { - Some(pin) if enforce.cert_pinning => Some(pin.clone()), - _ => None, - }; + // Block C enforce phase. Refuses BEFORE dialling when there is no pin + // to enforce with — see check_cert_pinning for why "no pin" cannot be + // treated as "nothing to enforce". + let pinned_pubkey = + check_cert_pinning(node_id, route.reported_pin.as_deref(), enforce.cert_pinning)?; let opts = RemoteCallOpts { timeout_secs: timeout_for_request(&req), pinned_pubkey, @@ -334,12 +341,16 @@ pub async fn fan_out_reporting( // worker". Log it at ERROR with the security wording // so it can't be lost among routine exclusions in // `journalctl -u hyperion-web -g fan_out`. - if matches!(e, DispatchError::ResponseAuthFailed { .. }) { + if matches!( + e, + DispatchError::ResponseAuthFailed { .. } + | DispatchError::CertPinMissing { .. } + ) { tracing::error!( node = %nid, error = %e, - "SECURITY: fan_out excluded a node whose response failed \ - authentication — this page is rendering an INCOMPLETE \ + "SECURITY: fan_out excluded a node the master could not \ + authenticate — this page is rendering an INCOMPLETE \ aggregate, not an empty one" ); } else { @@ -449,6 +460,53 @@ async fn cluster_enforcement(state: &SharedState) -> Enforcement { } } +/// Decide the `--pinnedpubkey` value for this dispatch, or refuse it. +/// +/// The pin the master holds is trust-on-first-use and **write-once**: +/// `nodes.tls_spki_pin` is only ever FILLED (`COALESCE(tls_spki_pin, ?)` +/// in `touch_last_seen`), a heartbeat presenting a DIFFERENT pin is +/// refused and warned about (`tofu_report`), and clearing it is an +/// explicit operator action (`node_reset_crypto`, re-enrollment). So an +/// attacker cannot silently re-aim a pin that has landed. +/// +/// What they can still do is stop one from ever landing: the heartbeat +/// travels over `curl -k`, so stripping `tls_spki_pin` from it leaves +/// the column NULL forever. Treating NULL as "nothing to enforce" would +/// hand the attacker the choice of WHICH nodes the toggle protects, and +/// the connection would then be made with `-k` and no pin — precisely +/// the state the toggle exists to end. So under enforcement, no pin on +/// file is a refusal, exactly like no response-signing key on file is in +/// [`check_response_auth`]. +/// +/// With enforcement OFF this returns `Ok(None)` for every input and +/// nothing is ever refused: the warn-only observation in +/// [`warn_on_pin_mismatch`] is the whole behaviour, unchanged. +fn check_cert_pinning( + node_id: &str, + reported_pin: Option<&str>, + enforce: bool, +) -> Result, DispatchError> { + if !enforce { + return Ok(None); + } + match reported_pin { + Some(pin) => Ok(Some(pin.to_string())), + None => { + tracing::error!( + node = node_id, + "SECURITY: worker TLS certificate pinning is enforced but this node has no \ + pin on file, so the RPC would run over an unverified connection with nothing \ + to check the cert against. Refused. Restart the node's agent and wait one \ + heartbeat for it to report its pin (the 🔒 chip on Nodes), or turn off \ + Enforce worker TLS certificate pinning in Settings → Cluster." + ); + Err(DispatchError::CertPinMissing { + node_id: node_id.to_string(), + }) + } + } +} + /// The four-way response-authentication matrix, evaluated on every /// remote dispatch. `resp_pubkey` is the key the node published over /// its authenticated heartbeat; `out.resp_sig` is what arrived on @@ -653,6 +711,17 @@ impl From for crate::error::AppError { connectivity problem: check `journalctl -u hyperion-agent` on the node \ and whether anything is intercepting master→worker traffic." )), + // Also NOT NodeUnreachable: the node is fine, the master + // declined to talk to it unverified. The two fixes are named + // in the order an operator should try them — re-reporting the + // pin keeps the protection, switching the toggle off drops it. + DispatchError::CertPinMissing { node_id } => AppError::Rpc(format!( + "node {node_id} has not reported a TLS certificate pin, and Enforce worker TLS \ + certificate pinning is on in Settings → Cluster — so the master refused to \ + dispatch over a connection it cannot check. Restart hyperion-agent on that node \ + and wait one heartbeat (about 30 s) for the 🔒 chip to appear on Nodes, or turn \ + the setting off." + )), DispatchError::UnknownNode(n) => AppError::BadRequest(format!( "node {n} is not enrolled — pick a different target" )), @@ -792,6 +861,67 @@ mod tests { assert!(is_auth_failure(&err), "wrong variant: {err}"); } + const PIN: &str = "/4IrPU/vEdcxQgcB9m3gD/9oaQ9/8WmdvXZIDD+ZVxg="; + + /// Nothing changes until the operator flips the toggle — including + /// for a node that has never reported a pin. This is the arm that + /// keeps a mixed cluster working, so it is asserted first. + #[test] + fn cert_pinning_is_inert_until_the_operator_enforces_it() { + assert_eq!(check_cert_pinning(NODE, Some(PIN), false).unwrap(), None); + assert_eq!(check_cert_pinning(NODE, None, false).unwrap(), None); + } + + /// Enforcing pins the value on file, verbatim — it is what curl gets + /// after `sha256//`, so any rewriting here would break every call. + #[test] + fn enforced_pinning_pins_exactly_the_pin_on_file() { + assert_eq!( + check_cert_pinning(NODE, Some(PIN), true) + .expect("a node with a pin is dispatched to") + .as_deref(), + Some(PIN) + ); + } + + /// The write-once rule's consequence. Because the master only ever + /// FILLS `nodes.tls_spki_pin` and refuses a changed one, an attacker + /// cannot re-aim a pin that landed — the only lever left is keeping + /// one from ever landing, by stripping `tls_spki_pin` from the + /// node's (unverified) heartbeats. Enforcement therefore has to read + /// "no pin on file" as a refusal; reading it as "nothing to check" + /// would let the attacker choose who the toggle protects. + #[test] + fn enforced_pinning_refuses_a_node_whose_pin_never_landed() { + let err = check_cert_pinning(NODE, None, true) + .expect_err("a pinless node must not be dispatched to under enforcement"); + assert!( + matches!(err, DispatchError::CertPinMissing { .. }), + "wrong variant: {err}" + ); + assert!(err.to_string().contains("no TLS certificate pin"), "{err}"); + } + + /// ...and the refusal must not read as downtime either: an operator + /// who restarts a healthy agent looking for a network fault is an + /// operator who never finds the toggle that caused this. + #[test] + fn missing_pin_does_not_render_as_node_unreachable() { + let app: crate::error::AppError = DispatchError::CertPinMissing { + node_id: NODE.to_string(), + } + .into(); + assert!( + !matches!(app, crate::error::AppError::NodeUnreachable { .. }), + "an enforcement refusal must not present as downtime" + ); + let text = app.to_string(); + assert!( + text.contains("Settings"), + "names where to turn it off: {text}" + ); + } + /// A forged response must never reach the operator as "node /// unreachable" — that reads as downtime and invites a retry. #[test] diff --git a/bin/hyperion-web/src/handlers/hostings.rs b/bin/hyperion-web/src/handlers/hostings.rs index d4476ca8..58e1000f 100644 --- a/bin/hyperion-web/src/handlers/hostings.rs +++ b/bin/hyperion-web/src/handlers/hostings.rs @@ -132,6 +132,19 @@ struct DetailTpl<'a> { /// Per-hosting recurring-backup cadence ("off"|"daily"|"weekly"|"monthly"), /// read from the owning node's hosting_kv; drives the Backups-card select. backup_cadence: String, + csrf_backup_target: String, + /// The off-site target this hosting is pinned to (`backup_targets.name`), + /// or "" for the node default. Owning node's hosting_kv, like the cadence. + backup_target: String, + /// Every configured off-site target + why it can't take a backup, when it + /// can't. Empty for non-admins, who get no picker at all. + backup_target_options: Vec, + /// Label of the "no pin" option — it names how many targets would actually + /// receive a copy, so leaving it alone isn't a guess. + backup_target_default_label: String, + /// Non-empty when the pin above resolves to nothing — the site is NOT + /// going off-site and the card has to say so. + backup_target_warning: String, csrf_expiry_set: String, csrf_expiry_clear: String, csrf_dns_check: String, @@ -159,6 +172,9 @@ struct DetailTpl<'a> { wp_flash: Option, backup_error: Option, backup_flash: Option, + /// Why an off-site pin was refused (unknown target), carried back through + /// the redirect so the operator doesn't think the change landed. + backup_target_error: Option, expiry_error: Option, expiry_flash: Option, cert_error: Option, @@ -1219,6 +1235,14 @@ pub async fn post_create( csrf_backup_now: csrf_token_for(&state, &ctx, "/hostings/backup-now"), csrf_backup_cadence: csrf_token_for(&state, &ctx, "/hostings/backup-cadence"), backup_cadence: "off".into(), + csrf_backup_target: csrf_token_for(&state, &ctx, "/hostings/backup-target"), + // A just-created hosting has no pin yet, and the create + // response renders in place — the picker shows up on the next + // GET of the detail page rather than being half-populated here. + backup_target: String::new(), + backup_target_options: vec![], + backup_target_default_label: String::new(), + backup_target_warning: String::new(), csrf_expiry_set: csrf_token_for(&state, &ctx, "/hostings/expiry/set"), csrf_expiry_clear: csrf_token_for(&state, &ctx, "/hostings/expiry/clear"), csrf_dns_check: csrf_token_for(&state, &ctx, "/hostings/dns-check"), @@ -1248,6 +1272,7 @@ pub async fn post_create( }), backup_error: None, backup_flash: None, + backup_target_error: None, expiry_error: None, expiry_flash: None, cert_error: None, @@ -1941,10 +1966,11 @@ pub async fn get_detail( .map(|(_, v)| v.trim().to_string()) .filter(|s| !s.is_empty()) .unwrap_or_else(|| format!("staging.{}", detail.domain)); - // Recurring-backup cadence lives in the OWNING node's hosting_kv (seeded by - // profile_apply there + overridable here), so read it from the owner rather - // than the master kv_pairs above. Best-effort: default "off". - let backup_cadence = match crate::dispatcher::dispatch_to_node( + // Recurring-backup cadence and the off-site pin live in the OWNING node's + // hosting_kv (seeded by profile_apply there + overridable here), so read + // them from the owner rather than the master kv_pairs above. Best-effort: + // cadence defaults to "off", pin to the node default. + let owner_kv = match crate::dispatcher::dispatch_to_node( &state, target, Request::HostingKvList { @@ -1953,14 +1979,53 @@ pub async fn get_detail( ) .await { - Ok(RpcResponse::HostingKvList(v)) => v - .into_iter() - .find(|(k, _)| k == "backup_cadence") - .map(|(_, v)| v) - .filter(|v| matches!(v.as_str(), "daily" | "weekly" | "monthly")) - .unwrap_or_else(|| "off".into()), - _ => "off".into(), + Ok(RpcResponse::HostingKvList(v)) => v, + _ => vec![], + }; + let backup_cadence = owner_kv + .iter() + .find(|(k, _)| k == "backup_cadence") + .map(|(_, v)| v.clone()) + .filter(|v| matches!(v.as_str(), "daily" | "weekly" | "monthly")) + .unwrap_or_else(|| "off".into()); + // Off-site destination picker. Admin-only, and the target list is fetched + // only for admins — a non-admin render would otherwise pair a live pin + // with an empty list and report a working target as unconfigured. Same + // reason the picker disappears entirely when the list can't be read. + let targets = if ctx.is_admin_or_higher() { + configured_backup_targets(&state).await + } else { + None }; + let (backup_target, backup_target_options, backup_target_default_label, backup_target_warning) = + match targets { + Some(mut options) => { + let pin = owner_kv + .iter() + .find(|(k, _)| k == "backup_target") + .map(|(_, v)| v.trim().to_string()) + .unwrap_or_default(); + let warning = pinned_target_warning(&pin, &options); + // Say what leaving the picker alone actually does. "Every enabled + // target" reads like a promise when the answer is sometimes none. + let default_label = match options.iter().filter(|o| o.unusable.is_empty()).count() { + 0 => "Node default — no target can take a copy, local disk only".to_string(), + 1 => "Node default — the one enabled target".to_string(), + n => format!("Node default — all {n} enabled targets"), + }; + // A pin that matches nothing still has to appear in the select — + // otherwise the picker would read "Node default" for a site whose + // backups are in fact going nowhere. + if !pin.is_empty() && !options.iter().any(|o| o.name == pin) { + options.push(BackupTargetOption { + name: pin.clone(), + unusable: "not one of the configured off-site targets".into(), + }); + } + (pin, options, default_label, warning) + } + None => (String::new(), Vec::new(), String::new(), String::new()), + }; // Internal preview URL on the owner node's wildcard cert (shown only // when the node actually has one + the domain isn't already under it). let preview_domain = compute_preview_domain(&state, target, &detail.domain).await; @@ -1995,6 +2060,11 @@ pub async fn get_detail( csrf_db_reset: csrf_token_for(&state, &ctx, "/hostings/db/reset-password"), csrf_backup_cadence: csrf_token_for(&state, &ctx, "/hostings/backup-cadence"), backup_cadence, + csrf_backup_target: csrf_token_for(&state, &ctx, "/hostings/backup-target"), + backup_target, + backup_target_options, + backup_target_default_label, + backup_target_warning, csrf_profile_apply: csrf_token_for(&state, &ctx, "/profiles/apply"), profile_apply, applied_profile_name, @@ -2014,6 +2084,7 @@ pub async fn get_detail( }), backup_error: q.backup_error, backup_flash: q.backup.map(|_| "Backup started — see list below.".into()), + backup_target_error: q.backup_target_error, expiry_error: q.expiry_error, expiry_flash: q.expiry.map(|s| { if s == "cleared" { @@ -2155,6 +2226,10 @@ pub struct DetailQuery { pub backup: Option, #[serde(default)] pub backup_error: Option, + /// Refusal from the off-site-target POST (unknown target), so the page + /// says the pin did NOT change instead of redirecting silently. + #[serde(default)] + pub backup_target_error: Option, #[serde(default)] pub expiry: Option, #[serde(default)] @@ -2306,6 +2381,109 @@ pub(crate) async fn resolve_s3_targets(state: &SharedState) -> Vec Option> { + let rows = match hyperion_rpc_client::call(&state.agent_socket, Request::BackupTargetList).await + { + Ok(RpcResponse::BackupTargetList(v)) => v, + _ => return None, + }; + let mut out = Vec::new(); + for r in rows { + let unusable = if r.kind != "s3" { + format!("a \"{}\" target — backups are only pushed to S3", r.kind) + } else if !r.enabled { + "disabled in Settings → Backups".to_string() + } else { + match r.secret_key_id.as_deref() { + None => "missing its secret key".to_string(), + Some(p) if tokio::fs::metadata(p).await.is_err() => { + "missing its secret key file on this node".to_string() + } + Some(_) => String::new(), + } + }; + out.push(BackupTargetOption { + name: r.name, + unusable, + }); + } + Some(out) +} + +/// What the card says about the target a hosting is pinned to. Empty when +/// there's no pin or the pin is fine; otherwise the sentence that stops a dead +/// pin from reading like a working one — a pinned site whose target can't take +/// the upload gets NO off-site copy at all (the runner refuses to reroute a +/// client's data to a bucket nobody chose). +fn pinned_target_warning(pin: &str, options: &[BackupTargetOption]) -> String { + if pin.is_empty() { + return String::new(); + } + match options.iter().find(|o| o.name == pin) { + None => format!( + "\"{pin}\" is not one of the configured off-site targets — backups of this site are \ + NOT being copied off-site." + ), + Some(o) if !o.unusable.is_empty() => format!( + "\"{}\" is {} — backups of this site are NOT being copied off-site until that is fixed.", + o.name, o.unusable + ), + Some(_) => String::new(), + } +} + +#[cfg(test)] +mod backup_target_tests { + use super::{pinned_target_warning, BackupTargetOption}; + + fn opt(name: &str, unusable: &str) -> BackupTargetOption { + BackupTargetOption { + name: name.into(), + unusable: unusable.into(), + } + } + + #[test] + fn only_a_pin_that_cannot_receive_a_backup_warns() { + let options = vec![opt("wasabi-eu", ""), opt("b2-cold", "disabled in Settings")]; + // No pin: the site follows the node default, nothing to warn about. + assert!(pinned_target_warning("", &options).is_empty()); + // Pinned to a target that can take the upload: silence. + assert!(pinned_target_warning("wasabi-eu", &options).is_empty()); + // Pinned to a configured but unusable one — the reason has to reach + // the operator, and the sentence must not imply copies are happening. + let disabled = pinned_target_warning("b2-cold", &options); + assert!(disabled.contains("b2-cold"), "names the target: {disabled}"); + assert!(disabled.contains("disabled in Settings"), "{disabled}"); + assert!(disabled.contains("NOT being copied off-site"), "{disabled}"); + // Pinned to something that isn't configured at all (deleted/renamed). + let gone = pinned_target_warning("gone", &options); + assert!(gone.contains("not one of the configured"), "{gone}"); + assert!(gone.contains("NOT being copied off-site"), "{gone}"); + } +} + pub async fn post_backup_now( State(state): State, ctx: AuthCtx, @@ -5571,6 +5749,102 @@ pub async fn post_set_backup_cadence( .into_response()) } +#[derive(serde::Deserialize)] +pub struct BackupTargetForm { + pub selector: String, + /// A `backup_targets.name`, or empty for "node default". + #[serde(default)] + pub target: String, +} + +/// Pin this hosting's backups to ONE off-site target, or clear the pin back to +/// the node default. Stored in the OWNING node's hosting_kv ("backup_target"), +/// which is where `backup_now` reads it. +/// +/// Admin-only on the server, not just in the markup: the target list is +/// cluster config, and an operator with BackupRun on one site must not be able +/// to aim that site's archives at another client's bucket. +/// +/// A name that isn't configured is refused rather than stored — a pin that +/// resolves to nothing stops the site going off-site entirely, and that must +/// never be the silent result of a stale form. +pub async fn post_set_backup_target( + State(state): State, + ctx: AuthCtx, + Form(form): Form, +) -> Result { + if !ctx.is_admin_or_higher() { + return Err(AppError::Forbidden); + } + let sel = match require_manage_for_selector(&state, &ctx, &form.selector, Capability::BackupRun) + .await + { + Ok(s) => s, + Err(r) => return Ok(r), + }; + let sel_url = urlencoding(&form.selector); + let (detail, target) = match find_hosting_anywhere(&state, sel).await { + Ok(v) => v, + Err(_) => { + return Ok(Redirect::to(&format!("/hostings/{}#backups", sel_url)).into_response()); + } + }; + let want = form.target.trim().to_string(); + if !want.is_empty() { + let refusal = match configured_backup_targets(&state).await { + Some(options) if options.iter().any(|o| o.name == want) => None, + Some(_) => Some(format!( + "\"{want}\" is not a configured off-site target — nothing was changed." + )), + None => Some( + "The configured off-site targets could not be read, so the destination was \ + left as it was." + .to_string(), + ), + }; + if let Some(msg) = refusal { + return Ok(Redirect::to(&format!( + "/hostings/{}?backup_target_error={}#backups", + sel_url, + urlencoding(&msg) + )) + .into_response()); + } + } + // A write that didn't land must not redirect as if it had: this key + // decides where a paying client's data ends up. + let saved = crate::dispatcher::dispatch_to_node( + &state, + target.as_deref(), + Request::HostingKvSet { + hosting_id: detail.id.as_str().to_string(), + key: "backup_target".into(), + value: want, + }, + ) + .await; + match saved { + Ok(RpcResponse::HostingKvSet) => Ok(Redirect::to(&format!( + "/hostings/{}?flash_saved=backup-target#backups", + sel_url + )) + .into_response()), + Ok(RpcResponse::Error(e)) => Ok(Redirect::to(&format!( + "/hostings/{}?backup_target_error={}#backups", + sel_url, + urlencoding(&format!("the owning node refused the change: {e}")) + )) + .into_response()), + Ok(_) => Err(AppError::Internal("unexpected response".into())), + Err(e) => Ok(Redirect::to(&format!( + "/hostings/{}?backup_target_error={}#backups", + sel_url, + urlencoding(&format!("the owning node could not be reached: {e}")) + )) + .into_response()), + } +} + /// Lazily-loaded SFTP panel (FTP tab). Dispatched to the OWNING node — /// the system user, home dir and authorized_keys all live there. #[derive(Template)] diff --git a/bin/hyperion-web/src/handlers/settings.rs b/bin/hyperion-web/src/handlers/settings.rs index c78cde0c..aaaa94c4 100644 --- a/bin/hyperion-web/src/handlers/settings.rs +++ b/bin/hyperion-web/src/handlers/settings.rs @@ -58,6 +58,10 @@ struct SettingsTpl<'a> { /// the "Raw TOML" tab. Failing to read shows "(could not /// read /etc/hyperion/agent.toml: …)". raw_toml: String, + /// The master's `[fail2ban]` section, for the Brute force tab. + /// `None` when agent.toml couldn't be read or parsed — the card then + /// says so instead of pre-filling a form with numbers nothing backs. + fail2ban: Option, /// Live MTA (postfix) state — mode (smart-host / direct-mx / /// not-installed / default), myhostname, relayhost, mailq depth, /// recent mail.log. Drives the new "MTA" card under the SMTP @@ -175,6 +179,71 @@ fn fmt_date(ts: Option) -> String { } } +/// The `[fail2ban]` section as it currently reads on disk, for the Brute +/// force tab's form. +/// +/// Not part of `AgentConfigView`: the section is read straight out of the +/// same `/etc/hyperion/agent.toml` the Raw TOML tab already reads, so the +/// form shows the file rather than a value re-derived somewhere else. +pub struct Fail2banView { + /// The values in the file, defaults filled in for absent keys. + pub cfg: hyperion_core::Fail2banConfig, + /// False ⇒ agent.toml has no `[fail2ban]` table at all, so these are + /// the built-in defaults rather than anything an operator chose. Said + /// out loud in the card so the form can't imply a decision nobody made. + pub in_toml: bool, + /// True ⇒ at least one value on disk is outside the floors + /// `Fail2banConfig::sanitized` applies at agent start, so the running + /// scanner is NOT using the numbers shown. Only reachable by hand-editing + /// the file — the form's own validation refuses to write such a value. + pub clamped: bool, +} + +/// Parse the `[fail2ban]` table out of agent.toml for the Brute force tab. +/// +/// An absent table is not an error: the agent runs the built-in defaults +/// then, which is exactly what's rendered — flagged with `in_toml = false`. +/// `None` means the file didn't parse at all, and the card says that instead +/// of showing numbers it can't stand behind. +fn parse_fail2ban_section(raw: &str) -> Option { + let doc: toml::Value = raw.parse().ok()?; + let table = doc.get("fail2ban"); + let d = hyperion_core::Fail2banConfig::default(); + let flag = |key: &str, fallback: bool| { + table + .and_then(|t| t.get(key)) + .and_then(|v| v.as_bool()) + .unwrap_or(fallback) + }; + let num = |key: &str, fallback: i64| { + table + .and_then(|t| t.get(key)) + .and_then(|v| v.as_integer()) + .unwrap_or(fallback) + }; + // Thresholds are u32 in the config; a hand-edited negative or absurd + // value saturates rather than wrapping into a huge threshold that would + // silently disable a scanner. + let count = + |key: &str, fallback: u32| num(key, fallback as i64).clamp(0, u32::MAX as i64) as u32; + let cfg = hyperion_core::Fail2banConfig { + enabled: flag("enabled", d.enabled), + window_secs: num("window_secs", d.window_secs), + ban_ttl_secs: num("ban_ttl_secs", d.ban_ttl_secs), + repeat_ttl_secs: num("repeat_ttl_secs", d.repeat_ttl_secs), + repeat_lookback_secs: num("repeat_lookback_secs", d.repeat_lookback_secs), + http_threshold: count("http_threshold", d.http_threshold), + ssh_threshold: count("ssh_threshold", d.ssh_threshold), + ftp_threshold: count("ftp_threshold", d.ftp_threshold), + mail_threshold: count("mail_threshold", d.mail_threshold), + }; + Some(Fail2banView { + in_toml: table.is_some(), + clamped: cfg.clone().sanitized() != cfg, + cfg, + }) +} + /// One test node's wildcard-cert row in the Settings card. pub struct NodeWildcardRow { pub node_id: String, @@ -283,12 +352,15 @@ pub async fn get_settings( Ok(RpcResponse::NodesList(v)) => v, _ => Vec::new(), }; - // Read agent.toml for the Raw TOML tab. Mask anything that - // looks like a password / token line — token values are - // single-line strings so a regex on `password = "..."` / - // `token = "..."` / `webhook = "https://hooks..."` suffices. - let raw_toml = match tokio::fs::read_to_string("/etc/hyperion/agent.toml").await { - Ok(s) => mask_secrets_in_toml(&s), + // Read agent.toml once — for the Raw TOML tab (masked: anything that + // looks like a password / token line; token values are single-line + // strings so a regex on `password = "..."` / `token = "..."` / + // `webhook = "https://hooks..."` suffices) and for the Brute force + // tab's `[fail2ban]` form, which is parsed from the unmasked text. + let agent_toml = tokio::fs::read_to_string("/etc/hyperion/agent.toml").await; + let fail2ban = agent_toml.as_deref().ok().and_then(parse_fail2ban_section); + let raw_toml = match &agent_toml { + Ok(s) => mask_secrets_in_toml(s), Err(e) => format!("(could not read /etc/hyperion/agent.toml: {e})"), }; // Master's Cloudflare DNS-01 token state (live-verified if present) for the @@ -384,6 +456,7 @@ pub async fn get_settings( resp_auth_pending, resp_auth_total, raw_toml, + fail2ban, mta, mail_node, cloudflare, @@ -933,6 +1006,7 @@ fn section_to_tab(section: &str) -> &'static str { "slack" => "notifications", "backup_remote" | "backup_retention" => "backups", "cluster" => "cluster", + "fail2ban" => "bruteforce", _ => "mail", } } @@ -946,6 +1020,7 @@ fn sanitize_return_tab(v: &str) -> Option<&'static str> { "tls" => Some("tls"), "notifications" => Some("notifications"), "backups" => Some("backups"), + "bruteforce" => Some("bruteforce"), "cluster" => Some("cluster"), "testnodes" => Some("testnodes"), "retention" => Some("retention"), @@ -1805,4 +1880,57 @@ from_address = "ops@example.cz" // doesn't match "password" exactly. assert!(out.contains("# password = \"never-stored-but-comment\"")); } + + // ============================================================ + // [fail2ban] section → Brute force tab + // ============================================================ + + /// No section at all is the common case on an install that never + /// touched it: the agent runs the defaults, so the form shows the + /// defaults — but flagged, so nothing implies an operator chose them. + #[test] + fn absent_section_shows_the_defaults_the_agent_actually_runs() { + let v = super::parse_fail2ban_section("[agent]\nsocket_path = \"/run/x.sock\"\n") + .expect("valid toml"); + assert!(!v.in_toml); + assert!(!v.clamped); + assert_eq!(v.cfg, hyperion_core::Fail2banConfig::default()); + } + + /// A partially-filled section keeps the operator's values and fills the + /// rest from the same defaults the agent would. + #[test] + fn partial_section_merges_with_the_agent_defaults() { + let v = super::parse_fail2ban_section( + "[fail2ban]\nenabled = false\nhttp_threshold = 4\nban_ttl_secs = 7200\n", + ) + .expect("valid toml"); + let d = hyperion_core::Fail2banConfig::default(); + assert!(v.in_toml); + assert!(!v.cfg.enabled); + assert_eq!(v.cfg.http_threshold, 4); + assert_eq!(v.cfg.ban_ttl_secs, 7200); + assert_eq!(v.cfg.ssh_threshold, d.ssh_threshold); + assert_eq!(v.cfg.window_secs, d.window_secs); + assert!(!v.clamped); + } + + /// A hand-edited value the agent overrides at start-up must be reported, + /// not quietly rendered as if the scanner were using it. + #[test] + fn a_value_the_agent_clamps_is_flagged() { + let v = + super::parse_fail2ban_section("[fail2ban]\nban_ttl_secs = 0\n").expect("valid toml"); + assert!(v.in_toml); + assert_eq!(v.cfg.ban_ttl_secs, 0, "the file is shown as it reads"); + assert!(v.clamped, "…and the card has to say the agent uses 60 s"); + } + + /// Unparseable agent.toml yields nothing, so the card says the values + /// are unknown rather than showing built-in defaults as if they were + /// this node's settings. + #[test] + fn unparseable_toml_yields_no_view() { + assert!(super::parse_fail2ban_section("[fail2ban\nenabled = ").is_none()); + } } diff --git a/bin/hyperion-web/src/lib.rs b/bin/hyperion-web/src/lib.rs index 9224ffbd..e6acf3ef 100644 --- a/bin/hyperion-web/src/lib.rs +++ b/bin/hyperion-web/src/lib.rs @@ -124,6 +124,10 @@ pub fn build_router(state: SharedState) -> Router { "/hostings/backup-cadence", post(handlers::hostings::post_set_backup_cadence), ) + .route( + "/hostings/backup-target", + post(handlers::hostings::post_set_backup_target), + ) .route( "/hostings/expiry/set", post(handlers::hostings::post_set_expiry), diff --git a/bin/hyperion-web/templates/hostings_detail.html b/bin/hyperion-web/templates/hostings_detail.html index 6ed52155..018a6943 100644 --- a/bin/hyperion-web/templates/hostings_detail.html +++ b/bin/hyperion-web/templates/hostings_detail.html @@ -413,6 +413,12 @@

Backup failed: {{ e }}
{% endif %} +{% if let Some(e) = backup_target_error %} +
+ +
Off-site destination not changed: {{ e }}
+
+{% endif %} {% if let Some(m) = expiry_flash %}
@@ -2554,7 +2560,7 @@

-

Backups land on local disk; off-site targets (S3 + age) are configured cluster-wide in Settings → Backups.

+

Backups land on local disk; off-site targets (S3 + age) are configured cluster-wide in Settings → Backups, and each site picks one of them below.

@@ -2584,6 +2590,35 @@

+ {% if !backup_target_options.is_empty() %} + {# Off-site destination for THIS site. Unset = the node default (every + enabled target), which is what every site did before this picker + existed. Rendered only for admins — the list is cluster config. #} +
+ + +
+ + +
+ +
+ {% if !backup_target_warning.is_empty() %} +
+
+ Off-site copies are not happening for this site. + {{ backup_target_warning }} + Fix the target in Settings → Backups, or + pick a different destination above. +
+
+ {% endif %} + {% endif %} {# Live region: a running backup flips to ok, and a backup started from the job progress page shows up on return — no F5. Driver in base.html. #}
diff --git a/bin/hyperion-web/templates/install.html b/bin/hyperion-web/templates/install.html index a9389a7e..58739135 100644 --- a/bin/hyperion-web/templates/install.html +++ b/bin/hyperion-web/templates/install.html @@ -229,11 +229,20 @@

signal for the hardening ladder in Settings → Control plane → Security. Turn a step on only once every node below carries its chip: while a step is on, a node missing that chip is refused, and its sites drop out of every - list until it publishes the key. + list until it reports the missing value. Both are pinned on first sight and a changed key is refused, so a node whose agent you deliberately reinstalled stays refused until you use Clear pinned crypto on its card. Only reach for that after a key change you expected.

+

+ Neither chip says anything about the other direction — whether a node verifies + this master's certificate when it enrolls and heartbeats. That is set on the node + itself, in /etc/hyperion/agent.toml under [enrollment] verify_tls, + and the install command below measures it while it runs. A node that stops appearing under + Last seen shortly after you changed this master's certificate is the + symptom: check journalctl -u hyperion-agent -g 'TLS policy' there. The agent + never falls back to an unverified connection on its own. +

{# Server-side pre-tagged: each `nodes` entry is `(NodeSummary, is_test, version_skew)` so the template doesn't string-search or @@ -261,8 +270,15 @@

{{ n.label }}

{% else %} {{ n.agent_version }} {% endif %} + {# Rendered in BOTH states, for the same reason as the + response-auth chip below: an absent chip is invisible + when you are scanning a grid of cards, and a node + without a pin is the one that gets cut off the moment + step 1 of the hardening ladder goes on. #} {% if let Some(pin) = n.tls_spki_pin %} - 🔒 TLS pin on file + 🔒 TLS pin on file + {% else %} + ⚠ No TLS pin {% endif %} {# Response-signing readiness. Rendered in BOTH states, unlike the TLS pin above: the operator has to confirm that EVERY @@ -629,7 +645,8 @@

  • Create a labeled token above. The plaintext is shown once — copy it immediately; only its hash is stored on the master.
  • Run the printed curl … | sudo bash … command on the new machine.
  • The install script provisions dependencies, builds hyperion-agent, saves the token and master URL into /etc/hyperion/agent.toml, and starts the service.
    • -
  • The agent calls POST /api/enroll with the token, signs an mTLS certificate, and joins the cluster. The new node appears in Enrolled nodes above within about 30 s.
    • +
  • Before writing that file the installer probes this master's TLS certificate from the new machine and records what it found: verify_tls = true when the certificate verified, verify_tls = false (plus the reason) when it did not, and nothing at all when the master could not be reached — in which case the agent decides from the URL, verifying any https:// master with a DNS hostname.
    • +
  • The agent calls POST /api/enroll with the token, signs an mTLS certificate, and joins the cluster. The new node appears in Enrolled nodes above within about 30 s. If the certificate check fails the agent stops rather than retrying unverified, and journalctl -u hyperion-agent on the node carries the fix.

    • {% endblock %} diff --git a/bin/hyperion-web/templates/settings.html b/bin/hyperion-web/templates/settings.html index b18dc76c..2ebd74e1 100644 --- a/bin/hyperion-web/templates/settings.html +++ b/bin/hyperion-web/templates/settings.html @@ -90,6 +90,14 @@

    Settings

    {% if config.backup_remote.enabled %}● {% else %}local{% endif %} + + + Brute force + {% if let Some(f) = fail2ban %} + {% if f.cfg.enabled %}● + {% else %}off{% endif %} + {% endif %} + Control plane @@ -822,6 +830,176 @@

    +{# ========================================================== + BRUTE FORCE — the native scanner's [fail2ban] section + ========================================================== #} +
    + +
    +
    + Hyperion watches four sources for login floods and drops offenders at the + firewall: each site's own access.log (POSTs to + wp-login.php / xmlrpc.php) and the node-wide + ssh, ftp and mail journals. + Bans in force are listed on Bans. +
    +
    + + {% if let Some(f) = fail2ban %} +
    +
    +

    + + Brute-force scanner +

    + {% if f.cfg.enabled %}scanning + {% else %}off{% endif %} +
    + + {# Two things the form itself cannot say. Neither is decoration: one + means the numbers were never chosen, the other means they are not + the numbers in use. #} + {% if !f.in_toml %} +
    +
    + agent.toml has no [fail2ban] section, so the + agent is running the built-in defaults — that is what's filled in below. + Saving writes the section for the first time. +
    +
    + {% endif %} + {% if f.clamped %} + + {% endif %} + +
    + + + + +

    + With this off, nothing is scanned — bans you add by hand still work, and + expired bans are still lifted. It also makes every care report say + not measured for attacks blocked rather than report a zero + nobody was watching for. +

    + +
    + + +
    + +
    + + Failures inside the window before a ban +
    +
    +
    + + +
    +
    + + +
    +
    + + +
    +
    + + +
    +
    +

    + Website login counts POSTs to wp-login.php and + xmlrpc.php in one site's access log, so the ban is recorded + against that site and shows up in its care report. This is also the only + source that can be switched off for a single site — the ssh, ftp and mail + journals cover the whole machine and can't be attributed to one customer, + so they stay on for everyone. +

    + +
    + + How long a ban lasts +
    +
    +
    + + +
    +
    + + +
    +
    + + +
    +
    +

    + An address that was already banned within the look-back window gets the + longer repeat-offender ban instead of the first-offence one. Set the + look-back to 0 to never escalate. A ban shorter than a minute + is refused: the ban list reads a zero duration as never expires. +

    + +

    + Only public, routable addresses are ever banned automatically. Private, + loopback, CGNAT and link-local addresses are skipped, because a node-wide + firewall drop on those is an outage — webmail talking to dovecot, the + master↔node link, a NAT gateway in front of the whole office. You can + still block any address by hand from Bans. +

    + {% if !nodes.is_empty() %} +

    + This form edits {{ config.hostname }} only. Every enrolled + node scans with its own [fail2ban] section; changing it here + does not reach them. +

    + {% endif %} +

    + Panel logins are guarded separately: repeated failures throttle, then + firewall the connecting address for an hour. That path is not governed by + the numbers above. +

    + +
    +
    + {% else %} +
    +

    + + Brute-force scanner +

    +

    + /etc/hyperion/agent.toml could not be read or did not parse, so + the current thresholds are unknown and this form would be guessing. The + Raw TOML tab shows what came back. +

    +
    + {% endif %} +
    {# /tab-bruteforce #} + {# ========================================================== CLUSTER — multi-node controls ========================================================== #} @@ -958,26 +1136,45 @@

    for example during a server move. Applies within about 30 s; no restart needed.

    - {# ─── Master → worker hardening ladder ──────────────────────── - The two toggles below are rungs, not independent switches, and - the order is load-bearing: step 1 keeps an on-path attacker out - of the channel, step 2 keeps the master from believing what - comes back if the channel is defeated anyway. Turning on step 2 - alone still leaves every secret in a reply (reset passwords, - enrollment tokens) readable on the wire, so the numbering is UI, - not decoration. #} + {# ─── Cluster channel hardening ─────────────────────────────── + Three rungs, and the order is load-bearing. Step 0 is not a + toggle: it lives in each node's own agent.toml because it is + the node that decides whether to trust the master's cert, and + it comes first because it is the leg that carries the node's + plaintext secret. Steps 1 and 2 are master→worker: step 1 keeps + an on-path attacker out of the channel, step 2 keeps the master + from believing what comes back if the channel is defeated + anyway. Turning on step 2 alone still leaves every secret in a + reply (reset passwords, enrollment tokens) readable on the + wire, so the numbering is UI, not decoration. #}
    - Master → worker channel hardening + Cluster channel hardening

    Every command the master sends already carries an Ed25519 request - signature, so a worker never acts on a forged instruction. These two steps + signature, so a worker never acts on a forged instruction. The steps below protect the rest of the exchange — who can read the channel, and whether the master can trust the answer. Work through them in order, and only after Nodes shows the matching chip on every node.

    +

    + Step 0 · Each node verifies this master's certificate. + Not a switch here — it is set per node, in /etc/hyperion/agent.toml under + [enrollment], because the node is the side that decides whom to trust. + It comes first because it is the leg that carries each node's own secret, in the + clear, on every heartbeat. The node installer measures it: if this master already + serves a CA-issued certificate it writes verify_tls = true; if the + certificate is self-signed it writes verify_tls = false and says so. + With verify_tls absent the agent verifies whenever this master's URL is + https:// with a DNS hostname. A failed check is never retried + unverified — the node stops heartbeating and prints the fix, so a node that has + gone stale on Nodes right after you changed the master's + certificate is telling you something. Check with + journalctl -u hyperion-agent -g 'TLS policy' on the node. +

    +