fix: value-integrity — 5 backends corrupted secrets on read (v0.20.0)

CHANGELOG.md

@@ -4,6 +4,30 @@ All notable changes to SecretEnv are documented here. The format follows [Keep a
 
 Alongside the standard sections (Added, Changed, Deprecated, Removed, Fixed, Security), releases from v0.14.0 onward use a `Known limitations` subsection for behavior that ships honestly but is incomplete by design.
 
+## [0.20.0] - 2026-08-01
+
+Value-integrity release. Five backends silently corrupted secrets on read; `set` was never affected, so no stored data is damaged and no migration is required. Live-backend smoke gains section 37, a byte-exact round-trip matrix covering 14 of 15 backends.
+
+### Fixed
+- **macOS Keychain returned multi-line and non-ASCII values hex-encoded (data corruption).** `security find-*-password -w` silently hex-encodes its output, with no marker, whenever the value contains any byte outside printable ASCII — a newline, a tab, or any non-ASCII character. The backend returned that payload verbatim, so every PEM block, certificate, SSH key and accented passphrase came back corrupted on every macOS install. `get` now takes `security`'s own declaration of the encoding: shape decides only whether to *ask*, and `find-*-password -g` decides what the value *is*. Encoding is never inferred from shape, because `deadbeefcafe0123` is both an ordinary API key and valid hex. When `-g` cannot determine the encoding, `get` refuses rather than returning possibly-corrupt bytes; a non-UTF-8 item now errors cleanly instead of being lossily converted. Note that `-g` prints the password to stderr, so that stream is deliberately never routed into an error message, a log line, or a span field.
+- **Vault, OpenBao, GCP and Azure truncated the final byte of any value ending in a newline (data corruption).** All four ended `get()` with `strip_suffix('\n')`, which is correct only when the CLI appends a newline of its own. `vault kv get -field=<f>`, `bao kv get -field=<f>` and `gcloud secrets versions access` all write the value's bytes and append nothing when stdout is a pipe; Azure parses the value out of `--output json`, where no CLI newline exists at all. The strip therefore ate the secret's own last byte. PEM blocks, certificates and SSH keys are conventionally stored with a trailing newline and are rejected by `openssl` / `ssh-keygen` without it, so this broke exactly the values most likely to be multi-line. All four now return the bytes verbatim.
+
+### Changed
+- Values fetched from `keychain`, `vault`, `openbao`, `gcp` and `azure` may now end in a trailing newline where previous versions truncated it. This is the corrected behavior, but it is a change in returned bytes — consumers that compensated for the truncation should drop the workaround.
+
+### Added
+- Live smoke section 37, `value encoding — multi-line / UTF-8 / trailing-newline / hex-shaped byte round-trip`: 104 assertions across 14 backends. Each value is seeded with the backend's native CLI, then read back twice — natively and through `secretenv` — so a backend that cannot hold the bytes records SKIP instead of being misreported as SecretEnv corruption. Includes a `hex-shaped` case (`deadbeefcafe0123`) that fails if anyone reintroduces shape-inferred decoding, and a cf-kv guard locking in that `wrangler kv key get --text` *does* append a newline and its strip is correct.
+- New `assert_value_bytes` / `assert_native_holds` harness assertions comparing with `cmp` rather than `grep`. Every pre-existing assertion was substring-based and therefore blind to a lost trailing newline or an undecoded hex payload — which is why a fully green 810-assertion matrix coexisted with corruption on five backends.
+- Per-backend regression tests (`get_preserves_legitimate_trailing_newline`) on vault, openbao, gcp and azure, plus a macOS-only `live` module in the keychain backend that exercises the real `security` binary via a self-cleaning per-PID item.
+
+### Known limitations
+- **Doppler rejects multi-line values** at the API; the smoke records this as an explicit SKIP rather than a failure. Not a SecretEnv defect.
+- **1Password truncates a trailing newline at write time** — `op item create` stores `abc` when given `abc\n`. SecretEnv reads back faithfully what 1Password holds; the loss happens in `op`, not in SecretEnv.
+- **Keeper is not covered by section 37.** Seeding it requires `record-add` with a record type and folder, which would mutate the operator's real vault. Both read paths were verified instead: `--format=json` extracts from parsed JSON and applies no strip, and `keeper get --format=password` was measured live to append a newline, so its strip of exactly one is correct. Recorded as an explicit SKIP so the gap stays visible.
+
+### Internal
+- Mock fixtures for vault, openbao, gcp and azure previously fed `"value\n"`, modelling those CLIs as appending a trailing newline they do not emit. The fixtures were corrected to match measured behavior rather than flipping the assertions, since the fixtures — not the assertions — encoded the wrong model.
+
 ## [0.19.0] - 2026-06-14
 
 Second non-backend hardening minor (hardening #2), consuming the v0.18 carry-forward queue and the deferred post-release documentation phase; no new backends (total 15), no new crates (workspace count 24); live-backend smoke 779 PASS / 0 FAIL / 2 expected SKIP across all 15 backends.

Cargo.toml

@@ -28,7 +28,7 @@ members = [
 ]
 
 [workspace.package]
-version = "0.19.0"
+version = "0.20.0"
 edition      = "2021"
 license      = "AGPL-3.0-only"
 repository   = "https://github.com/TechAlchemistX/secretenv"
@@ -46,28 +46,28 @@ readme       = "README.md"
 # (`secretenv-core`, `secretenv-backend-*`). The CLI is the exception: it
 # lives at `crates/secretenv-cli/` but publishes as `secretenv` so
 # `cargo install secretenv` and the binary name line up.
-secretenv-core               = { path = "crates/secretenv-core",                           version = "0.19.0", features = ["value-access"] }
-secretenv-telemetry          = { path = "crates/secretenv-telemetry",                      version = "0.19.0" }
-secretenv-migrate            = { path = "crates/secretenv-migrate",                        version = "0.19.0" }
-secretenv-registry-mutate    = { path = "crates/secretenv-registry-mutate",                version = "0.19.0" }
-secretenv-mcp-config         = { path = "crates/secretenv-mcp-config",                     version = "0.19.0" }
-secretenv-backends-init      = { path = "crates/secretenv-backends-init",                  version = "0.19.0" }
-secretenv-mcp                = { path = "crates/secretenv-mcp",                            version = "0.19.0" }
-secretenv-backend-local      = { path = "crates/backends/secretenv-backend-local",         version = "0.19.0" }
-secretenv-backend-aws-ssm    = { path = "crates/backends/secretenv-backend-aws-ssm",       version = "0.19.0" }
-secretenv-backend-1password  = { path = "crates/backends/secretenv-backend-1password",     version = "0.19.0" }
-secretenv-backend-vault      = { path = "crates/backends/secretenv-backend-vault",         version = "0.19.0" }
-secretenv-backend-aws-secrets = { path = "crates/backends/secretenv-backend-aws-secrets",  version = "0.19.0" }
-secretenv-backend-gcp        = { path = "crates/backends/secretenv-backend-gcp",           version = "0.19.0" }
-secretenv-backend-azure      = { path = "crates/backends/secretenv-backend-azure",         version = "0.19.0" }
-secretenv-backend-keychain   = { path = "crates/backends/secretenv-backend-keychain",      version = "0.19.0" }
-secretenv-backend-doppler    = { path = "crates/backends/secretenv-backend-doppler",       version = "0.19.0" }
-secretenv-backend-infisical  = { path = "crates/backends/secretenv-backend-infisical",     version = "0.19.0" }
-secretenv-backend-keeper     = { path = "crates/backends/secretenv-backend-keeper",        version = "0.19.0" }
-secretenv-backend-cf-kv      = { path = "crates/backends/secretenv-backend-cf-kv",         version = "0.19.0" }
-secretenv-backend-openbao    = { path = "crates/backends/secretenv-backend-openbao",       version = "0.19.0" }
-secretenv-backend-conjur     = { path = "crates/backends/secretenv-backend-conjur",        version = "0.19.0" }
-secretenv-backend-bitwarden-sm = { path = "crates/backends/secretenv-backend-bitwarden-sm", version = "0.19.0" }
+secretenv-core               = { path = "crates/secretenv-core",                           version = "0.20.0", features = ["value-access"] }
+secretenv-telemetry          = { path = "crates/secretenv-telemetry",                      version = "0.20.0" }
+secretenv-migrate            = { path = "crates/secretenv-migrate",                        version = "0.20.0" }
+secretenv-registry-mutate    = { path = "crates/secretenv-registry-mutate",                version = "0.20.0" }
+secretenv-mcp-config         = { path = "crates/secretenv-mcp-config",                     version = "0.20.0" }
+secretenv-backends-init      = { path = "crates/secretenv-backends-init",                  version = "0.20.0" }
+secretenv-mcp                = { path = "crates/secretenv-mcp",                            version = "0.20.0" }
+secretenv-backend-local      = { path = "crates/backends/secretenv-backend-local",         version = "0.20.0" }
+secretenv-backend-aws-ssm    = { path = "crates/backends/secretenv-backend-aws-ssm",       version = "0.20.0" }
+secretenv-backend-1password  = { path = "crates/backends/secretenv-backend-1password",     version = "0.20.0" }
+secretenv-backend-vault      = { path = "crates/backends/secretenv-backend-vault",         version = "0.20.0" }
+secretenv-backend-aws-secrets = { path = "crates/backends/secretenv-backend-aws-secrets",  version = "0.20.0" }
+secretenv-backend-gcp        = { path = "crates/backends/secretenv-backend-gcp",           version = "0.20.0" }
+secretenv-backend-azure      = { path = "crates/backends/secretenv-backend-azure",         version = "0.20.0" }
+secretenv-backend-keychain   = { path = "crates/backends/secretenv-backend-keychain",      version = "0.20.0" }
+secretenv-backend-doppler    = { path = "crates/backends/secretenv-backend-doppler",       version = "0.20.0" }
+secretenv-backend-infisical  = { path = "crates/backends/secretenv-backend-infisical",     version = "0.20.0" }
+secretenv-backend-keeper     = { path = "crates/backends/secretenv-backend-keeper",        version = "0.20.0" }
+secretenv-backend-cf-kv      = { path = "crates/backends/secretenv-backend-cf-kv",         version = "0.20.0" }
+secretenv-backend-openbao    = { path = "crates/backends/secretenv-backend-openbao",       version = "0.20.0" }
+secretenv-backend-conjur     = { path = "crates/backends/secretenv-backend-conjur",        version = "0.20.0" }
+secretenv-backend-bitwarden-sm = { path = "crates/backends/secretenv-backend-bitwarden-sm", version = "0.20.0" }
 # Unpublished test harness — path-only, no version. Revisit publishing once
 # the API has proven itself across Vault + AWS Secrets backends (v0.3+).
 secretenv-testing            = { path = "crates/secretenv-testing" }

crates/backends/secretenv-backend-azure/src/lib.rs

@@ -300,7 +300,17 @@ impl AzureBackend {
                 uri.raw
             )
         })?;
-        Ok(value.strip_suffix('\n').unwrap_or(&value).to_owned())
+        // Do NOT strip a trailing newline here.
+        //
+        // This value came out of `az ... --output json`, parsed above.
+        // A JSON string field carries the secret's exact bytes; there
+        // is no CLI-appended newline to remove, so the strip could only
+        // ever destroy data — it ate the final byte of any value ending
+        // in a newline (PEM, certs, SSH keys). This backend was the
+        // clearest case of the six: unlike the raw-stdout backends,
+        // no trailing newline is ever added on this path at all.
+        // Verified live (v0.19.1).
+        Ok(value)
     }
 }
 
@@ -988,16 +998,41 @@ mod tests {
     }
 
     #[tokio::test]
-    async fn get_strips_single_trailing_newline() {
+    async fn get_returns_json_value_verbatim() {
+        // The value comes out of `az ... --output json`, so it carries
+        // the secret's exact bytes — there is no CLI-appended newline
+        // to remove. The old code stripped one anyway, which could only
+        // ever destroy data.
         let dir = TempDir::new().unwrap();
         let mock = StrictMock::new("az")
-            .on(&show_argv("multi-line"), Response::success("{\"value\":\"line1\\nline2\\n\"}\n"))
+            .on(&show_argv("multi-line"), Response::success("{\"value\":\"line1\\nline2\"}\n"))
             .install(dir.path());
         let b = backend(&mock, None, None);
         let uri = BackendUri::parse("azure-prod:///multi-line").unwrap();
         assert_eq!(b.get(&uri).await.unwrap().expose_secret(), "line1\nline2");
     }
 
+    /// Regression: a value whose final byte is a legitimate newline must
+    /// survive. The old code stripped one '\n' unconditionally, silently
+    /// truncating every PEM block, certificate and SSH key. Fails on the
+    /// pre-fix code.
+    #[tokio::test]
+    async fn get_preserves_legitimate_trailing_newline() {
+        let dir = TempDir::new().unwrap();
+        let mock = StrictMock::new("az")
+            .on(
+                &show_argv("pem"),
+                Response::success("{\"value\":\"-----BEGIN X-----\\nabc\\n-----END X-----\\n\"}\n"),
+            )
+            .install(dir.path());
+        let b = backend(&mock, None, None);
+        let uri = BackendUri::parse("azure-prod:///pem").unwrap();
+        assert_eq!(
+            b.get(&uri).await.unwrap().expose_secret(),
+            "-----BEGIN X-----\nabc\n-----END X-----\n"
+        );
+    }
+
     #[tokio::test]
     async fn get_empty_value() {
         let dir = TempDir::new().unwrap();