From c2775c4cc11e21cc8a5314fdd982fc9dcd4c898e Mon Sep 17 00:00:00 2001 From: Brage Date: Tue, 19 May 2026 13:45:10 +0200 Subject: [PATCH 1/2] mTLS certificate rotation guide --- en/security/guide.md | 83 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) diff --git a/en/security/guide.md b/en/security/guide.md index 8dcaebde2a..a88612d06b 100644 --- a/en/security/guide.md +++ b/en/security/guide.md @@ -60,11 +60,94 @@ The certificate is placed inside the application package in `clients.pem` is placed correctly if the certificate is created with OpenSSL, while the Vespa CLI will handle this automatically. +### Rotating mTLS certificates + +This section provides a guide on how to rotate the certificate used for mTLS. +This is useful if you want to change certificates on a Vespa deployment without downtime. + `security/clients.pem` files can contain multiple PEM encoded certificates by concatenating them. This allows you to have multiple clients with separate private keys, making it possible to rotate to a new certificate without any downtime. +#### Rotate mTLS with Vespa CLI + +**Step 1: Generate a new certificate and private key** + +```bash +$ vespa auth cert --append --application scoober.albums.default +``` + +**Step 2: Deploy the updated application package** + +Both old and new certificates are now accepted, so existing clients continue to work without interruption. +Deploy the application. + +**Step 3: Migrate clients to the new certificate** + +Test that everything works with the new key and update all clients to use the new private key. + +**Step 4: Remove old certificates** + +Once all clients use the new key, remove the old certificates: + +```bash +$ vespa auth cert --prune --application scoober.albums.default +``` + +Deploy again to complete the rotation. Any clients still using the old certificate will lose access once the old certificate is removed and deployed. + +#### Rotate mTLS with OpenSSL + +**Step 1: Generate a new certificate and private key** + +```bash +$ openssl req -x509 -sha256 -days 1825 -newkey rsa:2048 -keyout new-key.pem -out new-cert.pem +``` + +**Step 2: Add the new certificate to `security/clients.pem`** + +Concatenate the existing and new certificates into a single `clients.pem`. +Both certificates will be trusted simultaneously: + +```bash +$ cat new-cert.pem security/clients.pem > security/clients-combined.pem +$ mv security/clients-combined.pem security/clients.pem +``` + +The resulting `clients.pem` will look like: + +``` +-----BEGIN CERTIFICATE----- + +-----END CERTIFICATE----- +-----BEGIN CERTIFICATE----- + +-----END CERTIFICATE----- +``` + +**Step 3: Deploy the updated application package** + +Deploy the application with the combined `clients.pem`. Both old and new +certificates are now accepted, so existing clients continue to work without +interruption or downtime. + +**Step 4: Migrate clients to the new certificate** + +Update all clients to use the new private key. Once all clients use the new +certificate, the old certificate can be removed. + +**Step 5: Remove the old certificate** + +Remove the old certificate from `security/clients.pem`, leaving only the new one: + +``` +$ cp new-cert.pem security/clients.pem +``` + +Deploy again to complete the rotation. The old certificate is no longer trusted. + + ### Permissions From bc8257460df6822e76e648b38b000318826a9ba3 Mon Sep 17 00:00:00 2001 From: Brage Date: Tue, 4 Aug 2026 12:48:40 +0200 Subject: [PATCH 2/2] rotation guide uses newest flags --- en/security/guide.md | 29 +++++++++++++++++++++++------ 1 file changed, 23 insertions(+), 6 deletions(-) diff --git a/en/security/guide.md b/en/security/guide.md index a88612d06b..960b924f83 100644 --- a/en/security/guide.md +++ b/en/security/guide.md @@ -36,9 +36,9 @@ $ vespa auth cert --application .. ``` $ vespa auth cert --application scoober.albums.default -Success: Certificate written to security/clients.pem -Success: Certificate written to $HOME/.vespa/scoober.albums.default/data-plane-public-cert.pem -Success: Private key written to $HOME/.vespa/scoober.albums.default/data-plane-private-key.pem +Success: Certificate written to '$HOME/.vespa/scoober.albums.default/data-plane-public-cert.pem' +Success: Private key written to '$HOME/.vespa/scoober.albums.default/data-plane-private-key.pem' +Success: Copied certificate from '$HOME/.vespa/scoober.albums.default/data-plane-public-cert.pem' to 'security/clients.pem' ``` The certificates can be created regardless of the application existence in Vespa @@ -75,9 +75,19 @@ downtime. **Step 1: Generate a new certificate and private key** ```bash -$ vespa auth cert --append --application scoober.albums.default +$ vespa auth cert --new-key --application scoober.albums.default ``` +This prompts for confirmation, then: +* backs up the current private key to `data-plane-private-key.pem.old` +* writes a new private key to `data-plane-private-key.pem` +* prepends the new certificate to `data-plane-public-cert.pem`, keeping the existing one +* copies the resulting certificate file into `security/clients.pem` of the application package + +Add `-f` (`--force`) to skip the confirmation prompt, and `-N` (`--no-add`) to skip updating +the application package. If a backup already exists, the +command fails - complete or abandon the previous rotation and delete the backup file first. + **Step 2: Deploy the updated application package** Both old and new certificates are now accepted, so existing clients continue to work without interruption. @@ -86,16 +96,23 @@ Deploy the application. **Step 3: Migrate clients to the new certificate** Test that everything works with the new key and update all clients to use the new private key. +The Vespa CLI already uses the new key pair after step 1. **Step 4: Remove old certificates** Once all clients use the new key, remove the old certificates: ```bash -$ vespa auth cert --prune --application scoober.albums.default +$ vespa auth cert --prune-old --application scoober.albums.default ``` +This keeps the newest certificate matching the current private key, and asks before removing +each old certificate. + +Add `-f` (`--force`) to remove all of them without prompting. + Deploy again to complete the rotation. Any clients still using the old certificate will lose access once the old certificate is removed and deployed. +When the deployment is complete, you may delete the `data-plane-private-key.pem.old` backup. A new rotation cannot start while it exists. #### Rotate mTLS with OpenSSL @@ -314,7 +331,7 @@ $ vespa curl --application .. /ApplicationStatus ``` $ curl --key $HOME/.vespa/scoober.albums.default/data-plane-private-key.pem \ - --cert $HOME/.vespa/scoober.albums.default/data-plane-public-key.pem \ + --cert $HOME/.vespa/scoober.albums.default/data-plane-public-cert.pem \ $ENDPOINT ```