From 911a0c3c18d8a94ba7d5d7a2cbe2eb8e11bf5bed Mon Sep 17 00:00:00 2001 From: PavelZeger Date: Sat, 13 Jun 2026 12:21:38 +0300 Subject: [PATCH 1/3] [improve][pip] PIP-476: Configurable mTLS principal mapping (SAN sources and DN mapping rules) --- pip/pip-476.md | 220 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 220 insertions(+) create mode 100644 pip/pip-476.md diff --git a/pip/pip-476.md b/pip/pip-476.md new file mode 100644 index 0000000000000..4116cf6e5834a --- /dev/null +++ b/pip/pip-476.md @@ -0,0 +1,220 @@ +# PIP-476: Configurable mTLS principal mapping (SAN sources and DN mapping rules) + +# Background knowledge + +When a Pulsar broker authenticates a client, it asks an `AuthenticationProvider` (in `pulsar-broker-common`) a simple question: "who is this client?". The provider looks at the connection and returns one `String` - the client's **role** (also called the principal). Pulsar then uses that role for two things: the `AuthorizationProvider` decides what the role is allowed to do, and the role is compared against the `superUserRoles` and `proxyRoles` lists in the config. So the role string is the link between "who you are" and "what you can do." + +With mTLS (mutual TLS), the client proves who it is by presenting an X.509 certificate during the TLS handshake. A certificate carries identity in a couple of places: + +- The **Subject**, written as a **Distinguished Name (DN)**, for example `CN=alice,OU=payments,O=Acme,C=US`. The DN is a list of small pieces called **RDNs**. Each RDN is one attribute: **CN** (Common Name), **OU** (Organizational Unit), **O** (Organization), **C** (Country), `emailAddress`, and so on. +- The **Subject Alternative Names (SANs)**, a separate part of the certificate that holds *typed* identities: a DNS name, an email address, a URI, or an IP address. + +These days the real identity often lives in a SAN, not in the CN. In service meshes and workload-identity systems like **SPIFFE/SPIRE**, the identity is a URI SAN such as `spiffe://acme.com/ns/payments/sa/checkout`, and the CN is usually empty. [RFC 9525] (the current standard, which replaced [RFC 6125] in 2023) actually says you should stop using the CN for identity and use SANs instead. (For the curious: the DN text format that Java's `X500Principal.getName()` emits is [RFC 2253] - note RFC 2253 was later obsoleted by [RFC 4514] as the general standard, but the Java API still defines this method's output in RFC 2253 terms. The SAN types are listed in [RFC 5280] §4.2.1.6 - email is type 1, DNS is type 2, URI is type 6.) + +For comparison, Apache Kafka lets operators control this with a config option called `ssl.principal.mapping.rules` ([KIP-371]). It's an ordered list of rules, each written as `RULE:pattern/replacement/[LU]`, applied to the certificate's DN. A special `DEFAULT` rule means "just use the whole DN." This lets people map their existing certificates to Kafka principals with a few lines of config instead of re-issuing certificates. Kafka also has a programmatic `KafkaPrincipalBuilder` SPI ([KIP-189]) for people who want to write code instead. This PIP copies the *config* idea from KIP-371, not the SPI from KIP-189 - Pulsar already has its own SPI (see *Alternatives*). + +# Motivation + +Pulsar's built-in TLS provider, `AuthenticationProviderTls`, only ever looks at the **Common Name**, and the way it reads it is hardcoded (`pulsar-broker-common/.../authentication/AuthenticationProviderTls.java`, +`authenticate()`): + +```java +public String authenticate(AuthenticationDataSource authData) throws AuthenticationException { + String commonName = null; + ErrorCode errorCode = ErrorCode.UNKNOWN; + try { + if (authData.hasDataFromTls()) { + // javadoc + Certificate[] certs = authData.getTlsCertificates(); + if (null == certs) { + errorCode = ErrorCode.INVALID_CERTS; + throw new AuthenticationException("Failed to get TLS certificates from client"); + } + String distinguishedName = ((X509Certificate) certs[0]).getSubjectX500Principal().getName(); + for (String keyValueStr : distinguishedName.split(",")) { + String[] keyValue = keyValueStr.split("=", 2); + if (keyValue.length == 2 && "CN".equals(keyValue[0]) && !keyValue[1].isEmpty()) { + commonName = keyValue[1]; + break; + } + } + } + + if (commonName == null) { + errorCode = ErrorCode.INVALID_CN; + throw new AuthenticationException("Client unable to authenticate with TLS certificate"); + } + authenticationMetrics.recordSuccess(); + } catch (AuthenticationException exception) { + incrementFailureMetric(errorCode); + throw exception; + } + return commonName; +} +``` + +That "first CN wins, otherwise fail" logic is a problem for two kinds of users that are becoming more common: + +1. **Companies with an existing PKI**, where the identity they care about is in some other field - an `OU`, an `emailAddress`, or the full DN. They often can't just re-issue thousands of certificates to move that value into the CN. +2. **SPIFFE/SPIRE and service meshes**, where the identity is a URI SAN (`spiffe://...`) and the CN is empty on purpose. Right now those certificates are rejected outright by the `commonName == null` check, even though they carry a perfectly valid, cryptographically-signed identity. + +You *can* work around this today by writing your own `AuthenticationProvider`, since the SPI lets you return any role string you want. But writing and maintaining a custom provider is a lot of work for something Kafka gives you in a few lines of config. The built-in provider - which is what almost everyone actually uses - should be able to read identity from a SAN and apply simple mapping rules. + +# Goals + +## In Scope + +- Let operators pick **which part of the certificate** holds the identity: the CN (today's behavior), the full DN, or a specific SAN type (URI / DNS / email). +- Let operators apply an ordered list of **mapping rules** to turn that raw value into the final Pulsar role, using the same rule format as Kafka so existing knowledge carries over. +- **Keep today's behavior as the default.** If you don't set anything new, nothing changes. +- Make it work the same way everywhere the provider is used: the binary TLS path, the HTTPS client-cert path, and the **Pulsar Proxy**. + +## Out of Scope + +- A general `PrincipalBuilder` SPI like Kafka's. Pulsar already has the `AuthenticationProvider` SPI for full customization; this PIP just improves the built-in provider. +- Anything about TLS trust, certificate chains, revocation (CRL/OCSP), or ciphers. By the time `authenticate()` runs, the TLS layer has already validated the certificate. We only change how the identity is *read* from it. +- Returning more than one role / groups from a certificate. The provider still returns a single role. + +# High Level Design + +We add two config options that `AuthenticationProviderTls` reads: + +1. **`tlsCertIdentitySource`** - picks the raw identity from the certificate. One of `CN` (default), `DN`, or `SAN:` where `` is `URI`, `DNS`, or `EMAIL`. +2. **`tlsAuthPrincipalMappingRules`** - an ordered list of rules that transform that raw value into the final role. The format matches Kafka's ([KIP-371]): each rule is either `DEFAULT` (use the value as-is) or `RULE://[LU]` (if the regex matches, build the replacement from the captured groups, optionally lower-cased with `L` or upper-cased with `U`). Rules run top to bottom and the first match wins. If nothing matches and there's no `DEFAULT`, authentication fails with a clear message. + +If you set neither option, the provider does exactly what it does today (read the first CN, fail if it's missing). So this is purely additive and safe to upgrade into. The final role string flows into authorization just like it does now, so nothing downstream needs to change. + +# Detailed Design + +## Design & Implementation Details + +### Reading the identity + +Add a small helper, `TlsIdentityExtractor`, called from `AuthenticationProviderTls.authenticate()` right after the existing null-check on the certificate: + +- `CN` - the current logic (first `CN=` part of the subject). +- `DN` - the full subject string from `((X509Certificate) certs[0]).getSubjectX500Principal().getName()` ([RFC 2253] format), used as-is. +- `SAN:URI` / `SAN:DNS` / `SAN:EMAIL` - read `X509Certificate.getSubjectAlternativeNames()` and take the first SAN of the matching type ([RFC 5280]: URI = 6, DNS = 2, email = 1). If there are several of the same type, the first one in the certificate is used, and a mapping rule can sort out which one you want. (Pulsar already reads SANs this way in `org.apache.pulsar.common.tls.TlsHostnameVerifier`, so this isn't new ground.) + +If the chosen source has no value (for example, you asked for a URI SAN but the certificate doesn't have one), authentication fails with a message that says which source was missing, and a failure metric is recorded with a specific error code. + +### Applying the rules + +A second helper, `PrincipalMappingRules`, parses `tlsAuthPrincipalMappingRules` once at `initialize()` and compiles the regexes into an ordered list. `apply(identity)`returns the first rule that matches. The format and behavior match Kafka's on purpose, so the docs and the operator's existing knowledge carry over. Example: + +``` +tlsCertIdentitySource = SAN:URI +tlsAuthPrincipalMappingRules = RULE:^spiffe://acme\.com/ns/([^/]+)/sa/([^/]+)$/$1__$2/L,DEFAULT +``` + +Here the SPIFFE id `spiffe://acme.com/ns/payments/sa/checkout` becomes the role `payments__checkout`. + +### Where it hooks in + +Only `AuthenticationProviderTls.authenticate(AuthenticationDataSource)` changes. The `AuthenticationProvider` interface, the `AuthenticationDataSource`, and every caller stay the same. Because HTTPS client-cert requests go through this same provider, both transports are covered by the one change. + +**The proxy needs the config too - this is real work, not free.** The Pulsar Proxy builds its `AuthenticationService` from a `ServiceConfiguration` that it gets by calling `PulsarConfigurationLoader.convertFrom(ProxyConfiguration)`. That conversion copies fields **by matching name**. So the two new keys have to be added to *both* `ServiceConfiguration` and `ProxyConfiguration`, with the same names. If they're only in `ServiceConfiguration`, the proxy quietly ignores them and falls back to CN - and then the proxy and the broker can map the same certificate to different roles. A test that goes through the proxy should confirm the config actually takes effect. + +## Public-facing Changes + +### Public API + +None. No new REST endpoint, no client API change, no change to the SPI signatures. + +### Binary protocol + +None. The TLS handshake and the auth command don't change. The only difference is how +the broker reads the certificate it already received. + +### Configuration + +Two new properties in `broker.conf` / `ServiceConfiguration`, also added to `proxy.conf` / `ProxyConfiguration` (see the proxy note above): + +- `tlsCertIdentitySource` - one of `CN` (default), `DN`, `SAN:URI`, `SAN:DNS`, `SAN:EMAIL`. Picks the raw identity from the certificate. +- `tlsAuthPrincipalMappingRules` - comma-separated, ordered list of `DEFAULT` / `RULE://[LU]`. Empty by default, which means "use the value as-is" and keeps today's behavior for `CN`. + +### CLI + +None. + +### Metrics + +Reuse the provider's existing `AuthenticationMetrics` failure path. The provider already has an `ErrorCode` enum (`UNKNOWN`, `INVALID_CERTS`, `INVALID_CN`); add two codes so operators can tell failures apart: + +- `INVALID_CN` (existing) - CN source chosen but no CN present. +- `NO_SAN_OF_TYPE` (new) - SAN source chosen but no SAN of that type present. +- `NO_MAPPING_RULE_MATCHED` (new) - identity read, but no rule (and no `DEFAULT`) matched. + +These show up on the existing per-provider failure counter, labeled by error code. + +# Monitoring + +After rolling out a mapping config, operators should watch the new failure error codes. A jump in `NO_MAPPING_RULE_MATCHED` or `NO_SAN_OF_TYPE` usually means a rule or source is misconfigured and is now rejecting clients that should be let in. When migrating from CN to SAN/DN identity, keep an eye on that per-code counter to confirm the new rules cover all your certificates before you remove any `DEFAULT` fallback. + +# Security Considerations + +This PIP changes how a certificate turns into a role, so it deserves a careful security review: + +- **A bad rule can hand out the wrong role.** A regex that's too loose (like `RULE:.*//admin/`) could map lots of certificates to a powerful role. The docs need to push people toward anchored patterns (`^...$`) and remind them that order matters. Rules are compiled at startup, so a broken rule stops the broker from booting instead of failing silently later. +- **Changing the role string can change who's a super-user or proxy.** `superUserRoles` and `proxyRoles` are matched by exact string. If you change the identity source, a certificate that used to map to a super-user role under CN might not anymore (or, with a sloppy rule, something might newly become one). Re-check those lists whenever you change the source. +- **This never weakens certificate validation.** The identity is only read *after* the TLS layer has already validated and trusted the certificate. Picking a SAN doesn't skip any of that. +- **The empty-CN change only happens if you opt in.** Today an empty CN fails. With a SAN or DN source, those certificates can now authenticate. But that only happens when an operator deliberately sets a non-CN source, so no existing setup changes on its own. +- **No change to multi-tenancy.** The role string is the same one authorization already uses, and tenant isolation is enforced downstream exactly as before. + +If anything here is unclear, we should confirm the rule format and SAN handling on the mailing list before merging. + +# Drawbacks and Pitfalls + +It's worth being honest about the downsides and the easy ways to get this wrong. + +## Drawbacks (cons) + +- **More config to get right.** Two new options on a security-sensitive provider means two more things an operator can misconfigure. The default keeps today's behavior, but anyone who turns this on takes on that responsibility. +- **Regex-on-a-DN is powerful but fiddly.** Mapping rules give a lot of freedom, and freedom means foot-guns. Kafka has lived with the same trade-off, so it's well-understood, but it's still a sharp tool. +- **Still one role per certificate.** If you're a SPIFFE shop that wants groups or multiple roles from one certificate, this doesn't get you there. That's a separate problem. +- **Doesn't touch revocation.** This is only about reading identity. CRL/OCSP and certificate trust are out of scope, so don't expect this to improve them. + +## Pitfalls (easy mistakes) + +- **Forgetting the proxy.** If you add the keys to the broker but not the proxy, the proxy silently falls back to C and you get a split-brain where the proxy and broker disagree on a client's role. Always configure both, and test through the proxy. +- **The DN isn't the string you think it is.** `X500Principal.getName()` returns the [RFC 2253] form, which (a) lists the RDNs in *reverse* order from how people usually write them, (b) escapes special characters, and (c) renders attributes with no short name as `OID=#hexvalue`. A rule written against a "nice-looking" DN may never match the real one. Write your rules against the actual canonical string, and the docs should show realistic examples. +- **"First SAN of that type" when there are several.** A certificate can carry more than one URI (or DNS, or email) SAN. We take the first one in certificate order, so if order matters to you, use a mapping rule to pin down the one you want. +- **Too-broad rules leak privilege.** This is the security point above, repeated here because it's the most common mistake: anchor your patterns and order your rules so a catch-all doesn't accidentally grant `admin`. +- **Rollback can lock people out.** If certificates authenticated *only* because of a SAN/DN source plus rules, they'll stop working after a downgrade to a version without this feature, or after you remove the config. If you need rollback safety, keep a working CN in those certificates as a fallback. + +# Backward & Forward Compatibility + +## Upgrade + +Nothing to do. If you leave the new options unset, `AuthenticationProviderTls` behaves exactly like before (first CN, fail on empty CN). You opt in by setting `tlsCertIdentitySource` / `tlsAuthPrincipalMappingRules`. + +## Downgrade / Rollback + +Remove the two properties, or downgrade the broker/proxy, and behavior goes back to CN-only. Be careful: any certificate that authenticated only because of a SAN/DN source plus rules will stop authenticating after rollback. If you need to be able to roll back, keep a usable CN in those certificates (see *Pitfalls*). + +## Pulsar Geo-Replication Upgrade & Downgrade/Rollback Considerations + +This is a per-broker (and per-proxy) setting. In a geo-replicated cluster, each broker reads roles from the certificates presented to *it* - brokers replicate data, not TLS sessions, so there's no cross-cluster coupling. Just make sure brokers (and proxies) that might serve the same clients use the same mapping config, so a client doesn't get a different role depending on which broker it happens to hit. + +# Alternatives + +- **Add a full `PrincipalBuilder` SPI like Kafka's [KIP-189].** More flexible, but Pulsar already has `AuthenticationProvider` as its extension point. Adding a second SPI for the same job would be redundant. Improving the built-in provider covers the common cases with no code from operators. +- **Ship a separate `AuthenticationProviderTlsSan` provider.** Rejected. It forks the TLS provider and forces operators to choose between providers instead of just configuring one identity policy. One configurable provider is simpler. +- **Hardcode "try SAN first, then CN."** Rejected. It's too opinionated - different organizations want different sources - and silently changing the default would be a breaking change. + +# General Notes + +The change is deliberately small and contained: one helper to read the identity, one helper to compile the rules, two new config keys (mirrored in `ServiceConfiguration` and `ProxyConfiguration`), and two new failure error codes, all behind `AuthenticationProviderTls`. Reusing Kafka's rule format is a conscious choice to make life easier for people who run both systems. + +# Links + +* Apache Kafka KIP-371 - *Add a configuration to build custom SSL principal name* (`ssl.principal.mapping.rules`): https://cwiki.apache.org/confluence/display/KAFKA/KIP-371%3A+Add+a+configuration+to+build+custom+SSL+principal+name +* Apache Kafka KIP-189 - *Improve principal builder interface and add support for SASL* (`KafkaPrincipalBuilder` SPI): https://cwiki.apache.org/confluence/display/KAFKA/KIP-189%3A+Improve+principal+builder+interface+and+add+support+for+SASL +* RFC 9525 - *Service Identity in TLS* (current standard; obsoletes RFC 6125 - use SAN, not CN, for identity): https://www.rfc-editor.org/rfc/rfc9525 +* RFC 6125 - *Representation and Verification of Domain-Based Application Service Identity ...* (obsoleted by RFC 9525; kept here for historical reference): https://www.rfc-editor.org/rfc/rfc6125 +* RFC 2253 - *LDAPv3 UTF-8 String Representation of Distinguished Names* (the DN string format that `X500Principal.getName()` is defined to return; obsoleted as a general standard by RFC 4514, but still the format the Java API emits): https://www.rfc-editor.org/rfc/rfc2253 +* RFC 4514 - *LDAP: String Representation of Distinguished Names* (current standard; obsoletes RFC 2253): https://www.rfc-editor.org/rfc/rfc4514 +* RFC 5280 - *Internet X.509 PKI Certificate and CRL Profile*, §4.2.1.6 Subject Alternative Name (SAN types): https://www.rfc-editor.org/rfc/rfc5280#section-4.2.1.6 +* SPIFFE ID specification (URI-SAN workload identity): https://github.com/spiffe/spiffe/blob/main/standards/SPIFFE-ID.md +* Mailing List discussion thread: TBD +* Mailing List voting thread: TBD From 9856c6c739966c40313d50387b16dc5d9c7c326d Mon Sep 17 00:00:00 2001 From: PavelZeger Date: Sat, 13 Jun 2026 12:24:47 +0300 Subject: [PATCH 2/3] Revert "[improve][pip] PIP-476: Configurable mTLS principal mapping (SAN sources and DN mapping rules)" This reverts commit 911a0c3c18d8a94ba7d5d7a2cbe2eb8e11bf5bed. --- pip/pip-476.md | 220 ------------------------------------------------- 1 file changed, 220 deletions(-) delete mode 100644 pip/pip-476.md diff --git a/pip/pip-476.md b/pip/pip-476.md deleted file mode 100644 index 4116cf6e5834a..0000000000000 --- a/pip/pip-476.md +++ /dev/null @@ -1,220 +0,0 @@ -# PIP-476: Configurable mTLS principal mapping (SAN sources and DN mapping rules) - -# Background knowledge - -When a Pulsar broker authenticates a client, it asks an `AuthenticationProvider` (in `pulsar-broker-common`) a simple question: "who is this client?". The provider looks at the connection and returns one `String` - the client's **role** (also called the principal). Pulsar then uses that role for two things: the `AuthorizationProvider` decides what the role is allowed to do, and the role is compared against the `superUserRoles` and `proxyRoles` lists in the config. So the role string is the link between "who you are" and "what you can do." - -With mTLS (mutual TLS), the client proves who it is by presenting an X.509 certificate during the TLS handshake. A certificate carries identity in a couple of places: - -- The **Subject**, written as a **Distinguished Name (DN)**, for example `CN=alice,OU=payments,O=Acme,C=US`. The DN is a list of small pieces called **RDNs**. Each RDN is one attribute: **CN** (Common Name), **OU** (Organizational Unit), **O** (Organization), **C** (Country), `emailAddress`, and so on. -- The **Subject Alternative Names (SANs)**, a separate part of the certificate that holds *typed* identities: a DNS name, an email address, a URI, or an IP address. - -These days the real identity often lives in a SAN, not in the CN. In service meshes and workload-identity systems like **SPIFFE/SPIRE**, the identity is a URI SAN such as `spiffe://acme.com/ns/payments/sa/checkout`, and the CN is usually empty. [RFC 9525] (the current standard, which replaced [RFC 6125] in 2023) actually says you should stop using the CN for identity and use SANs instead. (For the curious: the DN text format that Java's `X500Principal.getName()` emits is [RFC 2253] - note RFC 2253 was later obsoleted by [RFC 4514] as the general standard, but the Java API still defines this method's output in RFC 2253 terms. The SAN types are listed in [RFC 5280] §4.2.1.6 - email is type 1, DNS is type 2, URI is type 6.) - -For comparison, Apache Kafka lets operators control this with a config option called `ssl.principal.mapping.rules` ([KIP-371]). It's an ordered list of rules, each written as `RULE:pattern/replacement/[LU]`, applied to the certificate's DN. A special `DEFAULT` rule means "just use the whole DN." This lets people map their existing certificates to Kafka principals with a few lines of config instead of re-issuing certificates. Kafka also has a programmatic `KafkaPrincipalBuilder` SPI ([KIP-189]) for people who want to write code instead. This PIP copies the *config* idea from KIP-371, not the SPI from KIP-189 - Pulsar already has its own SPI (see *Alternatives*). - -# Motivation - -Pulsar's built-in TLS provider, `AuthenticationProviderTls`, only ever looks at the **Common Name**, and the way it reads it is hardcoded (`pulsar-broker-common/.../authentication/AuthenticationProviderTls.java`, -`authenticate()`): - -```java -public String authenticate(AuthenticationDataSource authData) throws AuthenticationException { - String commonName = null; - ErrorCode errorCode = ErrorCode.UNKNOWN; - try { - if (authData.hasDataFromTls()) { - // javadoc - Certificate[] certs = authData.getTlsCertificates(); - if (null == certs) { - errorCode = ErrorCode.INVALID_CERTS; - throw new AuthenticationException("Failed to get TLS certificates from client"); - } - String distinguishedName = ((X509Certificate) certs[0]).getSubjectX500Principal().getName(); - for (String keyValueStr : distinguishedName.split(",")) { - String[] keyValue = keyValueStr.split("=", 2); - if (keyValue.length == 2 && "CN".equals(keyValue[0]) && !keyValue[1].isEmpty()) { - commonName = keyValue[1]; - break; - } - } - } - - if (commonName == null) { - errorCode = ErrorCode.INVALID_CN; - throw new AuthenticationException("Client unable to authenticate with TLS certificate"); - } - authenticationMetrics.recordSuccess(); - } catch (AuthenticationException exception) { - incrementFailureMetric(errorCode); - throw exception; - } - return commonName; -} -``` - -That "first CN wins, otherwise fail" logic is a problem for two kinds of users that are becoming more common: - -1. **Companies with an existing PKI**, where the identity they care about is in some other field - an `OU`, an `emailAddress`, or the full DN. They often can't just re-issue thousands of certificates to move that value into the CN. -2. **SPIFFE/SPIRE and service meshes**, where the identity is a URI SAN (`spiffe://...`) and the CN is empty on purpose. Right now those certificates are rejected outright by the `commonName == null` check, even though they carry a perfectly valid, cryptographically-signed identity. - -You *can* work around this today by writing your own `AuthenticationProvider`, since the SPI lets you return any role string you want. But writing and maintaining a custom provider is a lot of work for something Kafka gives you in a few lines of config. The built-in provider - which is what almost everyone actually uses - should be able to read identity from a SAN and apply simple mapping rules. - -# Goals - -## In Scope - -- Let operators pick **which part of the certificate** holds the identity: the CN (today's behavior), the full DN, or a specific SAN type (URI / DNS / email). -- Let operators apply an ordered list of **mapping rules** to turn that raw value into the final Pulsar role, using the same rule format as Kafka so existing knowledge carries over. -- **Keep today's behavior as the default.** If you don't set anything new, nothing changes. -- Make it work the same way everywhere the provider is used: the binary TLS path, the HTTPS client-cert path, and the **Pulsar Proxy**. - -## Out of Scope - -- A general `PrincipalBuilder` SPI like Kafka's. Pulsar already has the `AuthenticationProvider` SPI for full customization; this PIP just improves the built-in provider. -- Anything about TLS trust, certificate chains, revocation (CRL/OCSP), or ciphers. By the time `authenticate()` runs, the TLS layer has already validated the certificate. We only change how the identity is *read* from it. -- Returning more than one role / groups from a certificate. The provider still returns a single role. - -# High Level Design - -We add two config options that `AuthenticationProviderTls` reads: - -1. **`tlsCertIdentitySource`** - picks the raw identity from the certificate. One of `CN` (default), `DN`, or `SAN:` where `` is `URI`, `DNS`, or `EMAIL`. -2. **`tlsAuthPrincipalMappingRules`** - an ordered list of rules that transform that raw value into the final role. The format matches Kafka's ([KIP-371]): each rule is either `DEFAULT` (use the value as-is) or `RULE://[LU]` (if the regex matches, build the replacement from the captured groups, optionally lower-cased with `L` or upper-cased with `U`). Rules run top to bottom and the first match wins. If nothing matches and there's no `DEFAULT`, authentication fails with a clear message. - -If you set neither option, the provider does exactly what it does today (read the first CN, fail if it's missing). So this is purely additive and safe to upgrade into. The final role string flows into authorization just like it does now, so nothing downstream needs to change. - -# Detailed Design - -## Design & Implementation Details - -### Reading the identity - -Add a small helper, `TlsIdentityExtractor`, called from `AuthenticationProviderTls.authenticate()` right after the existing null-check on the certificate: - -- `CN` - the current logic (first `CN=` part of the subject). -- `DN` - the full subject string from `((X509Certificate) certs[0]).getSubjectX500Principal().getName()` ([RFC 2253] format), used as-is. -- `SAN:URI` / `SAN:DNS` / `SAN:EMAIL` - read `X509Certificate.getSubjectAlternativeNames()` and take the first SAN of the matching type ([RFC 5280]: URI = 6, DNS = 2, email = 1). If there are several of the same type, the first one in the certificate is used, and a mapping rule can sort out which one you want. (Pulsar already reads SANs this way in `org.apache.pulsar.common.tls.TlsHostnameVerifier`, so this isn't new ground.) - -If the chosen source has no value (for example, you asked for a URI SAN but the certificate doesn't have one), authentication fails with a message that says which source was missing, and a failure metric is recorded with a specific error code. - -### Applying the rules - -A second helper, `PrincipalMappingRules`, parses `tlsAuthPrincipalMappingRules` once at `initialize()` and compiles the regexes into an ordered list. `apply(identity)`returns the first rule that matches. The format and behavior match Kafka's on purpose, so the docs and the operator's existing knowledge carry over. Example: - -``` -tlsCertIdentitySource = SAN:URI -tlsAuthPrincipalMappingRules = RULE:^spiffe://acme\.com/ns/([^/]+)/sa/([^/]+)$/$1__$2/L,DEFAULT -``` - -Here the SPIFFE id `spiffe://acme.com/ns/payments/sa/checkout` becomes the role `payments__checkout`. - -### Where it hooks in - -Only `AuthenticationProviderTls.authenticate(AuthenticationDataSource)` changes. The `AuthenticationProvider` interface, the `AuthenticationDataSource`, and every caller stay the same. Because HTTPS client-cert requests go through this same provider, both transports are covered by the one change. - -**The proxy needs the config too - this is real work, not free.** The Pulsar Proxy builds its `AuthenticationService` from a `ServiceConfiguration` that it gets by calling `PulsarConfigurationLoader.convertFrom(ProxyConfiguration)`. That conversion copies fields **by matching name**. So the two new keys have to be added to *both* `ServiceConfiguration` and `ProxyConfiguration`, with the same names. If they're only in `ServiceConfiguration`, the proxy quietly ignores them and falls back to CN - and then the proxy and the broker can map the same certificate to different roles. A test that goes through the proxy should confirm the config actually takes effect. - -## Public-facing Changes - -### Public API - -None. No new REST endpoint, no client API change, no change to the SPI signatures. - -### Binary protocol - -None. The TLS handshake and the auth command don't change. The only difference is how -the broker reads the certificate it already received. - -### Configuration - -Two new properties in `broker.conf` / `ServiceConfiguration`, also added to `proxy.conf` / `ProxyConfiguration` (see the proxy note above): - -- `tlsCertIdentitySource` - one of `CN` (default), `DN`, `SAN:URI`, `SAN:DNS`, `SAN:EMAIL`. Picks the raw identity from the certificate. -- `tlsAuthPrincipalMappingRules` - comma-separated, ordered list of `DEFAULT` / `RULE://[LU]`. Empty by default, which means "use the value as-is" and keeps today's behavior for `CN`. - -### CLI - -None. - -### Metrics - -Reuse the provider's existing `AuthenticationMetrics` failure path. The provider already has an `ErrorCode` enum (`UNKNOWN`, `INVALID_CERTS`, `INVALID_CN`); add two codes so operators can tell failures apart: - -- `INVALID_CN` (existing) - CN source chosen but no CN present. -- `NO_SAN_OF_TYPE` (new) - SAN source chosen but no SAN of that type present. -- `NO_MAPPING_RULE_MATCHED` (new) - identity read, but no rule (and no `DEFAULT`) matched. - -These show up on the existing per-provider failure counter, labeled by error code. - -# Monitoring - -After rolling out a mapping config, operators should watch the new failure error codes. A jump in `NO_MAPPING_RULE_MATCHED` or `NO_SAN_OF_TYPE` usually means a rule or source is misconfigured and is now rejecting clients that should be let in. When migrating from CN to SAN/DN identity, keep an eye on that per-code counter to confirm the new rules cover all your certificates before you remove any `DEFAULT` fallback. - -# Security Considerations - -This PIP changes how a certificate turns into a role, so it deserves a careful security review: - -- **A bad rule can hand out the wrong role.** A regex that's too loose (like `RULE:.*//admin/`) could map lots of certificates to a powerful role. The docs need to push people toward anchored patterns (`^...$`) and remind them that order matters. Rules are compiled at startup, so a broken rule stops the broker from booting instead of failing silently later. -- **Changing the role string can change who's a super-user or proxy.** `superUserRoles` and `proxyRoles` are matched by exact string. If you change the identity source, a certificate that used to map to a super-user role under CN might not anymore (or, with a sloppy rule, something might newly become one). Re-check those lists whenever you change the source. -- **This never weakens certificate validation.** The identity is only read *after* the TLS layer has already validated and trusted the certificate. Picking a SAN doesn't skip any of that. -- **The empty-CN change only happens if you opt in.** Today an empty CN fails. With a SAN or DN source, those certificates can now authenticate. But that only happens when an operator deliberately sets a non-CN source, so no existing setup changes on its own. -- **No change to multi-tenancy.** The role string is the same one authorization already uses, and tenant isolation is enforced downstream exactly as before. - -If anything here is unclear, we should confirm the rule format and SAN handling on the mailing list before merging. - -# Drawbacks and Pitfalls - -It's worth being honest about the downsides and the easy ways to get this wrong. - -## Drawbacks (cons) - -- **More config to get right.** Two new options on a security-sensitive provider means two more things an operator can misconfigure. The default keeps today's behavior, but anyone who turns this on takes on that responsibility. -- **Regex-on-a-DN is powerful but fiddly.** Mapping rules give a lot of freedom, and freedom means foot-guns. Kafka has lived with the same trade-off, so it's well-understood, but it's still a sharp tool. -- **Still one role per certificate.** If you're a SPIFFE shop that wants groups or multiple roles from one certificate, this doesn't get you there. That's a separate problem. -- **Doesn't touch revocation.** This is only about reading identity. CRL/OCSP and certificate trust are out of scope, so don't expect this to improve them. - -## Pitfalls (easy mistakes) - -- **Forgetting the proxy.** If you add the keys to the broker but not the proxy, the proxy silently falls back to C and you get a split-brain where the proxy and broker disagree on a client's role. Always configure both, and test through the proxy. -- **The DN isn't the string you think it is.** `X500Principal.getName()` returns the [RFC 2253] form, which (a) lists the RDNs in *reverse* order from how people usually write them, (b) escapes special characters, and (c) renders attributes with no short name as `OID=#hexvalue`. A rule written against a "nice-looking" DN may never match the real one. Write your rules against the actual canonical string, and the docs should show realistic examples. -- **"First SAN of that type" when there are several.** A certificate can carry more than one URI (or DNS, or email) SAN. We take the first one in certificate order, so if order matters to you, use a mapping rule to pin down the one you want. -- **Too-broad rules leak privilege.** This is the security point above, repeated here because it's the most common mistake: anchor your patterns and order your rules so a catch-all doesn't accidentally grant `admin`. -- **Rollback can lock people out.** If certificates authenticated *only* because of a SAN/DN source plus rules, they'll stop working after a downgrade to a version without this feature, or after you remove the config. If you need rollback safety, keep a working CN in those certificates as a fallback. - -# Backward & Forward Compatibility - -## Upgrade - -Nothing to do. If you leave the new options unset, `AuthenticationProviderTls` behaves exactly like before (first CN, fail on empty CN). You opt in by setting `tlsCertIdentitySource` / `tlsAuthPrincipalMappingRules`. - -## Downgrade / Rollback - -Remove the two properties, or downgrade the broker/proxy, and behavior goes back to CN-only. Be careful: any certificate that authenticated only because of a SAN/DN source plus rules will stop authenticating after rollback. If you need to be able to roll back, keep a usable CN in those certificates (see *Pitfalls*). - -## Pulsar Geo-Replication Upgrade & Downgrade/Rollback Considerations - -This is a per-broker (and per-proxy) setting. In a geo-replicated cluster, each broker reads roles from the certificates presented to *it* - brokers replicate data, not TLS sessions, so there's no cross-cluster coupling. Just make sure brokers (and proxies) that might serve the same clients use the same mapping config, so a client doesn't get a different role depending on which broker it happens to hit. - -# Alternatives - -- **Add a full `PrincipalBuilder` SPI like Kafka's [KIP-189].** More flexible, but Pulsar already has `AuthenticationProvider` as its extension point. Adding a second SPI for the same job would be redundant. Improving the built-in provider covers the common cases with no code from operators. -- **Ship a separate `AuthenticationProviderTlsSan` provider.** Rejected. It forks the TLS provider and forces operators to choose between providers instead of just configuring one identity policy. One configurable provider is simpler. -- **Hardcode "try SAN first, then CN."** Rejected. It's too opinionated - different organizations want different sources - and silently changing the default would be a breaking change. - -# General Notes - -The change is deliberately small and contained: one helper to read the identity, one helper to compile the rules, two new config keys (mirrored in `ServiceConfiguration` and `ProxyConfiguration`), and two new failure error codes, all behind `AuthenticationProviderTls`. Reusing Kafka's rule format is a conscious choice to make life easier for people who run both systems. - -# Links - -* Apache Kafka KIP-371 - *Add a configuration to build custom SSL principal name* (`ssl.principal.mapping.rules`): https://cwiki.apache.org/confluence/display/KAFKA/KIP-371%3A+Add+a+configuration+to+build+custom+SSL+principal+name -* Apache Kafka KIP-189 - *Improve principal builder interface and add support for SASL* (`KafkaPrincipalBuilder` SPI): https://cwiki.apache.org/confluence/display/KAFKA/KIP-189%3A+Improve+principal+builder+interface+and+add+support+for+SASL -* RFC 9525 - *Service Identity in TLS* (current standard; obsoletes RFC 6125 - use SAN, not CN, for identity): https://www.rfc-editor.org/rfc/rfc9525 -* RFC 6125 - *Representation and Verification of Domain-Based Application Service Identity ...* (obsoleted by RFC 9525; kept here for historical reference): https://www.rfc-editor.org/rfc/rfc6125 -* RFC 2253 - *LDAPv3 UTF-8 String Representation of Distinguished Names* (the DN string format that `X500Principal.getName()` is defined to return; obsoleted as a general standard by RFC 4514, but still the format the Java API emits): https://www.rfc-editor.org/rfc/rfc2253 -* RFC 4514 - *LDAP: String Representation of Distinguished Names* (current standard; obsoletes RFC 2253): https://www.rfc-editor.org/rfc/rfc4514 -* RFC 5280 - *Internet X.509 PKI Certificate and CRL Profile*, §4.2.1.6 Subject Alternative Name (SAN types): https://www.rfc-editor.org/rfc/rfc5280#section-4.2.1.6 -* SPIFFE ID specification (URI-SAN workload identity): https://github.com/spiffe/spiffe/blob/main/standards/SPIFFE-ID.md -* Mailing List discussion thread: TBD -* Mailing List voting thread: TBD From 8423bc86a8e833b5024bd0d3ab6a5c2f08fc0ff9 Mon Sep 17 00:00:00 2001 From: PavelZeger Date: Fri, 17 Jul 2026 12:58:33 +0300 Subject: [PATCH 3/3] [improve][broker] PIP-490: Zero-downtime token signing-key rotation for the built-in JWT provider --- pip/pip-490.md | 183 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 183 insertions(+) create mode 100644 pip/pip-490.md diff --git a/pip/pip-490.md b/pip/pip-490.md new file mode 100644 index 0000000000000..261d481b7b875 --- /dev/null +++ b/pip/pip-490.md @@ -0,0 +1,183 @@ +# PIP-490: Zero-downtime token signing-key rotation for the built-in JWT provider + +# Background knowledge + +`AuthenticationProviderToken` (in `pulsar-broker-common`) authenticates clients that present a signed JWT. Today it is configured with one validation key: either a symmetric secret (`tokenSecretKey`) or an asymmetric public key (`tokenPublicKey`, whose algorithm is set by `tokenPublicAlg`). It checks every incoming JWT against that one key. The token's `sub` claim (or a configured `tokenAuthClaim`) becomes the client's role. The binary listener, the HTTP/HTTPS and WebSocket endpoints, and the Pulsar Proxy all authenticate tokens through this provider. It uses the jjwt library (`io.jsonwebtoken`), version 0.13. + +A JWT can carry a `kid` ("key id") in its header that names the key used to sign it. Systems that hold several signing keys at once use the `kid` to pick the right key to verify with. This is what makes smooth rotation possible: you add a new key, start signing new tokens with it, and keep accepting both the old and the new key until the old tokens expire. A JWKS (JSON Web Key Set) is a JSON document that holds a list of keys; each entry states its own `kid`, its key type (`kty`), and, optionally, its algorithm (`alg`). That makes a JWKS a natural way to describe "a set of validation keys". + +Pulsar already resolves keys by `kid`, but only in `AuthenticationProviderOpenID`. That provider is built on a different library (`com.auth0.jwt`) and expects to fetch keys from a remote `jwks_uri`. So this proposal borrows the *idea*, not the code. The two things it needs are already in the jjwt 0.13 library that the token provider depends on: `keyLocator`, which lets the parser pick a key per token from the header, and a local JWKS parser (`io.jsonwebtoken.security.Jwks` / `JwkSet`). For comparison, Kafka's OAUTHBEARER path also uses a JWKS and treats online key rotation as normal. + +# Motivation + +The provider loads one key when it starts, and never looks at the token's `kid` (`AuthenticationProviderToken.java`): + +```java +private Key validationKey; +this.validationKey = getValidationKey(config); +this.parser = Jwts.parser() + .setAllowedClockSkewSeconds(allowedSkew) + .setSigningKey(this.validationKey).build(); +``` + +It also has no way to reload the key: the key stays fixed for the life of the process. Because of this, you cannot rotate a signing key without downtime. There is no period when the old key and the new key are both accepted. To rotate, an operator has to re-issue every existing token with the new key and switch the broker's config at the same moment (in practice, a coordinated restart). During the switch, tokens signed with the wrong" key are rejected. + +This makes two normal security tasks painful: + +- **Responding to a possible key compromise.** The safe move is to add a replacement key and retire the old one gradually. Today that means an outage plus re-issuing every token at once. +- **Routine rotation.** Rotating keys on a schedule is a basic control (see NIST SP 800-57 on cryptoperiods). Today it is expensive enough that operators tend to skip it and keep long-lived keys. + +Graceful rotation is also a building block for any future work on broker-issued, revocable delegation tokens. The fix needs no new cryptography and no new library. It only changes how the provider validates tokens: it adds `kid`-based selection, support for several keys, and reloading without a restart. It does this carefully, so that supporting several keys does not open the "algorithm confusion" weakness that a naive "just try every key" approach would (explained under Security Considerations). + +# Goals + +## In Scope + +- Let `AuthenticationProviderToken` hold several validation keys at once. Each key is tied to a `kid` and to one algorithm. +- Choose the key using the token's `kid`. If a token has no `kid` (older tokens), fall back to trying the configured keys, each with its own algorithm. +- Reload the set of keys without restarting the broker. +- Keep today's behavior exactly the same when only the existing single-key settings are used. +- Work everywhere the provider is used: the binary protocol, HTTP/HTTPS, WebSocket, and the Pulsar Proxy. + +## Out of Scope + +- Broker-issued / delegation tokens and revocation. That is possible future work, and this proposal is a step toward it. +- Changing the JWT library, or adding algorithms that jjwt does not already support. +- Generating or distributing keys, or fetching keys from a remote `jwks_uri`. That is what the OIDC provider is for. Keys stay local and operator-supplied. +- Claim handling (`exp`, `tokenAuthClaim`, `tokenAudienceClaim`). Only the choice of *which key* verifies the signature changes. + +# High Level Design + +Replace the single `validationKey` with a read-only holder that maps each `kid` to a key and its algorithm. The provider looks up the key for each token, and can replace the whole holder in one step when the keys are reloaded. + +The main way to supply several keys is a local JWKS document (a file, or inline text). A JWKS is a good fit because each key already states its `kid` and key type, and can also state an exact `alg`. That is exactly the information needed to tie each key to the one algorithm it is allowed to verify - which is the heart of the security design. The existing `tokenSecretKey` / `tokenPublicKey` settings still work; such a key is treated as a single key with no `kid`. + +How a token is resolved: + +- If the token has a `kid` that matches a configured key, verify it against that key only, using that key's algorithm. +- If the token has no `kid` (or its `kid` matches nothing, and fallback is on), try each configured key in turn, each with its own algorithm. The first key that verifies wins. +- If no key verifies, the token is rejected, just as today (`INVALID_TOKEN`), with an error code that shows whether the `kid` was the problem. + +The holder is kept in a `volatile` field and replaced as a whole on reload, so a request never sees a half-updated set. The jjwt parser is still built once; its `keyLocator` reads the current holder for each token. + +The rotation steps are all additive, and no valid token is ever rejected in flight: + +1. Add key **B** to the JWKS. Both A and B are now accepted. +2. Start issuing new tokens signed with **B**, stamped with `kid=B`. +3. Wait until all tokens signed with **A** have expired. +4. Remove key **A**. + +Stamping a `kid` on new tokens already works today: `AuthTokenUtils.createToken` passes through a headers map, and `pulsar tokens create --headers kid=B` already sets it. This proposal only changes the validation side, and adds an optional `--key-id` flag for convenience. + +# Detailed Design + +## Design & Implementation Details + +### Choosing the key + +Replace the single `Key validationKey` with a read-only holder - call it `TokenValidationKeys` - that maps each `kid` to its key and algorithm, plus the ordered list of keys that older, `kid`-less tokens may be tried against. Keep the holder in a `volatile` field so a reload can replace it in one step. Wire it into the parser once, using jjwt's key locator instead of `setSigningKey`: + +- **When a token has a `kid` (one lookup).** Build the parser with `Jwts.parser().keyLocator(locator)`. For each token, `locator.locate(header)` reads `header.getKeyId()` and `header.getAlgorithm()` from the (still unverified) `JwsHeader` - both methods exist in jjwt 0.13 - and returns the key for that `kid`, or `null` (in which case jjwt reports the usual verification failure). +- **The algorithm is checked in the locator, not left to jjwt's defaults.** `locate` returns the key only if the token's `alg` matches the key's algorithm; otherwise it returns `null`. Because the keys come from a JWKS, each key already has its algorithm attached, so this match is checked for you. This is what stops the RS256-to-HS256 confusion attack (see Security Considerations). +- **When a token has no `kid` (older tokens).** jjwt's locator picks one key per parse, so it cannot "try them all" on its own. The fallback is therefore a short loop in the provider: try to verify against each candidate key in turn, each with its own algorithm (an HMAC key only runs HMAC, an RSA key only runs RSA), and stop at the first that works. This loop is the cost referred to in the DoS note below. Setting `tokenRequireKid=true`removes the loop entirely (strict matching only). + +Claim handling (`exp`, `tokenAuthClaim`, `tokenAudienceClaim`) does not change. + +### Loading and reloading + +`getValidationKeys(config)` builds the holder from two possible sources, which can be combined: the existing `tokenSecretKey` / `tokenPublicKey` (one key, no `kid`, with the algorithm from `tokenPublicAlg`, or HMAC for a secret), and the new `tokenValidationKeysJwks` source, parsed with jjwt's `Jwks` / `JwkSet`. A scheduled task(`tokenValidationKeysRefreshSeconds`; `0` means load once, as today) re-reads the source, builds a new read-only holder, and swaps it in. It only reads local files; it never makes a network call. If the source fails to parse, the provider keeps the previous holder and logs and counts the failure, so a bad edit never leaves the broker with no keys. + +To stay exactly compatible: if only the old setting is present and `tokenValidationKeysJwks` is empty, the provider uses the original single-key path (the parser is built with that one key), so behavior is identical to today. + +The reload timer is a single-thread `ScheduledExecutorService`. The provider creates it only when `tokenValidationKeysRefreshSeconds > 0`, and shuts it down in `close()` (which does nothing today). + +### Creating tokens + +Nothing new is needed to set a `kid`: `createToken` already passes headers to `JwtBuilder.setHeaderParams`, and the CLI already accepts `--headers key=value`. This proposal adds an optional `--key-id` flag to `pulsar tokens create` for convenience; it validates the value (rejecting an empty or duplicate `kid`). + +## Public-facing Changes + +### Public API + +None. No REST change, no client-API change, and no change to the `AuthenticationProvider` interface. + +### Binary protocol + +None. The `kid` is a standard JWT header and does not affect the Pulsar protocol. + +### Configuration + +New settings on `ServiceConfiguration` (`broker.conf`). They must be added to `ProxyConfiguration` (`proxy.conf`) too, with the same names. The proxy builds its `AuthenticationService` from a `ServiceConfiguration` created by `PulsarConfigurationLoader.convertFrom(ProxyConfiguration)`, which copies fields by matching their names. If a setting exists only on `ServiceConfiguration`, the proxy would validate against a different key than the broker. An integration test should go through the proxy. + +- `tokenValidationKeysJwks` - a local JWKS source (`file://`, `data:`, or inline base64, using the same loader as the single-key settings). It holds one or more keys, each with a `kid` and key type, and optionally an exact `alg`. Empty by default. +- `tokenValidationKeysRefreshSeconds` - how often to reload the source. `0` (the default) means load once. +- `tokenRequireKid` - `false` by default. When `true`, a token whose `kid` does not match a configured key is rejected instead of going through the fallback. + +The existing `tokenSecretKey` / `tokenPublicKey` / `tokenPublicAlg` still work and combine with the set (as one unnamed key). Like the other token settings, the three new ones respect `tokenSettingPrefix`, so a prefixed or chained provider picks them up. + +### CLI + +An optional `--key-id` flag on `pulsar tokens create` sets the JWT `kid` header. It is a validated shortcut for `--headers kid=...`. + +### Metrics + +Reuse `AuthenticationMetricsToken`. Add one value to the `ErrorCode` enum (today `INVALID_AUTH_DATA`, `INVALID_TOKEN`, `INVALID_AUDIENCES`): + +- `INVALID_TOKEN_KID` (new) - the token named a `kid` that matches no configured key (either with `tokenRequireKid=true`, or after the fallback also failed). + +Also add a reload-health signal - a counter of reload failures and a gauge of how many keys are configured - using the provider's existing OpenTelemetry naming. This lets an operator see a broken JWKS without reading logs. + +# Monitoring + +During a rotation, watch the per-error-code failure counter and the reload health: + +- If `INVALID_TOKEN_KID` is above zero after you add key B but before clients have the new tokens, the set is missing a key that clients are already sending. Add it before going further. +- Before step 4 (removing key A), confirm that successful logins using key A have dropped to zero - that is, all A-signed tokens have expired. +- If the reload-failure count rises, the JWKS on disk no longer parses. The broker keeps serving the last good set, but your change has not taken effect. + +# Security Considerations + +- **Algorithm confusion is the main risk this design has to close, and does.** The classic JWT attack forges a token with `alg=HS256`, using the bytes of a public RSA key as the HMAC secret. A single-key provider is mostly safe because it has one key of one type. A multi-key provider that "tries every key" is not safe, unless each key is tied to one algorithm. Here, a key is only ever used with its own algorithm, and a token whose `alg` does not match is rejected before any signature check. Tying the key type alone is already enough to stop the RSA-to-HMAC case: a public RSA key is a `PublicKey` and can never be used as an HMAC `SecretKey`. An exact `alg` on the key narrows it further. Using a JWKS makes this automatic, because each key states its type (and optionally its `alg`), so the binding is checked by the code rather than remembered by the operator. Mixing symmetric and asymmetric keys is therefore safe. The docs should still advise a separate`kid` for each algorithm, and never reusing a `kid` across key types. +- **The `kid` comes from the attacker, and that is fine.** It only picks which key to try. The signature still has to verify against that key, using that key's algorithm. The algorithm binding and the operator-chosen key set stop a `kid` from steering the check toward a weaker key. +- **Fallback versus strict matching.** The fallback trades a little strictness for easier migration. Once every token has a `kid`, set `tokenRequireKid=true` to require an exact match (recommended after migration). +- **Cost and denial of service.** The fallback does up to N signature checks per unverified token, and public-key checks are not cheap. A rotation only needs two or three keys, so keep the set small; the docs should say so, and the code should log a warning if the set grows large. `tokenRequireKid` limits this to one check per token. +- **No new trust, and no network calls.** Every key is supplied locally by the operator, as today. Unlike OIDC, the provider never fetches a `jwks_uri`. +- **Safe reloads.** A failed reload keeps the last good set, so the broker never drops to zero keys. The swap is atomic, so no request is checked against a half-built set. +- **Multi-tenancy is unchanged.** Only signature checking changes. + +# Backward & Forward Compatibility + +## Upgrade + +Nothing to do. If only `tokenSecretKey` / `tokenPublicKey` are set and tokens have no `kid`, the provider uses the original single-key path and behaves exactly as before. Multiple keys are opt-in through `tokenValidationKeysJwks`. + +## Downgrade / Rollback + +Remove the new settings, or downgrade the broker and proxy. The provider goes back to single-key validation. One caution: before rolling back, make sure every live token is signed with the one key the old broker keeps. Tokens signed with any other key in the set will stop validating. The safe order is the reverse of rotation: re-issue tokens with the key you are keeping, wait for the others to expire, then downgrade. + +## Pulsar Geo-Replication Upgrade & Downgrade/Rollback Considerations + +Keys are per-broker and per-proxy configuration. Brokers do not share them, and geo-replication copies data, not login sessions. When rotating a key used across clusters, add the new key to every cluster's set before any cluster starts using it, and remove the old key everywhere only after all old tokens have expired everywhere. Keep brokers and proxies that may serve the same clients on the same set of keys. + +# Alternatives + +- **A plain `kid:file://...` list instead of a JWKS.** Rejected as the main format. A bare key file does not state its algorithm, so you would have to declare the `kid`-to-algorithm mapping separately, and any mistake there reopens the algorithm-confusion hole. A JWKS keeps the `kid`, key type, and `alg` together, is parsed by the library already in use, and lets the code check the binding. (The `data:` / `file:` loader is still reused, to load the JWKS document.) +- **Reuse the OIDC provider for local tokens.** Rejected. It expects an HTTP issuer with a remote `jwks_uri`, and uses a different library. Local token auth is deliberately free of network calls; only the `kid`-selection idea is borrowed. +- **Chain two token providers with `AuthenticationProviderList`.** Possible today, but rejected as the path forward: it doubles the work, makes metrics and failure attribution harder, cannot select by `kid`, and cannot hot-reload. +- **A remote `jwks_uri` for the local provider.** Out of scope - that is essentially the OIDC provider. + +# General Notes + +The change is small and contained: a read-only map of `kid` to key and algorithm, a `keyLocator` that ties each key to one algorithm, a timed atomic reload, three new settings mirrored into `ProxyConfiguration`, and one new error code. Both building blocks - `keyLocator` and the local `Jwks` parser - are already in the jjwt 0.13 library. The benefit is concrete: graceful key rotation and a real response to key compromise, with no downtime, and without opening the algorithm-confusion weakness that a naive multi-key design would. + +How this relates to nearby proposals: **PIP-488** (authentication audit events) works well with this: an `INVALID_TOKEN_KID` failure would show up as an `AuthenticationEvent`, giving a per-login view of a rotation in progress. **PIP-489** (FIPS mode) is unrelated here; its dual-verify HMAC change for the SASL role-token signer is a similar "accept two keys during migration" idea, but for a different key. + +# Links + +* jjwt 0.13 - `JwtParserBuilder.keyLocator(Locator)` and `io.jsonwebtoken.security.Jwks` / `JwkSet`. +* RFC 7517 - JSON Web Key (JWK) and JWK Set: https://www.rfc-editor.org/rfc/rfc7517 +* RFC 7515 §4.1.4 - the JWS `kid` header: https://www.rfc-editor.org/rfc/rfc7515#section-4.1.4 +* NIST SP 800-57 Part 1 - key management / cryptoperiods: https://csrc.nist.gov/pubs/sp/800/57/pt1/r5/final +* Related: a future broker-issued delegation-token PIP (this is a prerequisite). +* Mailing List discussion thread: TBD +* Mailing List voting thread: TBD