diff --git a/toc/rfc/rfc-draft-diego-c2c-mtls.md b/toc/rfc/rfc-draft-diego-c2c-mtls.md new file mode 100644 index 000000000..8bf50f186 --- /dev/null +++ b/toc/rfc/rfc-draft-diego-c2c-mtls.md @@ -0,0 +1,271 @@ +# Meta +[meta]: #meta +- Name: Full mTLS for Diego Container-to-Container Traffic +- Start Date: 2026-02-13 +- Author(s): @rkoster +- Status: Draft +- RFC Pull Request: [community#1437](https://github.com/cloudfoundry/community/pull/1437) + + +## Summary + +This RFC proposes implementing full mutual TLS (mTLS) for container-to-container (C2C) traffic in Diego, enabling applications to both authenticate themselves and verify the identity of connecting applications. + +The approach introduces a **new HTTP-based listener on port 62443** that runs alongside the existing TCP-based C2C port (61443), providing a dual opt-in model for both operators and application authors. This enables zero-trust networking where all app-to-app traffic can be authenticated and authorization decisions can be made based on app/space/org identity. + + +## Problem + +Cloud Foundry's Diego runtime currently runs Envoy as a per-container sidecar for handling **inbound traffic only**. While this provides TLS termination and route integrity, several limitations prevent true zero-trust app-to-app communication: + +### No Client-Side (Egress) Proxy + +Envoy is not configured as an egress proxy. Applications cannot automatically inject their instance identity certificate when making outbound connections. The current workaround requires applications to manually use `CF_INSTANCE_CERT` and `CF_INSTANCE_KEY` environment variables to present their identity. + +### No Identity Extraction on Server Side + +Even when mTLS is enabled, the receiving application has no way to know who connected. Envoy does not forward client certificate information via headers (no XFCC header configuration). + +### C2C Port (61443) is TCP-Based + +The C2C listener (port 61443) is a TCP proxy, not an HTTP proxy. This means: +- No HTTP connection manager +- No XFCC header support +- No mTLS client certificate validation +- No HTTP/2 support + +### Existing mTLS Only Affects Route Integrity + +The existing BOSH property `containers.proxy.require_and_verify_client_certificates` only affects route integrity ports (Gorouter → App), not C2C traffic. + +### Current Architecture + +```mermaid +flowchart LR + subgraph Container + Envoy["Envoy
(port 61001)"] + App["App
(port 8080)"] + Envoy -->|plain TCP| App + end + Gorouter -->|TLS| Envoy + + Note["Envoy only handles INBOUND traffic
(DownstreamTlsContext)"] +``` + +### Current Certificate Setup + +Diego generates two separate certificates per container in the [credmanager](https://github.com/cloudfoundry/executor/blob/main/depot/containerstore/credmanager.go): + +| Certificate | SAN Contents | Purpose | mTLS Support | HTTP/2 | +|------------|--------------|---------|--------------|--------| +| Instance Identity | Container IP address | Route integrity (Gorouter → App) | Yes (optional) | Yes | +| C2C Certificate | Internal route hostnames (e.g., `app.apps.internal`) | Container-to-container TLS | No | No | + +Both certificates include app identity in the certificate's `Subject.OrganizationalUnit` field (separate from the SAN). These claims are set by Cloud Controller and passed through the [BBS CertificateProperties](https://github.com/cloudfoundry/bbs/blob/main/models/certificate_properties.pb.go) model: + +- `app:` +- `space:` +- `organization:` + + +## Proposal + +### Design Principles + +1. **Backwards compatibility**: Keep port 61443 unchanged for existing applications +2. **Dual opt-in**: Both operator and application author must opt-in +3. **New port for new behavior**: Port 62443 for HTTP-based mTLS with XFCC header support + +### Port Allocation + +| Port | Type | Purpose | mTLS | XFCC | Status | +|------|------|---------|------|------|--------| +| 61001+ | HTTP | Route integrity | Optional | No | Existing | +| 61443 | TCP | C2C TLS (TCP-based) | No | No | Existing (unchanged) | +| **62443** | **HTTP** | **C2C mTLS with identity** | **Required** | **Yes** | **New** | +| 61445 | HTTP | Egress proxy (client-side) | N/A | N/A | New | + +### Implementation Phases + +The implementation is divided into three phases to allow gradual adoption and testing. + +--- + +### Phase 1: Server-Side - New HTTP-based C2C mTLS Port (62443) + +Add a new Envoy listener on port 62443 that provides HTTP-based C2C with mTLS and XFCC header forwarding. + +```mermaid +flowchart TB + subgraph ContainerB["Container B (Server)"] + subgraph Listeners + L61443["Port 61443
(TCP)"] + L62443["Port 62443
(HTTP, new)"] + end + AppB["App B
(port 8080)"] + + L61443 -->|"TLS termination only
No client cert validation
No XFCC header"| AppB + L62443 -->|"mTLS required
XFCC header forwarded"| AppB + end + + ClientA["Container A
(Client)"] -->|"Option 1: TLS (TCP proxy)"| L61443 + ClientA -->|"Option 2: mTLS with identity"| L62443 +``` + +#### Changes Required + +**New Port Constant**: Add `C2CMTLSPort = 62443` alongside the existing `C2CTLSPort = 61443`. + +**New HTTP-based Listener**: Create a new Envoy listener with: +- HTTP connection manager (not TCP proxy) +- `DownstreamTlsContext` with `RequireClientCertificate: true` +- Validation context trusting the instance identity CA +- XFCC header set from the verified client certificate, with any existing headers removed to prevent spoofing +- Subject, URI, and DNS fields included in the XFCC header + +**BOSH Properties**: + +```yaml +containers.proxy.enable_c2c_mtls_listener: + description: "Enable the HTTP-based C2C mTLS listener on port 62443. This listener + requires client certificates and forwards caller identity via the X-Forwarded-Client-Cert header." + default: false +``` + +**Port Reservation**: Add validation to prevent applications from using port 62443, similar to the existing reservation for port 61443. + +#### Result + +When an operator enables `containers.proxy.enable_c2c_mtls_listener`: +- Port 62443 becomes available on all containers +- Clients connecting to 62443 MUST present a valid instance identity certificate +- The receiving app gets an `X-Forwarded-Client-Cert` header with caller identity: + +``` +X-Forwarded-Client-Cert: Hash=abc123;Subject="CN=instance-guid,OU=app:client-app-guid,OU=space:space-guid" +``` + +--- + +### Phase 2: Client-Side Egress Proxy (Port 61445) + +Enable Envoy to act as an HTTP proxy for outbound connections, automatically injecting the instance identity certificate. + +```mermaid +flowchart LR + subgraph ContainerA["Container A"] + AppA["App A"] + EgressProxy["Envoy Egress
(port 61445)"] + AppA -->|HTTP| EgressProxy + end + + subgraph ContainerB["Container B"] + IngressProxy["Envoy Ingress
(port 62443)"] + end + + EgressProxy -->|"mTLS
(auto cert injection)"| IngressProxy + + Note["Certificate contains:
OU=app:client-app-guid
OU=space:space-guid"] +``` + +#### Changes Required + +**Egress Proxy Port**: Reserve port 61445 for the egress proxy. + +**Egress Listener**: Create a new Envoy listener that acts as an HTTP CONNECT proxy, bound to `127.0.0.1:61445` (localhost only). The proxy accepts all outbound traffic but only injects the instance identity certificate for connections to C2C mTLS backends. + +**UpstreamTlsContext**: Configure the egress proxy to attach the instance identity certificate only when connecting to C2C mTLS backends, identified by destination port (62443) and hostname pattern matching the configured `internal_domains`. + +**Environment Variable**: Inject a CF-specific environment variable for applications to discover the egress proxy: + +``` +CF_INSTANCE_MTLS_PROXY=http://127.0.0.1:61445 +``` + +**Enforcing Proxy Usage**: Operators can make the egress proxy mandatory for all applications by setting `HTTP_PROXY` and `HTTPS_PROXY` to `http://127.0.0.1:61445` using [running environment variable groups](https://docs.cloudfoundry.org/devguide/deploy-apps/environment-variable.html#evgroups). This ensures all HTTP/HTTPS traffic from applications is routed through the egress proxy, with instance identity certificates injected only for C2C mTLS backends. + +**BOSH Properties**: + +```yaml +containers.proxy.enable_egress_proxy: + description: "Enable Envoy egress proxy for outbound mTLS on port 61445" + default: false + +containers.proxy.internal_domains: + description: "Domains for C2C mTLS backends where the instance identity certificate should be injected." + example: ["apps.internal.", "my.apps.internal."] + default: [] +``` + +This follows the same naming and format as the [`internal_domains` property in `bosh-dns-adapter`](https://github.com/cloudfoundry/cf-networking-release/blob/develop/jobs/bosh-dns-adapter/spec). + +#### Considerations + +- **SNI Handling**: Envoy needs to extract the target hostname for proper TLS handshake +- **NO_PROXY**: Applications should configure `NO_PROXY` for traffic that should not go through the proxy +- **Non-HTTP Traffic**: The HTTP CONNECT-based egress proxy only supports HTTP/HTTPS traffic. Support for TCP-based protocols could be addressed in a follow-up RFC. +- **Performance**: Additional hop adds latency +- **Memory**: Additional memory allocation may be needed for the egress proxy + +--- + +### Phase 3: Full Integration + +With both phases implemented, the complete mTLS flow enables zero-trust app-to-app communication: + +```mermaid +sequenceDiagram + participant AppA as App A + participant EgressA as Envoy Egress
(Container A:61445) + participant IngressB as Envoy Ingress
(Container B:62443) + participant AppB as App B + + AppA->>EgressA: HTTP request + Note over EgressA: Attaches instance cert
OU=app:aaa-guid
OU=space:xxx + EgressA->>IngressB: mTLS connection + Note over IngressB: Validates client cert
against instance identity CA + IngressB->>AppB: HTTP request +
X-Forwarded-Client-Cert header + Note over AppB: Reads XFCC header
to authorize request + AppB-->>IngressB: HTTP response + IngressB-->>EgressA: Response + EgressA-->>AppA: Response +``` + +#### Opt-in Model + +| Actor | How to Opt-in | +|-------|---------------| +| **Operator** | Set `containers.proxy.enable_c2c_mtls_listener: true` and `containers.proxy.enable_egress_proxy: true` | +| **Server App** | Listen for connections and read `X-Forwarded-Client-Cert` header | +| **Client App** | Connect to port 62443 (directly or via egress proxy at 61445) | + +#### App-Level Authorization + +Applications can parse the XFCC header to make authorization decisions based on the caller's app GUID, space GUID, or organization GUID embedded in the certificate's OrganizationalUnit field. + +--- + +### Benefits + +- Backwards compatible: port 61443 stays as-is, existing apps keep working +- Opt-in for both operators (BOSH properties) and app authors (connect to 62443) +- Apps can migrate gradually, no big-bang required +- Server apps can verify who's calling and authorize based on app/space/org +- No more manual certificate handling for C2C calls + +--- + +## Key Repositories + +The following repositories require modifications to implement this proposal: + +| Repository | Purpose | +|------------|---------| +| [cloudfoundry/executor](https://github.com/cloudfoundry/executor) | Envoy config generation, environment variable injection | +| [cloudfoundry/diego-release](https://github.com/cloudfoundry/diego-release) | BOSH property definitions | + +## References + +- [Envoy XFCC Header Documentation](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/headers#x-forwarded-client-cert) +- [Envoy Dynamic Forward Proxy](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/dynamic_forward_proxy_filter) +- [Diego Envoy Proxy Configuration](https://github.com/cloudfoundry/diego-release/blob/develop/docs/060-envoy-proxy-configuration.md)