TLS for everything

[jmprice]

What is this issue about?

There has been a lot of excellent progress in securing all CF traffic with TLS and as far as I can tell there are only a few things that are still unencrypted.
Is there a timeline or any plans for these last few things?

1. routing-api - still using both TLS and non-TLS in the cf-deployment. The http endpoint is what is registered in the router. Is there a reason for still enabling both?
2. metrics-discovery-registrar-windows - not using nats-tls hostname, falling back to 4222. We have pull request in for this one already (Fixing nats-tls config for Windows to use hostname instead of IPs. metrics-discovery-release#6)
3. route_registrar - not using nats-tls
4. gorouter - not using nats-tls

What version of cf-deployment are you using?

[cf-deployment v13.19.0]

Tag your pair, your PM, and/or team!

@amhuber

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/174846740

The labels on this github issue will be updated when the story is started.

[46bit]

As well as the things mentioned above, my team believes that Silk (apps.internal) VXLAN traffic between cells is unencrypted.

[heycait]

Hi all, metric egress team here. We own the metrics-discover-release. The issue with windows was unintentional on our part. We are aware of it and will prioritize it soon see cloudfoundry/metrics-discovery-release#6. It should make it into CF-D fairly quickly after we cut a release.

[davewalter]

I reached out to the Networking folks in Slack for assistance with the connections related to routing and nats.

[mcwumbly]

@davewalter thanks for the ping. Per this discussion on cf-dev, I suggest we keep this issue open as the "canonical home" for tracking this issue going forward.

That said, I suggest we scope this issue does to TLS for all platform components can exclude this bit:

my team believes that Silk (apps.internal) VXLAN traffic between cells is unencrypted.

while it is true that the platform does not encrypt the VXLAN traffic between cells today, my take is that this is a slightly different concern, and those who are deploying apps to the platform can encrypt the traffic between their apps. I don't dispute it'd be nice if this were a built-in feature of the platform, but I do think this is a reasonable place to divide this issue.

Perhaps we can rename this one and @46bit can open another one for that feature?

For the rest of the items above, can they all technically be addressed by changes to cf-deployment? Or will they need changes in any of the respective BOSH releases, like https://github.com/cloudfoundry/routing-release? (I can probably answer some of that myself, but haven't yet taken the time to dig in).

One other note: there's this open issue on nats-release right now, which I believe may in part be caused by the fact that we do have TLS turned on between the Diego Cell route-emitters and NATS: cloudfoundry/nats-release#25

So, we might discover that flipping all these things on has some side effects that are only seen with large deployments, and which do not rear their heads in CI.

[amhuber]

For the different items reported originally:

1. The routing-api can be configured to TLS only but cf-deployment is currently configuring it to listen on both HTTP and HTTPS endpoints (https://github.com/cloudfoundry/cf-deployment/blob/master/cf-deployment.yml#L1039). We haven't been able to find anything that still connects directly to the HTTP endpoint, and having both endpoints enabled appears to force the routing-api to register it's HTTP interface with gorouter vs. the HTTPS one.

2. We've already submitted a pull request for this as listed.

3. The route_registrar already supports TLS connections to nats but it's not enabled in cf-deployment (https://github.com/cloudfoundry/routing-release/blob/develop/jobs/route_registrar/templates/registrar_settings.json.erb#L106).

4. I haven't found code in gorouter to support TLS connections to nats so I think this one actually requires some code changes to fix.

For what it's worth, regarding your mention of the nats issue, I think having nats currently split between TLS and non-TLS makes issues like that more likely. Right now cf-deployment is using 2 different nats daemons per VM, replicating all messages between the TLS and non-TLS endpoints, with clients split between both. It would be far more stable (and less resource intensive) to complete the move to nats-tls only and get rid of the duplicate daemons listening on the non-TLS enpoints and the requisite replication between them.

[ameowlia]

Hi all,

I am an engineer on the CF for VMs networking team.

The work for "tls everywhere" is currently not a priority for our team since there is a workaround (ipsec). However, we would be happy to support it if any community members wanted to PR it in.

To the best of my knowledge, the following paths related to routing, cf-networking, and silk are not encrypted:

• NATS -> gorouter (for route registration)
• Routing API -> gorouter (routing api sets its own route on gorouter)
• NATS -> service discovery controller (internal route registration)
• app -> app (container to container networking)

I don't know what work there is (if any) for other components.
Additionally, the following routes to components are not encrypted by default:

• proxy
• internal blobstore
• routing api

There maybe more unencrypted routes to components. I just checked on a pretty minimal test env. But you can check by looking at the routes on the router and seeing which are tls: false.

[amhuber]

Setting aside the technical details for a moment, I had hoped at least there was consensus that it's not OK to have unencrypted sensitive traffic on any network. IPSec is a workaround, but it's painful and expensive because we end up double encrypting most traffic just to encrypt the few stragglers that aren't using TLS yet. This was previously communicated to the community as a high priority from a security perspective now so I'm hoping we can get back to a place where teams are working on closing the few remaining gaps that exist. It seems odd for so many people to have put effort into moving huge parts of the platform to mTLS just to leave the whole platform vulnerable to simple attacks. For example, right now anyone with network access to a CF foundation can trivially own nats as the password is in clear text on the network and then cause whatever routing chaos they want.

[jmprice]

Additionally for those of us who are also running Windows cells as part of our foundations, IPsec encryption is not supported for Windows containers (microsoft/hcsshim#244) so we are stuck on Windows 2012R2 which is not only an aging OS but is no longer being actively maintained or supported as part of CF. At some point, and I fear sooner than later, we are going to reach an impasse where we must upgrade to Win2019 and can no longer use IPsec.

[voelzmo]

Regarding the topic of using IPSec as a workaround, I'd like to point out that this might be true for products and commercial distributions, but I'm not aware of an open-source solution solving this issue. So from a vendor perspective, it might be true that there is a workaround available, if we're talking about open-source, this does not seem to be the case.

@ameowlia: can you help us understand how we can get this work prioritized in the open source team? It might be helpful to get an understanding what the conflicting priorities are and when it would be possible to prioritize getting TLS everywhere in.

[ameowlia]

Hi all,

I hear your pain.

@jmprice, regarding windows, I believe the only part that was not encrypted is route emitter to nats. As of diego-release v2.41.0, that is now encrypted. Additionally, I know they worked on adding a sidecar to preform TLS from gorouter to the app container (though this might only be available on newer version of windows). Thus, if ipsec is used everywhere for linux, then the entire foundation should be encrypted. Please correct me if I am wrong here, I am not very involved in the windows world.

@voelzmo, my mistake, you are correct. I forgot that SAP decided to sunset OS ipsec.

I am tagging some people who make these platform wide decisions on the vmware side: @dieucao, @zrob, @emalm, @dsabeti, @mkocher

✨ of course we always welcome PRs from the community

[ameowlia]

Hi @plowin,

Per your question here, cloudfoundry/routing-release#185 (comment). This issues contains more information and is the best place to ask questions.

[voelzmo]

Hi there, it's been a while. I'm still looking to understand this

can you help us understand how we can get this work prioritized in the open source team? It might be helpful to get an understanding what the conflicting priorities are and when it would be possible to prioritize getting TLS everywhere in.

especially now that we've come to a common understanding that the initial assumption that there exists a workaround doesn't hold true.

I appreciate the "PRs welcome" message, however, given that we don't have people with context on the codebase for the involved projects, this will most likely remain a theoretical option, sorry.

[voelzmo]

Just another ping after 3 more weeks have passed. Can we talk about this, either in this ticket or in a direct meeting?