[OpAMP] Exchange initial API key (enrollment token) for per-Agent API key (access API key)

[ycombinator]

Describe the enhancement:

Implement a two-token authentication flow for OpAMP clients, analogous to the one used by Elastic Agent with Fleet Server today.

In this flow, OpAMP clients will send an API key when they first connect to Fleet Server. Let's call this the enrollment token. This enrollment token is expected to be shared amongst several OpAMP clients.

As each OpAMP client successfully enrolls with Fleet, i.e. a document for it is created in the .fleet-agents index so it shows up in the Fleet UI, a second API key, specific to this Agent, should be created by Fleet Server. Let's call this the access API key. This API key's ID should be stored in this Agent's document in .fleet-agents. The ServerToAgent response should set the connection_settings.opamp.headers field to Authorization: ApiKey <new_access_key>. When Fleet Server receives requests from an OpAMP client, it should validate the received access API key by comparing it's ID with the ID stored in the corresponding Agent's document in .fleet-agents.

IMPORTANT: A two-token authentication flow for OpAMP requires that OpAMP clients support the AcceptsOpAMPConnectionSettings capability. Therefore, before going down the path described above, Fleet Server must first check if this capability is set in the AgentToServer message it receives from the OpAMP client. If it is not set, Fleet Server must continue to work as it does today, where the single, shared enrollment token API key is used for authentication for all OpAMP clients connecting to Fleet Server.

As of this writing, the OpAMP extension does not have the ability to support this capability. However, the OpAMP supervisor does have the ability to support this capability.

Describe a specific use case for the enhancement or feature:

We want to have per-Agent access API keys — as opposed to a single API key that's shared amongst multiple Agents — so we can revoke access for a specific Agent.

[michel-laterman]

We should also support a scenario where a client that initially connect without the AcceptsOpAMPConnectionSettings capability using the enrollment token can change its capabilities and enable AcceptsOpAMPConnectionSettings to receive an api key through offered settings as described above.

Additionally, fleet-server should handle clients that have ReportsConnectionSettingsStatus set to ensure that sent settings are applied correctly

[cmacknz]

Discussed in the agent team meeting, with the current opamp-extension not having any persistence the security benefit to do this for the monitoring use case is limited. The extension would need the enrollment token to always be present in it's configuration, allowing it to generate a new key if it's current key were revoked without issue.

If we could get to the point where the enrollment token is not persisted anywhere but the per agent/collector API key generated is, then there is a benefit to this. This is how agent enrollment works, where the enrollment token is only used as a command line argument and is not stored by agent.

The opamp-supervisor does have the ability to persist state, so this would make sense there, but for EDOT collector we'll like just continue to use elastic-agent as the supervisor with it's existing enrollment flow instead.

We can look to enhance upstream to enable what we want, for example if the opamp-extension could be associated with a storage extension it would gain the ability to persist it's instance ID and a per collector API key, but users would also have to configure it since it would be optional.