Implement apm support for presenting client certificates to elasticsearch

[pkoutsovasilis]

Summary

This PR implements client certificate support for APM Server when connecting to an Elasticsearch cluster that has client authentication enabled.

Relates to #9081

Changes

• APM ElasticsearchSelector: Changes ElasticsearchRef from ObjectSelector to ElasticsearchSelector, adding clientCertificateSecretName support to APM Server's elasticsearchRef
• APM Server config: Configures output.elasticsearch.ssl.certificate and output.elasticsearch.ssl.key when client authentication is required, pointing to the mounted client certificate files
• APM Server pod spec: Mounts client certificate volumes alongside existing CA certificate volumes for associations that have client certificates configured. Refactored withAssociationCACertsVolumes into withAssociationCertsVolumes to handle both CA and client certificate volume mounts
• Webhook validation: Updated to use CheckElasticsearchSelectorRefs for the APM Server's elasticsearchRef
• APM e2e tests: Added TestClientAuthRequiredTransition (verifies APM Server remains healthy when ES transitions from client auth required to disabled, and that client certificate secrets are cleaned up) and TestClientAuthRequiredCustomCertificate (verifies APM Server works with a user-provided client certificate)

API

apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
spec:
  elasticsearchRef:
    name: elasticsearch
    # Optional: use a custom client certificate
    clientCertificateSecretName: my-custom-client-cert

[prodsecmachine]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[pkoutsovasilis]

buildkite test this -f p=gke,t=TestClientAuthRequired.*,E2E_TAGS=apm -m s=9.3.2,s=8.19.13,s=9.2.7

[pkoutsovasilis]

buildkite test this -f p=gke,t=TestClientAuthRequired.*,E2E_TAGS=apm -m s=9.3.3,s=8.19.14,s=9.2.8

[simitt]

Overall, LGMT, just some nits - but feel free to ignore.

nit: I wonder whether a test scenario where the ES cluster uses a publicly trusted cert but mTLS is configured is worth testing in the unit tests.

[github-actions]

🔍 Preview links for changed docs

• docs/reference/api-reference/main.md

[github-actions]

Vale Linting Results

Summary: 1 warning found

⚠️ Warnings (1)

File	Line	Rule	Message
docs/reference/api-reference/main.md	207	Elastic.BritishSpellings	Use American English spelling 'customization' instead of British English 'customisation'.

---

The Vale linter checks documentation changes against the Elastic Docs style guide.

To use Vale locally or report issues, refer to Elastic style guide for Vale.

[pkoutsovasilis]

nit: I wonder whether a test scenario where the ES cluster uses a publicly trusted cert but mTLS is configured is worth testing in the unit tests.

A couple of thoughts on this:

1. I'm not sure how we'd fabricate a publicly trusted cert in our e2e CI environment.
2. A publicly trusted cert affects the client-side validation of the server (ES in our case) certificate - it would be trusted without needing a defining a custom trusted CA. But the server-side validation of client certificates (the mTLS part) remains the same regardless of how the server's own cert is issued. The client still needs to present a valid certificate that the server trusts.
3. I'd also expect mTLS with publicly trusted server certificates to be fairly uncommon in practice - In my head, although I can be off here, mTLS is typically used in internal/private zero-trust environments where self-signed or internal CAs are the norm.

Is there a particular edge case you have in mind @simitt where the combination would behave differently? Happy to discuss and fabricate a test if there's a concrete scenario worth covering.

[simitt]

@pkoutsovasilis , looking at this pod_test, my concern actually is already covered - client cert only, no CA. So you can safely drop my previous comment.

Is there a particular edge case you have in mind

No particular use case, just something I thought missing in test coverage.