Elastic Defend Integration fails to connect to Kafka with authentication

[ptamba]

Stack Version : 9.2.0
Elastic Agent Version : 9.2.0 - 9.2.3
Related Integration : Elastic Defend v9.2.0

Elastic Defend Integration fails to connect to Kafka with SASL Authentication enabled. The following is observed in endpoint log

{"@timestamp":"2026-01-06T03:52:57.882454154Z","agent":{"id":"3756ca50-e27f-486a-b7e3-b5bc05cdfba3","type":"endpoint"},"ecs":{"version":"8.10.0"},"log":{"level":"error","origin":{"file":{"line":62,"name":"KafkaClient.cpp"}}},"message":"KafkaClient.cpp:62 Kafka Error: Local: Authentication failure [-169] | sasl_ssl://****:9092/bootstrap: SASL authentication error: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256 (after 382ms in state AUTH_REQ, 4 identical error(s) suppressed)","process":{"pid":284390,"thread":{"id":284437}}}

The same kafka output with the same authentication is used by other integration, and it's working fine. see attachment

relevant output section:

output:
kafka:
__mark_redact_password: true
broker_timeout: 30
client_id: fleet-agents
compression: none
hosts:
- :9092
partition:
random:
group_events: 1
password:
required_acks: 1
sasl:
mechanism: SCRAM-SHA-256
ssl:
certificate_authorities:
-
verification_mode: certificate
timeout: 30
topic: '%{[labels.kafka_topic]}'
type: kafka
username: elastic-agent
version: 2.6.0
revision: 17

[brian-mckinney]

Hi @ptamba , would it be possible to get some more information about the setup here? I've just verified locally that endpoint can talk to a properly configured broker via sasl using SCRAM-SHA-256.

9092 is typically the plaintext port for kafka installations, and when username and password are specified, kibana forces the config to use ssl for the connection which is usually port 9093. Additionally, the error

Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256

comes from the underlying librdkafka library Endpoint uses for the connection. It also points towards an authentication issue.

Can you please provide answers to the following so we can investigate further:

• Can you confirm that a sasl_ssl listener is listening on port 9092 on the broker?
• Double check that the username and password are valid
• Provide an agent diagnostics for an affected endpoint. This will let us see more endpoint errors, and should also validate that the other integrations are able to talk to kafka

[ptamba]

hi @brian-mckinney,

1. yes, sasl_ssl listener is listening on port 9092 on the broker.

2. yes, I've validated that the username and password is configured. I used the same kafka output for elastic-defend, auditd, and system integration. endpoint log under /opt/Elastic/Endpoints showed SASL auth error, agent log under /opt/Elastic/Agent/ show SASL authentication succeeded
3. Is there a way for us to send you the diagnostics separately? or is it safe to attach it here ?

[brian-mckinney]

Hi @ptamba, I've created an secure upload link for you. You can upload the diagnostics here and I'll get notified, and it will auto delete after 14 days. https://upload.elastic.co/u/98d68747-65db-4d27-a6e0-14c422a532a4

[brian-mckinney]

A few other questions that could help troubleshoot the issue.

• Is this a production environment?
• Is the Kafka broker a production instance? Docker stack?
• If you have flexibility, are you able to change the log level of the kafka broker and collect debug logs as endpoint is failing?
• Again, if you are able to, have you tried setting up any other listener types on the broker and seeing if Endpoint is able to connect.

[ptamba]

Hi @brian-mckinney ,

This is dev env. Kafka broker is installed as software on ubuntu. I have uploaded both the diagnostics file as well as the debug log from authentication. the agent is coming from ip 10.66.13.24

i tried creating other listener:

1. no auth: works with Elastic Endpoint
2. SASL_PLAINTEXT: failed. and endpoint log showed "SSL handshake failed:" despite config is using SASL_PLAINTEXT. elastic-endpoint inspect showed ssl section , which is probably causing it. I've uploaded the diagnostic with later timestamp which is using SASL_PLAIN

output:
  kafka:
    __mark_redact_password: true
    broker_timeout: 30
    client_id: Elastic
    compression: none
    hosts:
    - *****:9094
    - ******:9094
    - *****:9094
    partition:
      random:
        group_events: 1
    password: <REDACTED>
    required_acks: 1
    sasl:
      mechanism: PLAIN
    ssl:
      verification_mode: none
    timeout: 30
    topic: *****
    type: kafka
    username: elastic-agent
    version: 2.6.0
revision: 26

[brian-mckinney]

Hi @ptamba, quick update. I'm still working on this, just didn't have a chance to look at it again until now.

Looking at the logs, I definitely see that beats is having no problem authenticating, but Defend is.

Defend

SASL authentication error: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256

Beats

kafka message: SASL authentication succeeded

Considering in my testing, endpoint authenticates fine with SCRAM-SHA-256, there must be a difference somewhere in our broker configurations. The fact that beats uses a go library, while Defend uses C++ also adds some wrinkles.

I'm going to play around with my dev broker settings to see if I can better replicate your scenario, and hopefully provide you with a better answer soon.

Additional Notes

SSL Question

SASL_PLAINTEXT: failed. and endpoint log showed "SSL handshake failed:" despite config is using SASL_PLAINTEXT. elastic-endpoint inspect showed ssl section , which is probably causing it. I've uploaded the diagnostic with later timestamp which is using SASL_PLAIN

This is expected actually, kibana seems to require ssl when choosing any form of sasl authentication and send an ssl section regardless. Endpoint assumes sasl_ssl when an ssl section is present.

Potential Topic issue

Defend does not support all of the functionality that beats does, and unfortunately the differences are not always documented really well. Your use of a processor field

processors:
  - add_fields:
      fields:
        labels:
          kafka_topic: logs_mssp_client2

Isn't being translated correctly, and endpoint is treating it as a static string: %{[labels.kafka_topic]}. I don't think it would interfere with the sasl authentication though. Defend supports dynamic topics that use message fields for substitutions, so choosing a field in the dynamic topic selector would work, or you could define your own in the static topic field

[ptamba]

Hi @brian-mckinney,

can you share your broker configuration so i can see the difference? Here's my running config. I'm running Kafka 4.0.1 with Kraft on ubuntu :

listeners=BROKER://:9092,CONTROLLER://:9093
listener.security.protocol.map=CONTROLLER:SASL_SSL,BROKER:SASL_SSL
advertised.listeners=BROKER://*.*.*.*:9092
sasl.enabled.mechanisms=SCRAM-SHA-256,PLAIN
listener.name.broker.sasl.enabled.mechanisms=SCRAM-SHA-256,PLAIN
listener.name.broker.scram-sha-256.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required;
listener.name.broker.ssl.keystore.location=/opt/kafka/config/certs/kafka-node-1.keystore.jks
listener.name.broker.ssl.keystore.password=*****
listener.name.broker.ssl.key.password=*****
listener.name.broker.ssl.truststore.location=/opt/kafka/config/certs/truststore.jks
listener.name.broker.ssl.truststore.password=*****
listener.name.broker.ssl.client.auth=none
listener.name.broker.ssl.enabled.protocols=TLSv1.2,TLSv1.3
listener.name.broker.ssl.protocol=TLSv1.3

on your comment:

This is expected actually, kibana seems to require ssl when choosing any form of sasl authentication and send an ssl section regardless. Endpoint assumes sasl_ssl when an ssl section is present.

I've suspected this as well, since the generated elastic-agent.yaml contains an ssl section, although it is empty. However, in a fleet output configuration, there is no option to disable SSL, even with Plaintext configuration? it looks like even if user selected the PLAIN SASL mechanism, the generated config always contains ssl section, which makes it impossible to use plain SASL. Is my understanding correct?

Isn't being translated correctly, and endpoint is treating it as a static string: %{[labels.kafka_topic]}. I don't think it would interfere with the sasl authentication though. Defend supports dynamic topics that use message fields for substitutions, so choosing a field in the dynamic topic selector would work, or you could define your own in the static topic field

Yes, I noticed endpoint doesn't have processor configuration so I created different output for endpoint with exact authentication configuration, but static topic instead of dynamic topic. SASL authentication still fails so I don't think it has effect on the auth.

[brian-mckinney]

Had chance to get into this again today. I was able to reproduce the issue finally. Using this docker compose stack and certs generated from kafka-generate-ssl.sh

services:
  kafka:
    image: bitnamilegacy/kafka:latest
    hostname: localhost
    ports:
      - 9092:9092
      - 9093:9093
    environment:
      # KRaft
      - KAFKA_CFG_NODE_ID=0
      - KAFKA_CFG_PROCESS_ROLES=controller,broker
      # Listeners
      - KAFKA_CFG_LISTENERS=BROKER://:9092,CONTROLLER://:9093
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:SASL_PLAINTEXT,BROKER:SASL_SSL
      - KAFKA_CFG_ADVERTISED_LISTENERS=BROKER://:9092
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=BROKER
      - KAFKA_CLIENT_LISTENER_NAME=BROKER
      - KAFKA_CONTROLLER_QUORUM_VOTERS=0@localhost:9093
      # SASL
      - KAFKA_CFG_SASL_ENABLED_MECHANISMS=PLAIN,SCRAM-SHA-256
      - KAFKA_CFG_SASL_MECHANISM_BROKER_PROTOCOL=SCRAM-SHA-256
      - KAFKA_CFG_SASL_MECHANISM_CONTROLLER_PROTOCOL=PLAIN
      - KAFKA_CFG_SASL_MECHANISM_INTER_BROKER_PROTOCOL=SCRAM-SHA-256
      - KAFKA_CONTROLLER_USER=controller_user
      - KAFKA_CONTROLLER_PASSWORD=controller_password
      - KAFKA_INTER_BROKER_USER=interbroker_user
      - KAFKA_INTER_BROKER_PASSWORD=interbroker_password
      - KAFKA_CLIENT_USERS=user
      - KAFKA_CLIENT_PASSWORDS=password
      # SSL
      - KAFKA_TLS_TYPE=JKS
      - KAFKA_CERTIFICATE_PASSWORD=password
    volumes:
      - ./certificates/keystore/kafka.keystore.jks:/opt/bitnami/kafka/config/certs/kafka.keystore.jks:ro
      - ./certificates/truststore/kafka.truststore.jks:/opt/bitnami/kafka/config/certs/kafka.truststore.jks:ro

I verified that against this broker, I could authenticate and produce messages with a python client, but endpoint fails to produce with the error:

Kafka error callback: Local: Authentication failure [-169] | sasl_ssl://localhost:9092/bootstrap: SASL authentication error: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256 (after 320ms in state AUTH_REQ)

I compared all of the config properties between endpoint and the python script, and they were all identical. The only thing I couldn't check was that endpoint is getting the right username and password, since those are masked in the logs. I am currently building a debug endpoint with some extra logging around the creds to see if maybe that is the problem. It is possible that maybe an extra space or something is getting added in there.

Rebuilding the endpoint is taking a long time, so I at least wanted to provide an update before the weekend. Hope to have a real answer soon. 🤞

[brian-mckinney]

I made more progress today. It looks librdkafka needs to be upgraded on Defend. Defend uses 2.1.1 which is well behind the current 2.13.0. This was why my python script worked fine, but endpoint failed to authenticate. It appears to be related to the fact that OpenSSL was upgraded to > 3.0, and librdkafka needed to be updated as well to play nice with newer crypto functions.

This also explains why the other Elastic services were working fine, since they are written in go. I'm creating an internal issue to update the librdkafka version on the Endpoint. I am unsure if SCRAM-SHA-512 has the same issue, or if ssl authentication works right now. I'll be testing those out as well, if either work I will let you know. I will also try to post back here when I have an idea of when Endpoint will be upgraded.

CC @nfritts

[ptamba]

Hi @brian-mckinney ,

Thanks a lot for the update. I’m looking forward to hearing more updates on this, especially workaround on the ssl side. At the moment we’re forced to use logstash to route the data to kafka, but direct kafka connection is preferred

[brian-mckinney]

I tested it out today using SSL authentication and Endpoint was able to authenticate and write to the same broker it failed SASL with. If you are able, try setting up SSL authentication in your test environment and see if that works for you. I'll update this thread when I know what release(s) the upgraded librdkafka will be in.