Add recipe for individual Logstash pipeline management via ConfigMaps

[LolloneS]

No description provided.

[prodsecmachine]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scanner	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[pebrc]

Thanks for contributing this recipe! I have some feedback before this can be merged.

Bugs

There are a few bugs that will prevent this recipe from working (see inline comments):

1. Secret name mismatch (kafka-credentials vs kafka-pipeline-secret)
2. Incorrect Elasticsearch environment variable names (ES_HOSTS should be ECK_ES_HOSTS, etc.)
3. Wrong option name (cacert should be ssl_certificate_authorities)

Missing Documentation

Please update config/recipes/logstash/README.asciidoc to document this new recipe.

Architectural Concern

I'm wondering about the value proposition of this approach compared to using pipelinesRef with Secrets, which ECK manages automatically.

The existing logstash-pipeline-as-secret.yaml recipe shows:

spec:
  pipelinesRef:
    secretName: logstash-pipeline

This approach requires zero volume mount boilerplate – ECK handles it all.

The ConfigMap approach in this PR requires 30+ lines of manual volume/volumeMount configuration, which is error-prone (as the bugs above demonstrate) and requires the user to keep paths in sync between pipelines[].path.config and volumeMounts[].mountPath.

Questions:

• Is there a compelling reason to use ConfigMaps over Secrets here? Pipeline configs typically don't contain secrets since credentials are injected via env vars.
• If the goal is "modular, separate resources per pipeline," wouldn't multiple Secrets with pipelinesRef achieve the same with less complexity?

If there's a strong use case for this pattern (e.g., company policies requiring ConfigMaps for non-sensitive data), please add a note in the recipe explaining when users should choose this approach over pipelinesRef.

[LolloneS]

Hi @pebrc and thank you for your comments. I addressed most of them and will update the README ASAP.

I'll try to answer your questions as well as I can.

Is there a compelling reason to use ConfigMaps over Secrets here? Pipeline configs typically don't contain secrets since
credentials are injected via env vars.

Reasons:

1. Simply said, pipelines are not secrets. Indeed, even in virtual machines, they are stored in plaintext. They might reference some secrets such as credentials, but they are not anything that should be hidden. Actually, I believe we should have a Pipeline CR that can reference Secrets for credentials, but that's another story :) long story short, there is a mismatch between the "classic" Logstash experience and the ECK one.

2. ConfigMaps are easier to edit and manage (think GitOps approaches), and using them allows for separation of concerns. Why would you hide an entire pipeline rather than only hiding the credentials it uses? I would rather have a setup where the pipeline is represented as a ConfigMap managed e.g. via ArgoCD and the credentials are managed e.g. via the External Secrets Operator rather than having a massive, single Secret that I can't easily and dynamically edit.

If the goal is "modular, separate resources per pipeline," wouldn't multiple Secrets with pipelinesRef achieve the same with less complexity?

No. Again, think about a GitOps setup. Why would you store the pipeline as an encrypted, unreadable object in Git (think SOPS encryption etc.) rather than having it readable and just loading the few required secrets dynamically? The approach with Secrets leads to more complexity, not less. But I agree with the general point that we are introducing an overhead (ConfigMap + Secret) to make up for the absence of a Pipeline CR.

However, to your point, having multiple ConfigMaps (or Pipeline CRs) also indeed helps having a modular concept that's similar to what customers are used to when running Logstash on VMs rather than on K8s.

If there's a strong use case for this pattern (e.g., company policies requiring ConfigMaps for non-sensitive data), please add a note in the recipe explaining when users should choose this approach over pipelinesRef.

I honestly believe the use-case is simply using the most appropriate tool for the job. This very point came up during a consulting engagement with a customer yesterday. They are heavy Kubernetes users and were pretty surprised that we don't allow for a custom CR or a ConfigMap to be used for configuring pipelines, because Secrets are a pain to work with and our current setup does not integrate well with GitOps workflows and external operators handling real Secrets (i.e. credentials etc.).

[pebrc]

After another pass, I am still a little sceptical about this recipe.

1. Recipe not runnable without Kafka

The recipe defines a Kafka input pipeline pointing at kafka-broker:9092, but the YAML doesn’t deploy Kafka. Our other Logstash recipes are self-contained: everything needed is in the file and kubectl apply -f gives a working stack. This one breaks that expectation unless Kafka is deployed separately, so I would not want to add it as a first-class recipe in its current form.

2. Overlap and recipe bloat

We already cover “pipeline config from outside the spec” in several ways: logstash-pipeline-as-secret.yaml (pipelinesRef), logstash-pipeline-as-volume.yaml (one pipeline from one volume), and inline config in logstash-eck.yaml. Adding another recipe that combines multiple pipelines, multiple ConfigMaps, and manual volume wiring would overlap with these and add only incremental value imo. I would want to try to keep the recipe set small so users aren’t confused by many similar options, so I would not add this as a new standalone recipe.

3. Alternative direction

I get the ConfigMap argument. For “single pipelines definition in a ConfigMap,” I’ve opened issue #9110 to add ConfigMap support to pipelinesRef. For the “multiple pipelines, each in its own ConfigMap” pattern, we’d rather document it in the Logstash docs in the pipelines as volumes section (with a short “when to use” note vs pipelinesRef) than add another recipe file.

Wdyt?

[pebrc]

I tihnk this incorrect an should be ECK_ES_HOSTS?