how to control amount of memory used by vector?

[yjagdale]

I tried controlling the batch at the sink level but still the amount of memory is not getting controller,

[pront]

Hi @yjagdale, this is a great question. Are you referring to a process wide limit or per component? Regardless, here is some information that you might find useful.

systemd service manager

If your Vector instance runs as a systemd service (Linux), you can configure resource limits in the service file which usually found at /etc/systemd/system/vector.service:

[Service]
MemoryLimit=1G

Configure in-memory buffers

Of course this is not really a process (or even a component) wide limit, but might be helpful:

    buffer:
    - type: memory
      max_events: 1000
      when_full: drop_newest

Tip

For monitoring memory usage you can use vector top, https://vector.dev/docs/reference/cli/#top.

[yjagdale]

When I set the memory limit, Then the service is going out of memory( I am seeing this at kubernetes pod level).
I am using kafka as input then few log to metrics and output to prometheus.

[jszwedko]

I think you may be interested in this feature request: #8792.

There is no way, at the moment, to globally limit Vector memory use in a way that Vector will apply back-pressure rather than just OOM. There are some knobs that can be turned like buffer limits mentioned above. You mentioned Kafka as the source so you may also be interested in some of the options mentioned in https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md

[Sasha-mina]

The OOMKill pattern you're seeing with Kafka → metrics → Prometheus is a well-known constraint in Vector's architecture — @jszwedko is right that there's no global backpressure mechanism yet.

A few things worth trying that go beyond the standard buffer tuning:

1. Limit Kafka fetch size at the librdkafka level
Rather than trying to cap memory after events are in flight, limit how fast they arrive:

[sources.kafka_in.librdkafka_options]
"fetch.max.bytes" = "1048576"       # 1MB per fetch
"max.partition.fetch.bytes" = "524288"  # 512KB per partition
"queued.max.messages.kbytes" = "16384"  # 16MB total queue

This slows ingestion at the source before events enter Vector's internal pipeline, which is the only real lever for memory control right now.

2. Set tight memory buffer limits per sink with drop_newest

[sinks.prometheus_out]
type = "prometheus_remote_write"
buffer.type = "memory"
buffer.max_events = 500
buffer.when_full = "drop_newest"

Dropping newest (rather than blocking) prevents the sink buffer from growing unboundedly when Prometheus is slow to accept writes.

3. Cap the log_to_metric transform output
If your log_to_metric transforms are expanding events (one log → multiple metrics), memory can multiply quickly. Consider adding a filter transform before the sink to drop low-priority metrics under load.

On the broader architecture — the root issue is that Vector applies backpressure at the buffer level rather than at the source read level, which means a burst of Kafka messages can flood the in-flight pipeline faster than the Prometheus sink can drain it. The systemd MemoryLimit approach then becomes a hard kill rather than graceful degradation.

For what it's worth, this is one of the core design tradeoffs we revisited when building WarpParse — applying backpressure at the read level so that memory stays bounded even when downstream sinks are slow. Happy to share more detail on how that works if it's useful.