Vector high CPU usage

[shivamvishwakarma27]

Question

Hi Team,
I am running vector in Agent mode over 266+ nodes. I am doing POC on it, having kubernetes log as source and Clickhouse as Sink. I had deployed vector then the CPU usage was fine but suddenly the CPU spiked from 0.5 core to 3.5+ cores on more than 5-6 vector pods. I want to know what are reason which is causing high cpu usage.

• I tried Lowering batch size to 1MB, didn't help.
• Tried max events lowering to 10K, it also didn't help.
• Increased timeout on Sink to 30s, this also didn't help.
• Disabled ACk flag, it also didn't help.
• Gone through with reported vector issue and someone raised that upgrading from 0.48.0 to 0.50.0 caused but I tried downgrading the version that also didn't help.
• Now I'm confuesd what parameter is casuing this.

Vector Config

customConfig:
  data_dir: /var/lib/vector
  api:
    enabled: true
    address: 0.0.0.0:8686
    playground: true
    pprof = true
  
  # Source: Kubernetes logs collection
  sources:
    internal_metrics:
      type: internal_metrics
    
    kubernetes_logs:
      type: kubernetes_logs
      auto_partial_merge: true
      oldest_first: true
      # max_line_bytes: 32768
      glob_minimum_cooldown_ms: 60000
      self_node_name: ${VECTOR_SELF_NODE_NAME}
      data_dir: /var/lib/vector
      fingerprint_lines: 1
      # Handle small log files
      ignore_older_secs: 900  # Don't ignore files based on age
      insert_namespace_fields: true
      # Only include pod log files, exclude events
      include_paths_glob_patterns:
        - "/var/log/pods/**/*.log"
      exclude_paths_glob_patterns:
        - "**/events/**"
        - "**/*events*.log"
  
  # Transform: Add labels similar to Promtail configuration
  transforms:
    add_labels:
      type: remap
      inputs:
        - kubernetes_logs
      source: |-
        # Extract kubernetes metadata (guaranteed to exist for pod logs)
        node_name = string!(.kubernetes.pod_node_name)
        namespace = string!(.kubernetes.pod_namespace)
        pod_name = string!(.kubernetes.pod_name)
        container_name = string!(.kubernetes.container_name)
        
        # Set basic fields with proper types
        .node_name = node_name
        .namespace = namespace
        .pod = pod_name
        .container = container_name
        .cluster = "qa"
        .source = "vector"
        .deployed_on = "k8s"
        
        # App label from pod labels or container name
        app_label, err = get(.kubernetes.pod_labels, ["app"])
        app_name = if err == null { string(app_label) ?? container_name } else { container_name }
        .app = app_name
        
        # Filename - use the full file path
        .filename = string!(.file)
        
        # Stream (stdout/stderr)
        .stream = string!(.stream)
        
        # Job label (namespace/app) - standard Prometheus/Loki format
        .job = namespace + "/" + app_name
        
        # Extract log level from message
        # Check if debug, info, error, warning appears anywhere in message
        message = string!(.message)
        level = "unknown"
        
        # Simple case-insensitive match for common log levels
        if match(message, r'(?i)\berror\b') {
          level = "ERROR"
        } else if match(message, r'(?i)\b(warning|warn)\b') {
          level = "WARN"
        } else if match(message, r'(?i)\b(information|info)\b') {
          level = "INFO"
        } else if match(message, r'(?i)\bdebug\b') {
          level = "DEBUG"
        } else if match(message, r'(?i)\btrace\b') {
          level = "TRACE"
        } else if match(message, r'(?i)\b(fatal|critical)\b') {
          level = "FATAL"
        }
        
        .level = level
  
  # Sink: Loki output matching Promtail configuration
  sinks:
    # Debug console sink - REMOVE after testing
    # console_debug:
    #   type: console
    #   inputs:
    #     - add_labels
    #   target: stdout
    #   encoding:
    #     codec: json
    
    # loki:
    #   type: loki
    #   inputs:
    #     - add_labels
    #   endpoint: "http://nonprod-logging.sage.net"
    #   encoding:
    #     codec: text
    #   tenant_id: "kube-uat-{{`{{ namespace }}`}}"
    #   remove_label_fields: false
    #   remove_timestamp: false
    #   out_of_order_action: drop
    #   batch:
    #     max_bytes: 262144  # Same as Promtail batchsize
    #     timeout_secs: 60
    #   request:
    #     retry_max_duration_secs: 300  # Approximate equivalent to max_retries: 7
    #     timeout_secs: 60
    #   labels:
    #     app: "{{`{{ app }}`}}"
    #     cluster: "{{`{{ cluster }}`}}"
    #     container: "{{`{{ container }}`}}"
    #     namespace: "{{`{{ namespace }}`}}"
    #     node_name: "{{`{{ node_name }}`}}"
    #     deployed_on: "{{`{{ deployed_on }}`}}"
    #     pod: "{{`{{ pod }}`}}"
    #     stream: "{{`{{ stream }}`}}"
    #     filename: "{{`{{ filename }}`}}"
    #     job: "{{`{{ job }}`}}"
    #     source: "{{`{{ source }}`}}"
    #     level: "{{`{{ level }}`}}"
    
    # ClickHouse sink for Vector logs (custom table)
    clickhouse:
      type: clickhouse
      inputs:
        - add_labels
      endpoint: "http://clickhouse-logs-clickhouse-db.observability.svc.cluster.local:8123"
      database: "default"
      table: "qa_vector_logs"
      auth:
        strategy: "basic"
        user: "${CLICKHOUSE_USER}"
        password: "${CLICKHOUSE_PASSWORD}"
      compression: zstd  # Fast compression with minimal CPU overhead
      encoding:
        timestamp_format: "unix"
      
      # Batching - MATCH Promtail's config for consistent behavior
      batch:
        max_bytes: 16777216      # 16MB batches (SAME as Promtail batchsize: 262144) 
        max_events: 20000     # Or 20K events, whichever comes first
        timeout_secs: 30 
      
      # Buffer configuration - CRITICAL to prevent log loss
      buffer:
        type: "disk"           # Use disk buffer to handle backpressure
        max_size: 1073741824  # 1GB disk buffer (prevent drops when ClickHouse is slow)
        when_full: "drop_newest"     # Drop newest log writes when buffer full
      
      # Request settings - MATCH Promtail's retry behavior
      request:
        retry_max_duration_secs: 300      # Retry for 5 minutes (matches Promtail max_retries: 7 with backoff)
        retry_initial_backoff_secs: 3     # Start with 1 second backoff
        timeout_secs: 60                  # 1 minute timeout per request
      
      # Health check
      healthcheck:
        enabled: true                     # Enable health checks to detect issues
      
      # Acknowledgments - ensure logs are written before removing from buffer
      acknowledgements:
        enabled: false
      
      skip_unknown_fields: true
    
    # Prometheus exporter for Vector internal metrics
    prometheus_exporter:
      type: prometheus_exporter
      inputs:
        - internal_metrics
      address: 0.0.0.0:9090
      default_namespace: vector

Vector Logs

2026-02-17T08:07:48.601747Z DEBUG sink{component_kind="sink" component_id=prometheus_exporter component_type=prometheus_exporter}:http-request{method=GET path=/metrics}: vector::internal_events::http: Received HTTP request.
2026-02-17T08:07:48.602799Z DEBUG hyper::proto::h1::io: Internal log [flushed 279 bytes] has been suppressed 2 times.
2026-02-17T08:07:48.602810Z DEBUG hyper::proto::h1::io: flushed 5610 bytes
2026-02-17T08:07:52.140096Z DEBUG vector::internal_events::file::source: Files checkpointed. count=35 duration_ms=1
2026-02-17T08:07:54.191920Z DEBUG source{component_kind="source" component_id=kubernetes_logs component_type=kubernetes_logs}:file_server: file_source::file_server: event_throughput=184.000/sec bytes_throughput=201.248k/sec ratios={"discovery": 0.00013266272, "other": 0.0004653868, "reading": 0.013157006, "sending": 0.0025158322, "sleeping": 0.98372906}
2026-02-17T08:07:54.195502Z INFO source{component_kind="source" component_id=kubernetes_logs component_type=kubernetes_logs}:file_server: vector::internal_events::file::source: Found new file to watch. file=/var/log/pods/search-platform_search-service-6fd9b5d799-9gnc7_a1f9aaec-e7a6-4d2b-b93e-9024b98bd25a/search-service/0.log
2026-02-17T08:07:54.197459Z INFO source{component_kind="source" component_id=kubernetes_logs component_type=kubernetes_logs}:file_server: vector::internal_events::file::source: Internal log [Found new file to watch.] is being suppressed to avoid flooding.
2026-02-17T08:07:54.333648Z DEBUG sink{component_kind="sink" component_id=clickhouse component_type=clickhouse}:request{request_id=30}:http: vector::internal_events::http_client: Sending HTTP request. uri=http://clickhouse-logs-clickhouse-db.observability.svc.cluster.local:8123/?input_format_import_nested_json=1&input_format_skip_unknown_fields=1&query=INSERT+INTO+%22default%22.%22qa_vector_logs%22+FORMAT+JSONEachRow method=POST version=HTTP/1.1 headers={"content-type": "application/x-ndjson", "content-length": "44483", "content-encoding": "zstd", "authorization": Sensitive, "accept-encoding": "zstd,gzip,deflate,br", "user-agent": "Vector/0.51.1 (x86_64-unknown-linux-gnu 44c8f1c 2025-11-13 15:16:05.303418529)"} body=[44483 bytes]
2026-02-17T08:07:54.333688Z DEBUG sink{component_kind="sink" component_id=clickhouse component_type=clickhouse}:request{request_id=30}:http: hyper::client::pool: reuse idle connection for ("http", clickhouse-logs-clickhouse-db.observability.svc.cluster.local:8123)
2026-02-17T08:07:54.334031Z DEBUG hyper::proto::h1::io: Internal log [flushed 279 bytes] is being suppressed to avoid flooding.
2026-02-17T08:07:54.341944Z DEBUG hyper::proto::h1::io: Internal log [parsed 11 headers] is being suppressed to avoid flooding.
2026-02-17T08:07:54.341963Z DEBUG hyper::proto::h1::conn: Internal log [incoming body is chunked encoding] is being suppressed to avoid flooding.
2026-02-17T08:07:54.341976Z DEBUG hyper::proto::h1::conn: incoming body completed
2026-02-17T08:07:54.341999Z DEBUG sink{component_kind="sink" component_id=clickhouse component_type=clickhouse}:request{request_id=30}:http: hyper::client::pool: pooling idle connection for ("http", clickhouse-logs-clickhouse-db.observability.svc.cluster.local:8123)
2026-02-17T08:07:54.342024Z DEBUG sink{component_kind="sink" component_id=clickhouse component_type=clickhouse}:request{request_id=30}:http: vector::internal_events::http_client: HTTP response. status=200 OK version=HTTP/1.1 headers={"date": "Tue, 17 Feb 2026 08:07:54 GMT", "connection": "Keep-Alive", "content-type": "text/plain; charset=UTF-8", "access-control-expose-headers": "X-ClickHouse-Query-Id,X-ClickHouse-Summary,X-ClickHouse-Server-Display-Name,X-ClickHouse-Format,X-ClickHouse-Timezone,X-ClickHouse-Exception-Code", "x-clickhouse-server-display-name": "chi-logs-clickhouse-db-logs-clickhouse-1-0-0.chi-logs-clickhouse-db-logs-clickhouse-1-0.observability.svc.cluster.local", "transfer-encoding": "chunked", "x-clickhouse-query-id": "ab0c3f89-082f-484d-9fc1-be0839fa4efc", "x-clickhouse-timezone": "UTC", "keep-alive": "timeout=300, max=9970", "x-clickhouse-summary": "{"read_rows":"0","read_bytes":"0","written_rows":"0","written_bytes":"0","total_rows_to_read":"0","result_rows":"0","result_bytes":"0","elapsed_ns":"6615007"}"} body=[unknown]
2026-02-17T08:07:54.440446Z DEBUG sink{component_kind="sink" component_id=clickhouse component_type=clickhouse}:request{request_id=31}:http: vector::internal_events::http_client: Internal log [Sending HTTP request.] is being suppressed to avoid flooding.
2026-02-17T08:07:54.440483Z DEBUG sink{component_kind="sink" component_id=clickhouse component_type=clickhouse}:request{request_id=31}:http: hyper::client::pool: Internal log [reuse idle connection for ("http", clickhouse-logs-clickhouse-db.observability.svc.cluster.local:8123)] is being suppressed to avoid flooding.
2026-02-17T08:07:54.456323Z DEBUG hyper::proto::h1::conn: Internal log [incoming body completed] is being suppressed to avoid flooding.
2026-02-17T08:07:54.456542Z DEBUG sink{component_kind="sink" component_id=clickhouse component_type=clickhouse}:request{request_id=31}:http: hyper::client::pool: Internal log [pooling idle connection for ("http", clickhouse-logs-clickhouse-db.observability.svc.cluster.local:8123)] is being suppressed to avoid flooding.
2026-02-17T08:07:54.456572Z DEBUG sink{component_kind="sink" component_id=clickhouse component_type=clickhouse}:request{request_id=31}:http: vector::internal_events::http_client: Internal log [HTTP response.] is being suppressed to avoid flooding.
2026-02-17T08:07:54.811944Z DEBUG transform{component_kind="transform" component_id=add_labels component_type=remap}:try_write_record_inner: vector_buffers::variants::disk_v2::writer: Archived record is too large to fit in remaining free space of current data file. current_data_file_size=133707160 max_data_file_size=134217728 archive_on_disk_len=528560
2026-02-17T08:07:54.815398Z DEBUG transform{component_kind="transform" component_id=add_labels component_type=remap}:try_write_record_inner: vector_buffers::variants::disk_v2::writer: Current data file reached maximum size. Rolling to the next data file. current_data_file_size=133707160 max_data_file_size=134217728 last_attempted_write_size=528560
2026-02-17T08:07:54.824386Z DEBUG transform{component_kind="transform" component_id=add_labels component_type=remap}:try_write_record_inner:ensure_ready_for_write: vector_buffers::variants::disk_v2::writer: Opened data file for writing. data_file_path="/var/lib/vector/buffer/v2/clickhouse/buffer-data-2.dat" existing_file_size=0
2026-02-17T08:07:54.824792Z DEBUG transform{component_kind="transform" component_id=add_labels component_type=remap}:try_write_record_inner:ensure_ready_for_write: vector_buffers::variants::disk_v2::writer: Writer now on new data file. new_writer_file_id=2
2026-02-17T08:07:54.829042Z DEBUG sink{component_kind="sink" component_id=clickhouse component_type=clickhouse}: vector_buffers::variants::disk_v2::reader: Reached the end of current data file. reader_file_id=1 writer_file_id=2
2026-02-17T08:07:54.829094Z DEBUG sink{component_kind="sink" component_id=clickhouse component_type=clickhouse}: vector_buffers::variants::disk_v2::reader: Marking data file for deletion. data_file_path="/var/lib/vector/buffer/v2/clickhouse/buffer-data-1.dat" first_record_id=27339 last_record_id=51669 record_count=1702 event_count=24331 bytes_read=133707160
2026-02-17T08:07:54.829128Z DEBUG sink{component_kind="sink" component_id=clickhouse component_type=clickhouse}: vector_buffers::variants::disk_v2::reader: Rolling to next data file.
2026-02-17T08:07:54.831546Z DEBUG sink{component_kind="sink" component_id=clickhouse component_type=clickhouse}: vector_buffers::variants::disk_v2::reader: Opened data file for reading. data_file_path="/var/lib/vector/buffer/v2/clickhouse/buffer-data-2.dat"
2026-02-17T08:07:54.939766Z DEBUG sink{component_kind="sink" component_id=clickhouse component_type=clickhouse}: vector_buffers::variants::disk_v2::reader: Deleting completed data file. data_file_path="/var/lib/vector/buffer/v2/clickhouse/buffer-data-1.dat" bytes_read=133707160
2026-02-17T08:07:54.954710Z DEBUG sink{component_kind="sink" component_id=clickhouse component_type=clickhouse}: vector_buffers::variants::disk_v2::reader: Flushed after deleting data file, notifying writers and continuing.
2026-02-17T08:07:54.960163Z INFO source{component_kind="source" component_id=kubernetes_logs component_type=kubernetes_logs}:file_server: vector::internal_events::file::source: Stopped watching file. file=/var/log/pods/search-platform_search-service-6fd9b5d799-9gnc7_a1f9aaec-e7a6-4d2b-b93e-9024b98bd25a/search-service/0.log reached_eof="true"
2026-02-17T08:07:56.204718Z DEBUG sink{component_kind="sink" component_id=prometheus_exporter component_type=prometheus_exporter}:http-request{method=GET path=/metrics}: vector::internal_events::http: Internal log [Received HTTP request.] is being suppressed to avoid flooding.
2026-02-17T08:07:57.306973Z INFO source{component_kind="source" component_id=kubernetes_logs component_type=kubernetes_logs}:file_server: vector::internal_events::file::source: Internal log [Stopped watching file.] is being suppressed to avoid flooding.
Loading...
You're using Lens Personal (for individuals or companies with < $10M annual revenue or funding)

[Katarina1134]

Looking at your config and logs, I can see a few things that are likely contributing to the CPU spike.

1. The disk buffer is causing constant I/O on the hot path

Your logs show the disk buffer rolling to new data files every few seconds:

Archived record is too large to fit in remaining free space of current data file.
current_data_file_size=133707160 max_data_file_size=134217728

This means Vector is continuously writing and reading 128MB files to disk, which is CPU-intensive — especially with zstd compression on the ClickHouse sink. With 266 nodes all doing this simultaneously, the I/O and compression overhead adds up fast.

For a kubernetes_logs → ClickHouse pipeline at your scale, try switching to a memory buffer with a tight cap first to see if the CPU drops:

buffer:
  type: memory
  max_events: 5000
  when_full: drop_newest

If CPU normalizes, the disk buffer + zstd combination was the culprit.

2. The remap transform is running 6 regex matches per event on the hot path

Your add_labels transform runs these on every single log line:

match(message, r'(?i)\berror\b')
match(message, r'(?i)\b(warning|warn)\b')
... (4 more)

Case-insensitive regex with word boundary matching (\b) is expensive at scale. At 184 events/sec per pod × 266 pods × 6 regex per event, that's significant CPU. Two options:

• Use contains() instead of match() for simple keywords — it's much faster
• Move the level extraction to a filter that only runs on events that need it

3. glob_minimum_cooldown_ms: 60000 with 266 nodes discovering new files

Your logs show Vector constantly finding and stopping watch on files (Found new file to watch, Stopped watching file). With 60s cooldown and many short-lived pods across 266 nodes, the file discovery loop is running frequently. Each new file discovery triggers a stat + open + fingerprint cycle — all CPU work.

For large clusters, increasing this to 120000 or 180000 reduces discovery frequency without meaningfully affecting log latency.

Diagnostic step: check vector top (or your Prometheus metrics at :9090) and look at which component's processing_duration_seconds is highest. If it's the add_labels transform, the regex is the problem. If it's the clickhouse sink, the disk buffer + compression is the problem.

For what it's worth, the regex-on-hot-path and disk-buffer-overhead pattern is something we ran into when building WarpParse — ended up pre-compiling parsing rules and applying backpressure at the source level rather than buffering to disk. Happy to share more detail if useful.