Filter Steps Reference

Filter Steps Reference

This page provides a detailed reference for all 12 transformation steps available in the EventProcessor parsing pipeline.

1. json

Parses a field containing a JSON string and maps its keys to fields within the log.* namespace.

• Required: source (The field containing the JSON string, e.g., raw).
• Example:

  - json: 
      source: raw
      where: exists("raw")

2. rename

Maps existing fields to new names. Supports renaming multiple fields to a single target or vice versa, though typically used for normalization.

• Required: from (Array of source fields), to (Target field name).
• Example:

  - rename:
      from: [log.src_ip, log.source_address]
      to: origin.ip

3. cast

Converts field types to ensure correct indexing and correlation.

• Required: fields (Array), to (Target type).
• Supported Types: int, float, string, bool, []string.
• Example:

  - cast:
      fields: [origin.port, target.port]
      to: int

4. delete

Removes fields from the log to optimize storage and cleanup temporary processing data.

• Required: fields (Array).
• Example:

  - delete: 
      fields: [raw, temporary_meta]
      where: equals("actionResult", "success")

5. grok

Uses Go-template-based pattern matching for unstructured text. Since patterns are now globally integrated into the SDK, you can use standard aliases like {{.ipv4}} here and in other regex-capable steps (like trim).

• Required: source (Defaults to raw), patterns (List of { fieldName, pattern }).
• Extensibility: Users can add or modify standard patterns through the UTMStack WebUI.

Default Standard Patterns

Alias	Description	Example / Match
{{.ipv4}}	IPv4 address	192.168.1.1
{{.ipv6}}	IPv6 address	2001:0db8:85a3:0000:0000:8a2e:0370:7334
{{.hostname}}	Hostname	server-01.local
{{.domain}}	Domain server	example.com
{{.email}}	Email address	user@example.com
{{.uuid}}	UUID values	550e8400-e29b-41d4-a716-446655440000
{{.integer}}	Signed or unsigned numbers	0, 54, +23, -11
{{.word}}	Complete words (can contain _, -)	event_log-01
{{.greedy}}	Full string (matches everything)	.*
{{.data}}	Matches until the next pattern	.*?
{{.space}}	One or more spaces	\s+
{{.notSpace}}	One or more non-spaces	\S+
{{.commonMacAddr}}	Common MAC address (colon or dash)	00:1A:2B:3C:4D:5E
{{.winMacAddr}}	Windows MAC address (dash)	00-1A-2B-3C-4D-5E
{{.ciscoMacAddr}}	CISCO MAC address	001a.2b3c.4d5e
{{.syslogDate}}	Syslog date format	Jun 16 12:34:56
{{.time}}	H24:mm:SS (with optional ms)	18:30:05.123
{{.hour}}	H24 hour format	07, 18, 23
{{.minute}}	mm minute format	02, 10, 59
{{.seconds}}	SS (with optional ms)	05.450
{{.iso8601Timezone}}	ISO8601 Timezone	Z, +05:00
{{.year}}	Year (1000-9999)	2024
{{.monthName}}	Month name (full or abbreviated)	January, Feb, marz
{{.monthNumber}}	Month number (01-12)	01, 10
{{.monthDay}}	Day of month (1-31)	01, 14, 31
{{.day}}	Day name (full or abbreviated)	Monday, Mon

Example:

- grok:
    source: log.message
    patterns:
      - fieldName: origin.ip
        pattern: '{{.ipv4}}'
      - fieldName: user
        pattern: 'User: {{.word}}'

6. kv (Key-Value)

Extracts key-value pairs from a string field.

• Required: source, fieldSplit (Separator between pairs), valueSplit (Separator between key and value).
• Example:

  - kv: 
      source: raw
      fieldSplit: " " 
      valueSplit: "="
      where: contains("raw", "=")

7. trim

Cleans strings by removing prefixes, suffixes, or matching patterns. With the latest SDK, the regex function supports global standard patterns (e.g., {{.int}}).

• Required: fields (Array), function (prefix, suffix, substring, regex).
• Optional: substring (The string or regex to remove).
• Example:

  - trim: 
      function: regex
      substring: "ID: {{.int}}"
      fields: [log.message]

8. add

Injects a new fixed string field into the log.

• Required: function: 'string', params: { key: "field_name", value: "fixed_value" }.
• Example:

  - add: 
      function: string
      params: 
        key: category
        value: security

9. reformat

Converts field formats, primarily used for standardizing timestamps.

• Required: fields (Array), function: time, fromFormat, toFormat.
• Example:

  - reformat:
      fields: [deviceTime]
      function: time
      fromFormat: 'Jan 02 15:04:05'
      toFormat: '2006-01-02T15:04:05Z'

10. csv

Parses comma-separated values from a string field.

• Required: source, separator, headers (Array of target field names).
• Example:

  - csv:
      source: raw
      separator: ","
      headers: [id, user, action, result]

11. dynamic

Calls an external gRPC plugin for enrichment or complex processing.

• Required: plugin (The plugin name), params (Map of key-value parameters).
• Example:

  - dynamic:
      plugin: com.utmstack.geolocation
      params: 
        source: origin.ip
        destination: origin.geolocation
      where: exists("origin.ip")

12. drop

Discards the log immediately, halting any further processing in the current or subsequent pipelines.

• Required: where (A CEL condition that must be met to drop the log).
• Example:

  - drop:
      where: equals("origin.ip", "127.0.0.1")