Add client-side regex filtering for all adapters

[maximelb]

Summary

Adds client-side filtering with type-safe configuration and GJSON support for efficient JSON field filtering. Users can specify patterns to filter out events/logs before they're sent to the cloud, reducing ingestion costs and network traffic for unwanted data like health checks, debug logs, and test users.

New in this version: Added filter mode support (exclude/include) allowing users to either filter OUT matching messages or keep ONLY matching messages.

Architecture

Core Components

FilterEngine (utils/filter.go)

• Type-safe FilterPattern configuration with validation
• Two pattern types: "regex" and "gjson" (JSON field filtering)
• Two filter modes: "exclude" (default) and "include"
• Compiles patterns once at initialization for performance
• Thread-safe statistics tracking using atomic operations
• Background goroutine logs statistics every 5 minutes
• Per-pattern match tracking
• Graceful shutdown with final stats report

FilteredClient (utils/filtered_client.go)

• Implements Shipper interface for transparent wrapping
• Intercepts Ship() calls to filter before transmission
• Maintains full compatibility with existing adapter code
• Zero changes required to adapter logic after initialization

Configuration

• Added Filters []utils.FilterPattern field to all 40 adapter configs
• Added FilterMode utils.FilterMode field to all 40 adapter configs
• Supports YAML/JSON configuration with explicit type discriminator
• Optional fields (no overhead when not configured)
• Built-in validation at initialization

Filter Modes

Exclude Mode (Default)

Messages matching any pattern are filtered OUT. Non-matching messages pass through.

# Filter OUT debug and trace logs
filters:
  - type: gjson
    path: "level"
    pattern: "^(DEBUG|TRACE)$"
filter_mode: exclude  # optional, this is the default

Include Mode

Only messages matching at least one pattern are allowed through. Non-matching messages are filtered OUT.

# Keep ONLY INFO, WARN, and ERROR logs
filters:
  - type: gjson
    path: "level"
    pattern: "^(INFO|WARN|ERROR)$"
filter_mode: include

Changes

• ✅ Enhanced: utils/filter.go - Added FilterMode support with exclude/include semantics
• ✅ Enhanced: utils/filtered_client.go - Updated to pass FilterMode to FilterEngine
• ✅ Enhanced: utils/filter_test.go - Comprehensive test suite (800+ lines, 50+ tests)
• ✅ New: utils/FILTERING_BENCHMARKS.md - Performance analysis and recommendations
• ✅ New: utils/IMPLEMENTATION_SUMMARY.md - Complete implementation guide
• ✅ Updated: All 40 adapter configs with Filters and FilterMode fields
• ✅ Updated: All adapter constructors to pass FilterMode to NewFilteredClient
• ✅ Updated: go.mod/go.sum with gjson v1.18.0 dependency and latest deps from master

Features

Filter Mode Support

• ✅ Exclude mode (default): Filter out messages matching any pattern
• ✅ Include mode: Keep only messages matching at least one pattern
• ✅ Empty mode defaults to "exclude" for backward compatibility
• ✅ Invalid mode values are rejected with clear error messages

Type-Safe Configuration

• ✅ Explicit FilterPattern struct with type discriminator
• ✅ Validation for all pattern fields at initialization
• ✅ Clear error messages for invalid configurations
• ✅ Self-documenting configuration syntax

GJSON-Based JSON Filtering

• ✅ Efficient JSON field extraction without full regex parsing
• ✅ Nested field access: user.profile.role
• ✅ Array indexing: items.0.id
• ✅ Conditional queries: users.#(age>45).email
• ✅ Full GJSON syntax support (see https://github.com/tidwall/gjson/blob/master/SYNTAX.md)

Regex Filtering

• ✅ Full regex support for text and JSON payloads
• ✅ Matches against TextPayload and JsonPayload (marshaled to JSON)
• ✅ Flexible pattern matching with anchors, character classes, etc.

Observability

• ✅ Per-pattern statistics tracking
• ✅ Automatic stats reporting every 5 minutes
• ✅ JSON marshal failure tracking (no double-counting)
• ✅ Debug logging for each filtered item
• ✅ Final stats report on shutdown

Thread Safety

• ✅ Atomic operations for all counters
• ✅ RWMutex protection for shared state
• ✅ Idempotent Close() using sync.Once
• ✅ Race detector clean

Example Configurations

Exclude Mode - Filter OUT Unwanted Data

zendesk:
  client_options:
    identity:
      installation_key: "..."
      oid: "..."
  api_token: "..."
  zendesk_domain: "company.zendesk.com"
  zendesk_email: "admin@company.com"
  
  # Filter OUT debug logs, test users, and system actions
  filters:
    - type: gjson
      path: "action"
      pattern: "^debug\\."
    - type: gjson
      path: "actor.email"
      pattern: ".*@test\\.example\\.com$"
    - type: gjson
      path: "source_type"
      pattern: "^system$"
  filter_mode: exclude  # default, can be omitted

Include Mode - Keep ONLY Specific Data

syslog:
  client_options:
    identity:
      installation_key: "..."
      oid: "..."
  listen_port: 514
  
  # Keep ONLY security-related logs
  filters:
    - type: gjson
      path: "facility"
      pattern: "^(auth|authpriv|security)$"
    - type: gjson
      path: "severity"
      pattern: "^(alert|crit|err|warning)$"
  filter_mode: include

Advanced GJSON Queries

filters:
  # Filter users over 45
  - type: gjson
    path: "users.#(age>45).email"
    pattern: ".*@company\\.com"
  
  # Filter first array element
  - type: gjson
    path: "events.0.severity"
    pattern: "^(high|critical)$"
  
  # Filter nested metadata
  - type: gjson
    path: "request.metadata.user_agent"
    pattern: ".*bot.*"

Example Output

[INFO] Filter engine initialized with 3 patterns (1 regex, 2 gjson) in exclude mode
[DEBUG] Filtered: matched gjson(path="level", pattern="DEBUG") in JsonPayload
[DEBUG] Filtered: matched gjson(path="user.email", pattern=".*@test\\.example\\.com$") in JsonPayload
[INFO] Filter stats (last 5m0s): checked=10000, filtered=342 (3.42%)
  - Pattern 0 [gjson] path="level", pattern="DEBUG": 156 matches
  - Pattern 1 [gjson] path="user.email", pattern=".*@test\\.example\\.com$": 124 matches
  - Pattern 2 [regex] pattern="health-?check": 62 matches
[INFO] Final filter stats: checked=50000, filtered=1710 (3.42%)

Performance

Benchmarks (see utils/FILTERING_BENCHMARKS.md for details):

• GJSON filtering: 9.5µs per operation
• Regex filtering: 5.8µs per operation on JSON
• Performance in context: <1% overhead vs typical 100-1000ms network/API latency
• Verdict: Usability benefits justify minimal performance cost

Testing

• ✅ 50+ tests covering all functionality including filter modes
• ✅ Specific tests for exclude and include mode behavior
• ✅ Test for marshal failure counting (no double-counting)
• ✅ Concurrency test: 50 goroutines × 100 messages
• ✅ Comprehensive benchmark suite (6 scenarios)
• ✅ All adapters build successfully
• ✅ All tests pass

Impact

• No breaking changes - Fully backward compatible
• Opt-in feature - Only activates when filters are configured
• Production-ready - Thread-safe, tested, documented
• Well-documented - Comprehensive docs and examples

Documentation

• utils/filter.go - Extensive godoc with FilterMode and GJSON syntax examples
• utils/FILTERING_BENCHMARKS.md - Performance analysis and recommendations
• utils/IMPLEMENTATION_SUMMARY.md - Complete implementation guide with examples

🤖 Generated with Claude Code