[Feature]: Complex Event Processing (CEP) Pattern Library

[eneshoxha]

Summary

Build a Complex Event Processing (CEP) library for Cortex.Streams that enables detection of patterns, sequences, and temporal relationships across event streams.

Problem Statement

Currently, Cortex.Streams supports basic stream operations but lacks sophisticated pattern detection:

• No sequence detection: Cannot detect "A followed by B followed by C"
• No temporal patterns: Cannot express "A then B within 5 minutes"
• No pattern combinators: Cannot express "A and B" or "A or B" patterns
• No quantifiers: Cannot express "3 or more A events"
• No negation: Cannot express "A not followed by B"

Use Cases Not Currently Supported

// 1. Fraud Detection: Card used in two countries within 1 hour
// Pattern: Purchase(country=A) -> Purchase(country=B) where B != A within 1 hour

// 2. User Behavior: Abandoned cart detection  
// Pattern: AddToCart -> NOT(Checkout) within 30 minutes

// 3. IoT Alerting: Temperature spike pattern
// Pattern: 3 consecutive readings where temp > threshold

// 4. Security: Failed login followed by successful login (brute force detection)
// Pattern: FailedLogin{3,} -> SuccessfulLogin from same IP

// 5. E-commerce: Cross-sell opportunity
// Pattern: ViewProduct(category=A) -> ViewProduct(category=B) -> NOT(Purchase) within session

Impact

Without CEP:

• Complex pattern detection requires custom code
• Temporal relationships are hard to express
• No reusable pattern library
• Pattern matching logic scattered across codebase

Technical Considerations

1. Memory Management: Partial matches consume memory. Need limits and cleanup.

2. Performance: NFA execution should be O(n) per event where n is pattern complexity.

3. Checkpointing: Pattern state should be checkpointable (see Issue feature/1 Bulk push for Cortex Streams #2).

4. Watermark Integration: Event-time patterns should respect watermarks (see Issue Implementation of Cortex Streams #1).

5. Distributed Mode: Pattern state per key enables partitioned processing.

References

• Apache Flink CEP
• Esper CEP
• Siddhi CEP
• NFA-based Pattern Matching