Skip to content

feat: add filter transformation for TimestampBasedKeyGenerator partition pruning#462

Merged
xushiyan merged 1 commit intoapache:mainfrom
yunchipang:286-timestamp-based-keygen
Mar 20, 2026
Merged

feat: add filter transformation for TimestampBasedKeyGenerator partition pruning#462
xushiyan merged 1 commit intoapache:mainfrom
yunchipang:286-timestamp-based-keygen

Conversation

@yunchipang
Copy link
Copy Markdown
Contributor

@yunchipang yunchipang commented Oct 9, 2025

Description

Implements filter transformation for TimestampBasedKeyGenerator to enable partition pruning when user filters reference timestamp columns but partitions are organized by date components (year, month, day, hour).

  • Add keygen module with KeyGeneratorFilterTransformer trait and TimestampBasedKeyGenerator
  • Support all 6 timestamp types: UNIX_TIMESTAMP, EPOCHMILLISECONDS, EPOCHMICROSECONDS, DATE_STRING, SCALAR, MIXED
  • Support input/output timezone configuration with proper fallback chain
  • Support legacy hoodie.deltastreamer.keygen.timebased.* config prefix
  • Support configurable scalar time units (NANOSECONDS through DAYS)
  • Detect timestamp-based keygen via both hoodie.table.keygenerator.class (v6) and hoodie.table.keygenerator.type (v8+)
  • Transform equality filters to multi-field partition filters (year, month, day, hour)
  • Transform range filters (>, >=, <, <=) to inclusive bounds on the coarsest partition field
  • Reject unsupported not-equal (!=) filters with a clear error message

Closes #286

How are the changes test-covered

  • Automated tests (unit and/or integration tests)

@yunchipang yunchipang requested a review from xushiyan as a code owner October 9, 2025 17:55
@xushiyan xushiyan force-pushed the 286-timestamp-based-keygen branch from e25b427 to 42e2304 Compare March 20, 2026 19:58
@xushiyan xushiyan changed the title feat: add filter transformation for TimestampBasedKeyGenerator partition pruning feat: add filter transformation for TimestampBasedKeyGenerator partition pruning Mar 20, 2026
@xushiyan xushiyan added this to the release-0.5.0 milestone Mar 20, 2026
@xushiyan xushiyan merged commit 673066f into apache:main Mar 20, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Handle partition filter properly for timestamp based keygen

2 participants