Skip to content

Conversation

@Savid
Copy link
Member

@Savid Savid commented Jan 22, 2026

Switch ClickHouse client from HTTP JSON interface to ch-go native binary protocol for improved performance and memory efficiency.

ClickHouse Client Changes:

  • Replace http.Client with ch-go connection pool (chpool.Pool)
  • Add Do() and Acquire() methods for native query execution
  • Update config to support native protocol options (compression, pool settings)
  • Change connectivity test from HTTP query to native Ping()
  • Add retry logic with exponential backoff for transient failures
  • Use chpool.Dial() for fail-fast connection validation
  • Add Start() idempotency guard to prevent goroutine leaks
  • Add configurable MaxRetries and RetryBaseDelay settings

Structlog Processor Simplification:

  • Remove batch_manager.go - channel-based batching no longer needed
  • Remove big_transaction_manager.go - handled by ch-go streaming
  • Remove clickhouse_time.go - use standard time handling
  • Add columns.go for ch-go columnar data preparation
  • Rewrite transaction processing to use ch-go OnInput streaming
  • Simplify from producer/consumer pattern to direct columnar inserts

Testing Infrastructure:

  • Add miniredis for in-memory Redis unit tests (no Docker required)
  • Add testcontainers for Redis/ClickHouse integration tests
  • Create internal/testutil package with test helpers
  • Replace skipIfRedisUnavailable() pattern with miniredis
  • Split CI workflow into unit-tests and integration-tests jobs

Benefits:

  • Native binary protocol is faster than HTTP JSON
  • Columnar streaming reduces memory allocations
  • Simplified code with fewer goroutines and channels
  • Tests always run (no skip patterns) with proper isolation
  • Improved resilience with automatic retry for transient errors

Savid and others added 3 commits January 22, 2026 16:57
Switch ClickHouse client from HTTP JSON interface to ch-go native binary
protocol for improved performance and memory efficiency.

ClickHouse Client Changes:
- Replace http.Client with ch-go connection pool (chpool.Pool)
- Add Do() and Acquire() methods for native query execution
- Update config to support native protocol options (compression, pool settings)
- Change connectivity test from HTTP query to native Ping()
- Add retry logic with exponential backoff for transient failures
- Use chpool.Dial() for fail-fast connection validation
- Add Start() idempotency guard to prevent goroutine leaks
- Add configurable MaxRetries and RetryBaseDelay settings

Structlog Processor Simplification:
- Remove batch_manager.go - channel-based batching no longer needed
- Remove big_transaction_manager.go - handled by ch-go streaming
- Remove clickhouse_time.go - use standard time handling
- Add columns.go for ch-go columnar data preparation
- Rewrite transaction processing to use ch-go OnInput streaming
- Simplify from producer/consumer pattern to direct columnar inserts

Testing Infrastructure:
- Add miniredis for in-memory Redis unit tests (no Docker required)
- Add testcontainers for Redis/ClickHouse integration tests
- Create internal/testutil package with test helpers
- Replace skipIfRedisUnavailable() pattern with miniredis
- Split CI workflow into unit-tests and integration-tests jobs

Benefits:
- Native binary protocol is faster than HTTP JSON
- Columnar streaming reduces memory allocations
- Simplified code with fewer goroutines and channels
- Tests always run (no skip patterns) with proper isolation
- Improved resilience with automatic retry for transient errors

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…hods

Replace generic JSON-based QueryOne/QueryMany with type-specific native
column binding methods to fix ch-go type validation errors.

Changes:
- Add QueryUInt64 for single UInt64 value queries
- Add QueryMinMaxUInt64 for min/max aggregate queries
- Update state manager to use new typed methods
- Remove json_number.go (types no longer needed)
- Update example_config.yaml for native protocol format
- Update tests and mocks for new interface
Savid and others added 2 commits January 23, 2026 12:20
This change ensures the application waits for ClickHouse to become
available at startup instead of failing after a fixed number of retries.

Changes:
- Add RetryMaxDelay config option (default: 10s) to cap exponential backoff
- Move dial from New() to Start() - clients are created without connecting
- State manager retries infinitely with capped backoff when starting clients
- Processor manager retries infinitely when starting processors
- Server waits for state manager to be ready before starting processors
- Fix idempotency bug where Start() would skip retry after failure

The startup order is now:
1. State manager connects to ClickHouse (infinite retry)
2. Once ready, processors start (also with infinite retry)

This allows the app to gracefully wait for ClickHouse during initial
deployment or restarts without crashing.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove context parameter from New() calls (signature changed)
- Add explicit toUInt64() casts in test queries for type strictness

The ch-go native protocol is strict about types, so literal values
like SELECT 42 return UInt8 instead of UInt64. Tests need explicit
casts while production code queries actual UInt64 columns.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@Savid Savid merged commit df6edc9 into master Jan 23, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants