Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,10 @@ concurrency:

jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
go-version: [1.23.x, 1.24.x]
name: Build and Test
runs-on: ubuntu-latest
steps:
Expand All @@ -33,4 +37,29 @@ jobs:
- name: Test
run: make test

- name: Test with Race Detection
run: go test -race -coverprofile=coverage.out -covermode=atomic ./...

- name: Upload coverage to Codecov (optional)
uses: codecov/codecov-action@v3
with:
file: ./coverage.out

lint:
name: Lint
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4

- name: Set up Go
uses: actions/setup-go@v5
with:
go-version-file: go.mod
cache: true

- name: Run golangci-lint
uses: golangci/golangci-lint-action@v3
with:
version: latest
args: --timeout=5m
18 changes: 18 additions & 0 deletions .github/workflows/goproxy.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
name: Go Module Proxy

on:
push:
branches: [ main, master ]

jobs:
proxy:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v4

- name: Check Go module proxy
run: |
go list -m -versions github.com/ryuux05/godex
go mod download
go mod verify
47 changes: 47 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
name: Release

on:
push:
tags:
- 'v0.1.0'

permissions:
contents: write

jobs:
release:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Set up Go
uses: actions/setup-go@v4
with:
go-version: 1.24.x

- name: Run tests
run: go test -race ./...

- name: Create release archives
run: |
# Create cross-platform binaries
mkdir -p dist

# Linux AMD64
GOOS=linux GOARCH=amd64 go build -o dist/godex-linux-amd64 ./cmd

# macOS AMD64
GOOS=darwin GOARCH=amd64 go build -o dist/godex-darwin-amd64 ./cmd

# Windows AMD64
GOOS=windows GOARCH=amd64 go build -o dist/godex-windows-amd64.exe ./cmd

- name: Create GitHub release
uses: softprops/action-gh-release@v1
with:
files: |
dist/godex-*
generate_release_notes: true
125 changes: 39 additions & 86 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,20 +5,27 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]
## [v0.1.0] - 2026-01-09

### Added
- TBD
- **GoDoc Documentation**: Comprehensive GoDoc comments for all public API functions in `pkg/godex/godex.go`
- **CI/CD Pipeline**: Complete GitHub Actions workflows for automated testing, linting, and releases
- Multi-Go version testing (1.23.x, 1.24.x)
- Race detection and coverage reporting
- golangci-lint integration
- Automated cross-platform releases on version tags
- **Go Module Proxy Validation**: Workflow to ensure module proxy compatibility

### Changed
- TBD
- **Public API Documentation**: Enhanced package documentation with quick start examples and API stability guarantees

### Fixed
- TBD

---

## [v0.1.0] - 2025-01-07
- **SQL Column Mismatch**: Fixed INSERT statement in Uniswap swap indexer to match database schema
- **Event Routing**: Corrected topic count matching for Initialize vs Swap events (3 vs 4 indexed topics)
- **Pool Lookup Logic**: Improved handling of missing pool data with RPC fallback mechanisms
- **Schema Compatibility**: Added missing columns to `uniswap_pools` table for V4 pool data
- **Topic Filtering**: Fixed Arbitrum contract address and event signature matching
- **Test Race Conditions**: Fixed data race in processor tests using atomic operations

### Added
- **Response Splitting**: Automatic handling of "response too big" errors with recursive binary search range splitting for RPC providers with size limits
Expand Down Expand Up @@ -47,88 +54,34 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

- **CLI Tool**:
- Basic scaffolding commands for new indexer projects
- Code generation utilities for ABI handling

- **Complete ERC20 Indexer Example**:
- Production-ready ERC20 transfer and approval indexer
- Docker Compose setup with PostgreSQL and Prometheus
- Custom transaction handlers with business logic
- Health checks and metrics endpoints
- Comprehensive README with deployment instructions

- **Integration Testing**:
- End-to-end tests with real PostgreSQL database
- Component interaction testing
- Error scenario simulation

- **Benchmark Testing**:
- Performance benchmarks for all core components
- Memory usage profiling
- Concurrent operation stress testing

- **Enhanced Error Handling**:
- Structured error classification (transient, permanent, reorg)
- Context-aware error propagation
- Detailed error logging with correlation IDs

- **Processor**
- Window-based HTTP log fetching with `eth_getLogs` / `eth_getBlockReceipts`
- Per-chain `ChainInfo` and `Options` (range size, confirmations, concurrency, topics, fetch mode)
- Arbiter-based ordered commit with reorg detection and rollback (`handleReorg`)
- Public `Processor.Logs(chainId)` API for consuming ordered `types.Log`

- **Sink**
- `sink.Sink` interface (`Store`, `Rollback`) for pluggable storage backends
- Postgres sink adapter (`adapters/sink/postgres`):
- Internal `chronicle_events` and `chronicle_cursors` schema
- Dual insert/COPY modes with `CopyThreshold`
- Transactional `Store` and `Rollback`
- `Handler` interface to run user-defined schema logic in the same transaction as internal writes

- **Decoder**
- `StandardDecoder` with ABI registration:
- `RegisterABI(name, abiJSON)` for named ABI sets
- `Decode(name, log)` → `*types.Event`
- `DecoderRouter` for complex multi-contract scenarios with configurable match conditions
- `types.Event` model with `Fields` map for decoded data

- **Metrics**
- `metrics.Metrics` interface:
- `IncBlocksProcessed`, `ObservedBlockLag`, `ObservedBlockFetchDuration`,
`SetIndexedHeight`, `IncSinkWrites`, `SetProcessorConcurrency`,
`IncSinkErrors`, `ObservedSinkWriteDuration`, `IncReorgs`
- Prometheus adapter (`adapters/metrics`):
- Counters/gauges/histograms for:
- `godex_block_processed_total`
- `godex_block_lag`
- `godex_block_fetched_duration_seconds`
- `godex_indexed_block_height`
- `godex_sink_events_writes_total`
- `godex_sink_events_errors_total`
- `godex_processor_concurrency`
- `godex_reorgs_total`

- **Documentation**
- `docs/processor.md`: explanation of processor flow, windows, arbiter, and reorg strategy
- `docs/metrics.md`: metrics reference and Prometheus usage
- `docs/indexer_architecture.md`: high-level architecture for Processor, Sink, Metrics, and Godex
- `docs/sink.md`: storage backend documentation
- `docs/rpc.md`: RPC client and retry logic
- `docs/decoder.md`: decoding strategies and ABI handling
- Complete README with quick start, configuration, and deployment guides

- **Cross-Chain Uniswap V4 Indexer Example**:
- Multi-chain indexing (Ethereum + Arbitrum)
- Uniswap V4 Swap and Initialize event handling
- Pool state tracking and cross-chain swap connections
- PostgreSQL persistence with optimized schemas
- Docker Compose setup for development

- **ERC20 Indexer Example**:
- Transfer event indexing with balance tracking
- Multi-contract support with address filtering
- Comprehensive PostgreSQL schema

- **PostgreSQL Sink Adapter**:
- Transactional event persistence
- Automatic table creation and schema management
- Configurable bulk insert thresholds
- Cursor state management for resumable indexing

### Changed
- **Architecture**: Refactored monolithic processor into modular components for better maintainability
- **Error Handling**: Enhanced with structured logging and context propagation
- **Testing**: Added comprehensive integration and benchmark tests
- **Documentation**: Comprehensive guides with real-world examples and deployment instructions
- **Architecture**: Modular design with clear separation of concerns
- **Error Handling**: Comprehensive error classification and retry logic
- **Performance**: Optimized concurrent processing and resource utilization

### Fixed
- Race condition in processor `isRunning` flag with proper mutex usage
- Context cancellation handling in fetch operations
- Magic number documentation and constant definitions
- Response splitting for RPC size limits
- Memory leaks in timestamp fetching and cache management
- **Reorg Detection**: Deterministic handling with proper ancestor validation
- **Context Cancellation**: Proper cleanup and resource management
- **Memory Usage**: LRU cache implementation for block hash storage

---

Expand Down
1 change: 0 additions & 1 deletion examples/erc20-indexer/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,6 @@ services:

volumes:
postgres_data:

networks:
default:
driver: bridge
Expand Down
2 changes: 1 addition & 1 deletion examples/swap-indexer/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ services:
test: ["CMD-SHELL", "pg_isready -U godex"]
interval: 5s
timeout: 5s
retries: 5
retries: 5
indexer:
build:
context: ../.. # Build from repo root
Expand Down
3 changes: 0 additions & 3 deletions pkg/core/processor/options.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,6 @@ type Options struct {
// StartBlock is the inclusive block height to begin indexing from.
// Use 0 to let the processor derive it (e.g., from a stored cursor).
StartBlock uint64
// EndBlock is an optional inclusive block height to stop indexing at.
// Use 0 to run continuously toward the moving head.
EndBlock uint64
// Confimation is range of block to wait.
// Confirmation is used to avoid most reorgs.
// Eth PoS confirmation is around 5-15 for "safe"
Expand Down
19 changes: 10 additions & 9 deletions pkg/core/processor/processor_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import (
"net/http"
"net/http/httptest"
"sync"
"sync/atomic"
"testing"
"time"

Expand Down Expand Up @@ -1059,10 +1060,10 @@ func TestMultiChainRun_Success(t *testing.T) {

func TestMultiChain_IndependentErrors(t *testing.T) {
// Ethereum server - always fails
ethCallCount := 0
var ethCallCount, polyCallCount int64
ethSrv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
ethCallCount++
atomic.AddInt64(&ethCallCount, 1)

// Always return error for Ethereum
_ = json.NewEncoder(w).Encode(map[string]any{
Expand All @@ -1077,10 +1078,9 @@ func TestMultiChain_IndependentErrors(t *testing.T) {
defer ethSrv.Close()

// Polygon server - works fine
polyCallCount := 0
polySrv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
polyCallCount++
atomic.AddInt64(&polyCallCount, 1)

var req struct {
Method string `json:"method"`
Expand Down Expand Up @@ -1199,14 +1199,16 @@ func TestMultiChain_IndependentErrors(t *testing.T) {
// Collect events from sink
polyEventCount := mockSink.GetEventCount()

finalEthCount := atomic.LoadInt64(&ethCallCount)
finalPolyCount := atomic.LoadInt64(&polyCallCount)
// Assertions
t.Logf("Run error: %v", runErr)
t.Logf("Ethereum calls: %d", ethCallCount)
t.Logf("Polygon calls: %d", polyCallCount)
t.Logf("Ethereum calls: %d", finalEthCount)
t.Logf("Polygon calls: %d", finalPolyCount)
t.Logf("Polygon events stored: %d", polyEventCount)

assert.Greater(t, ethCallCount, 0, "Ethereum should have attempted calls")
assert.Greater(t, polyCallCount, 0, "Polygon should have made calls")
assert.Greater(t, int(finalEthCount), 0, "Ethereum should have attempted calls")
assert.Greater(t, int(finalPolyCount), 0, "Polygon should have made calls")
assert.Error(t, runErr, "Should get error from Ethereum chain")
}

Expand Down Expand Up @@ -2043,4 +2045,3 @@ func TestRun_WithEnableTimestamps(t *testing.T) {
assert.Equal(t, uint64(1700000000), event1.Timestamp) // 0x65f5a000
assert.Equal(t, uint64(1700000012), event2.Timestamp) // 0x65f5a00c
}

14 changes: 14 additions & 0 deletions pkg/godex/godex.go
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,9 @@ const (
FetchModeReceipts FetchMode = processor.FetchModeReceipts
)

// NewProcessor creates a new blockchain indexing processor with the provided
// metrics collector and event sink. The processor orchestrates concurrent
// fetching, decoding, and persistence of blockchain events across multiple chains.
func NewProcessor(m Metrics, s Sink) *Processor {
return processor.NewProcessor(m, s)
}
Expand All @@ -79,10 +82,15 @@ type RPC = rpc.RPC
type HTTPRPC = rpc.HTTPRPC
type RetryConfig = rpc.RetryConfig

// NewHTTPRPC creates a new rate-limited HTTP RPC client for blockchain interactions.
// The client automatically handles request retries, exponential backoff, and
// context cancellation. Rate limiting prevents overwhelming RPC endpoints.
func NewHTTPRPC(endpoint string, rateLimit uint16, burstLimit uint16) *HTTPRPC {
return rpc.NewHTTPRPC(endpoint, rateLimit, burstLimit)
}

// DefaultRetryConfig returns the default retry configuration with sensible
// defaults for RPC request retries, including exponential backoff and jitter.
func DefaultRetryConfig() RetryConfig {
return rpc.DefaultRetryConfig()
}
Expand Down Expand Up @@ -120,10 +128,16 @@ type ReorgError = coreerrors.ReorgError
var ErrCursorNotFound = coreerrors.ErrCursorNotFound
var ErrReorgDetected = coreerrors.ErrReorgDetected

// IsRetryableError determines if an error should trigger a retry attempt.
// Returns true for transient errors like network timeouts, rate limits, or
// temporary RPC unavailability that may succeed on retry.
func IsRetryableError(err error) bool {
return coreerrors.IsRetryableError(err)
}

// IsResponseTooBigError checks if an error indicates an RPC response exceeded
// the provider's size limits. This typically triggers automatic range splitting
// in the fetcher to reduce individual request sizes.
func IsResponseTooBigError(err error) bool {
return coreerrors.IsResponseTooBigError(err)
}