Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
0d5e742
Build: override release.target=9 in mutiny module
astubbs Apr 12, 2026
e0a4a43
test: re-enable MultiInstanceRebalanceTest.largeNumberOfInstances for…
astubbs Apr 12, 2026
a340220
test: tag largeNumberOfInstances as performance, document flake rate
astubbs Apr 12, 2026
c6619a7
test: add cooperative assignor variant, deterministic unit tests, deb…
astubbs Apr 12, 2026
c3e82c0
refactor: extract ManagedPCInstance from MultiInstanceRebalanceTest
astubbs Apr 12, 2026
4b2fc7e
fix: prevent ConcurrentModificationException in ConsumerManager.poll(…
astubbs Apr 12, 2026
8b5d9cb
docs: identify silent stall root cause — numberRecordsOutForProcessin…
astubbs Apr 12, 2026
5e1b3b3
fix: adjust numberRecordsOutForProcessing on partition revoke (#857)
astubbs Apr 12, 2026
1c63d04
docs: update investigation — counter fix triggers but stall persists
astubbs Apr 12, 2026
6c42868
fix: reset pausedForThrottling on partition assignment (#857)
astubbs Apr 12, 2026
4659c87
diag: add #857 diagnostic logging to retrieveAndDistributeNewWork
astubbs Apr 12, 2026
ff7abe1
diag: identify root cause of silent stall — assignment=0 during rebal…
astubbs Apr 12, 2026
777b6d2
test: add gentleChaosRebalance test — passes 3/3, confirms rebalance …
astubbs Apr 12, 2026
761d050
cleanup: remove diagnostic logging, cooperative test passes 3/3
astubbs Apr 12, 2026
847582d
docs: update investigation — gentle chaos passes, aggressive stall un…
astubbs Apr 12, 2026
c20dfd3
docs: identify root cause — commitCommand lock contention causes dead…
astubbs Apr 12, 2026
2d434c0
fix: replace synchronized(commitCommand) with ReentrantLock.tryLock()…
astubbs Apr 12, 2026
a1fc5fb
docs: update investigation with tryLock fix results
astubbs Apr 12, 2026
fac3a66
docs: remaining 20% failure is the CME from overlapping PC lifecycles
astubbs Apr 12, 2026
967da9d
test: revert settling delay, document thread-safety investigation
astubbs Apr 12, 2026
efb5f0c
feat: add ThreadConfinedConsumer wrapper for thread-safety enforcemen…
astubbs Apr 12, 2026
4410413
fix: revert explicit consumer close, improve thread check logging
astubbs Apr 13, 2026
1397836
refactor: remove raw consumer field, enforce via ArchUnit (#857)
astubbs Apr 13, 2026
11c9929
fix: prevent duplicate run() invocations in ManagedPCInstance (#857)
astubbs Apr 13, 2026
cff79e3
test: fresh Kafka container for performance tests, prevent duplicate …
astubbs Apr 13, 2026
c25711b
fix: move started flag to start() to prevent double-submission (#857)
astubbs Apr 13, 2026
856d4ff
diag: add state dump on stall detection, expose started flag (#857)
astubbs Apr 13, 2026
33086ad
diag: epoch mismatch theory disproven — records not being polled, not…
astubbs Apr 13, 2026
499662f
diag: add assignedPartitions to state dump — confirms assignment=0 du…
astubbs Apr 13, 2026
f0ce51a
fix: always update assignment cache + add trace logging everywhere (#…
astubbs Apr 13, 2026
cfda431
fix: revert resetKafkaContainer calls + fix init cache + checkGroupId…
astubbs Apr 13, 2026
192ac14
docs: Kafka eager rebalance protocol is root cause of remaining stall
astubbs Apr 13, 2026
e6c9822
test: switch largeNumberOfInstances to CooperativeStickyAssignor
astubbs Apr 13, 2026
fa9b08d
docs: update largeNumberOfInstances javadoc with accurate findings
astubbs Apr 13, 2026
65112a7
fix: non-blocking stopAsync() for chaos monkey to prevent stall (#857)
astubbs Apr 13, 2026
c054ca9
docs: 90% pass rate achieved — document findings and set 80% acceptan…
astubbs Apr 13, 2026
dcd98b0
fix: PCMetrics memory leak — duplicate meter registrations (#859)
astubbs Apr 13, 2026
55af099
Merge branch 'bugs/857-paused-consumption-multi-consumers-bug' into b…
astubbs Apr 13, 2026
b25790c
test: regression tests for #859 PCMetrics memory leak
astubbs Apr 13, 2026
8518d19
docs(agents): add working directory and full test suite rules
astubbs Apr 13, 2026
d9d95f7
fix: flaky WorkManagerOffsetMapCodecManagerTest.largeOffsetMap
astubbs Apr 13, 2026
be1abbe
ci: trigger CI run
github-actions[bot] Apr 15, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
113 changes: 113 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
# Parallel Consumer - Agent Context

Project context for AI coding agents (Claude Code, Copilot, Cursor, etc.).

## Overview

Parallel Consumer is a Java library that enables concurrent message processing from Apache Kafka with a single consumer, avoiding the need to increase partition counts. It maintains ordering guarantees (by partition or key) while processing messages in parallel.

This is a community-maintained fork of `confluentinc/parallel-consumer` (the upstream is no longer actively maintained), published to Maven Central as `io.github.astubbs.parallelconsumer`.

## Build Requirements

- **JDK 17** (required - the project uses Jabel to compile Java 17 source to Java 8 bytecode)
- **Docker** (required for integration tests - TestContainers spins up Kafka brokers)
- **Maven** via wrapper (`./mvnw`) - do not use system Maven

## How to Build

```bash
# Quick local build (compile + unit tests)
bin/build.sh

# Full CI build with all tests (unit + integration)
bin/ci-build.sh

# Full CI build against a specific Kafka version
bin/ci-build.sh 3.9.1
```

## Module Structure

| Module | Purpose |
|--------|---------|
| `parallel-consumer-core` | Core library - consumer, producer, offset management, sharding |
| `parallel-consumer-vertx` | Vert.x integration for async HTTP |
| `parallel-consumer-reactor` | Project Reactor integration |
| `parallel-consumer-mutiny` | SmallRye Mutiny integration (Quarkus) |
| `parallel-consumer-examples` | Example implementations for each module |

## Key Architecture Decisions

- **Jabel cross-compilation**: Source is Java 17, bytecode targets Java 8 via Jabel annotation processor. This means `--release 8` is set in the compiler plugin, which restricts available APIs to Java 8 surface. The Mutiny module overrides this to `--release 9` because Mutiny uses `java.util.concurrent.Flow` (Java 9+).
- **Offset encoding**: Custom offset map encoding (run-length, bitset) stored in Kafka commit metadata for tracking in-flight messages.
- **Sharding**: Messages are distributed to processing shards by key or partition for ordering guarantees.

## Testing

- **Unit tests**: `mvn test` / surefire plugin. Source in `src/test/java/`.
- **Integration tests**: `mvn verify` / failsafe plugin. Source in `src/test-integration/java/`. Uses TestContainers with `confluentinc/cp-kafka` Docker image.
- **Test exclusion patterns**: `**/integrationTest*/**/*.java` and `**/*IT.java` are excluded from surefire, included in failsafe.
- **Kafka version matrix**: CI tests against multiple Kafka versions via `-Dkafka.version=X.Y.Z`.
- **Performance tests**: Tagged `@Tag("performance")` and excluded from regular CI by default. They run on a self-hosted runner via `.github/workflows/performance.yml` (see `docs/SELF_HOSTED_RUNNER.md`). Run locally with `bin/performance-test.sh` (or `bin/performance-test.cmd` on Windows). Override the test group selection with Maven properties: `-Dincluded.groups=performance` to run only perf, `-Dexcluded.groups=` to run everything.

## Known Issues

- **Mutiny module**: Has a `release.target=9` override in its pom.xml because Mutiny's `Multi` implements `java.util.concurrent.Flow.Publisher` which is not available with `--release 8`.

## Development Rules

- **Working directory**: This is a multi-module Maven project. All `./mvnw` and `git` commands MUST run from the **project root** (the directory containing `pom.xml` and `./mvnw`). Never `cd` into a submodule and run Maven from there — use `-pl <module>` to target a specific module instead. If you need to check the current directory, run `pwd` first. The `cd` command does NOT persist between tool calls.
- **Dependency injection**: Always wire new components through `PCModule` (and `PCModuleTestEnv` for tests). Don't bypass the DI by storing direct references to components.
- **Reuse test infrastructure**: Before creating new test utilities, check existing harnesses: `AbstractParallelEoSStreamProcessorTestBase`, `BrokerIntegrationTest`, `KafkaClientUtils`, `ManagedPCInstance`, `ModelUtils`, `LongPollingMockConsumer`, `ProgressTracker`, `PCModuleTestEnv`.
- **Never weaken test assertions**: Tests are critical for this project. When modifying test error handling, classify exceptions (whitelist expected ones) rather than ignoring them. Integration/load tests serve as both specific scenario tests AND general stability canaries.
- **License check**: Always pass `-Dlicense.skip` to Maven commands unless intentionally formatting headers. The plugin breaks in git worktrees.
- **Run full test suite before pushing**: After significant production code changes, run: `./mvnw clean verify -Dlicense.skip -Dexcluded.groups=performance` from the project root. Don't push until this passes.

## Code Style

- **Lombok**: Used extensively (builders, getters, logging). IntelliJ Lombok plugin required.
- **EditorConfig**: Enforced via `.editorconfig` - 4-space indent for Java, 120 char line length.
- **License headers**: Managed by `license-maven-plugin` (Mycila). See "License headers" section below.
- **Google Truth**: Used for test assertions alongside JUnit 5 and Mockito.

## License headers

The Mycila `license-maven-plugin` enforces a Confluent copyright header on all source files. It uses git-derived years via `${license.git.copyrightYears}`.

**Skipping the check** (for any Maven goal):
```
./mvnw <goal> -Dlicense.skip
```

**When to skip:**
- Running builds inside a git worktree — the git-years lookup fails with `Bare Repository has neither a working tree, nor an index`
- Local iteration where you don't want years auto-bumped on touched files
- Any command other than the canonical `mvn install` flow when copyright drift would create noise in `git status`

The default behavior on macOS dev machines is `format` mode (auto-fixes headers) via the `license-format` profile (auto-activated). The `ci` profile flips this to `check` mode (fails the build on drift). Both `bin/build.sh` and `bin/ci-build.sh` already pass `-Dlicense.skip` for this reason.

**When NOT to skip:**
- You're intentionally running `mvn license:format` to update headers
- You want to verify CI's check would pass before pushing

## CI

- **`.github/workflows/maven.yml`** — Build and test on every push/PR. Push uses default Kafka version, PRs run the full version matrix. Includes concurrency cancellation.
- **`.github/workflows/publish.yml`** — Publishes to Maven Central on every push to `master`. The pom.xml version is the source of truth: `-SNAPSHOT` versions deploy as snapshots, non-snapshot versions deploy as full releases (and create a git tag + GitHub release).
- **`.semaphore/`** — Legacy Confluent internal CI/release pipelines, retained but inactive on the fork.

## Releasing

The pom.xml version drives publishing — there is no `maven-release-plugin` dance.

**Cut a release:**
1. Open a PR removing `-SNAPSHOT` from `<version>` in the parent pom (e.g. `0.6.0.0-SNAPSHOT` → `0.6.0.0`)
2. Merge it to master → CI publishes to Maven Central, tags `v0.6.0.0`, creates a GitHub release
3. Open another PR bumping to the next snapshot (e.g. `0.6.0.1-SNAPSHOT`) and merge

**Required GitHub repo secrets** for `publish.yml`:
- `MAVEN_CENTRAL_USERNAME` — Sonatype Central Portal token username
- `MAVEN_CENTRAL_PASSWORD` — Sonatype Central Portal token password
- `MAVEN_GPG_PRIVATE_KEY` — Armored GPG private key for signing artifacts
- `MAVEN_GPG_PASSPHRASE` — Passphrase for the GPG key
Loading
Loading