codeQ

Reactive scheduling and completion system backed by persistent queues in KVRocks.

This repository contains the core runtime, HTTP API wiring, and a Helm chart for small clusters. The production service wrapper lives at: https://github.com/codecompany/codeq-service

Links

• GitHub: https://github.com/osvaldoandrade/codeq
• Issues: https://github.com/osvaldoandrade/codeq/issues
• Specs index: docs/README.md

Why codeQ

codeQ provides:

• Persistent queues on KVRocks (Redis protocol).
• Pull-based worker claims with leases.
• Multi-tenant queue isolation with automatic tenant ID extraction from JWT claims.
• NACK + backoff + delayed queues.
• DLQ for tasks that exceed maxAttempts.
• Result storage and optional callbacks (webhooks).
• Worker auth via JWT (JWKS), producer auth via Tikti access tokens (JWKS).
• Official SDKs for Java and Node.js/TypeScript with framework integrations.
• Distributed tracing with OpenTelemetry support for end-to-end observability.
• Optimized performance: O(1) queue operations with pipelined lease repair for low-latency claims even under high load.
• Load testing framework: Comprehensive k6 scenarios and Go benchmarks for performance validation and regression testing.

Get started

Local Development with Docker Compose (Recommended)

The quickest way to try codeQ locally:

git clone https://github.com/osvaldoandrade/codeq
cd codeq
docker compose up -d

This starts KVRocks, the codeQ API server, and seeds example tasks. Access at http://localhost:8080.

See Local Development Guide for details on hot reload, testing, and observability.

Install CLI (macOS/Linux/Windows via Git Bash)

curl -fsSL https://raw.githubusercontent.com/osvaldoandrade/codeq/main/install.sh | sh

Requires git and go.

Install CLI via npm (npmjs)

npm i -g @osvaldoandrade/codeq
codeq --help

Upgrade:

npm i -g @osvaldoandrade/codeq@latest

Use SDKs for Java and Node.js

Integrate codeQ into your microservices with official SDKs:

Java (Spring Boot, Quarkus, Micronaut):

<dependency>
    <groupId>io.codeq</groupId>
    <artifactId>codeq-sdk-java</artifactId>
    <version>1.0.0</version>
</dependency>

Node.js/TypeScript (Express, NestJS):

npm install @codeq/sdk

📚 SDK Documentation:

• SDK Overview & Quick Start
• Java Integration Guide
• Node.js Integration Guide
• Example Applications

1) Helm (small cluster)

The chart in this repo deploys codeQ and, by default, a single-node KVRocks instance.

git clone https://github.com/osvaldoandrade/codeq
cd codeq

helm install codeq ./helm/codeq \
  --set secrets.enabled=true \
  --set secrets.webhookHmacSecret=YOUR_SECRET \
  --set config.identityServiceUrl=https://your-auth-server.com \
  --set config.workerJwksUrl=https://your-jwks \
  --set config.workerIssuer=https://issuer

Disable embedded KVRocks and point to your own:

helm install codeq ./helm/codeq \
  --set kvrocks.enabled=false \
  --set config.redisAddr=your-kvrocks:6666

2) Service runtime

The API server and Dockerfile are in the service repo: https://github.com/codecompany/codeq-service

That repo consumes this module and exposes the HTTP API.

Quick API flow

Create a task (producer token):

curl -X POST http://localhost:8080/v1/codeq/tasks \
  -H 'Authorization: Bearer <producer-token>' \
  -H 'Content-Type: application/json' \
  -d '{"command":"GENERATE_MASTER","payload":{"jobId":"j-123"},"priority":3}'

Claim a task (worker JWT):

curl -X POST http://localhost:8080/v1/codeq/tasks/claim \
  -H 'Authorization: Bearer <worker-token>' \
  -H 'Content-Type: application/json' \
  -d '{"commands":["GENERATE_MASTER"],"leaseSeconds":120,"waitSeconds":10}'

Submit result:

curl -X POST http://localhost:8080/v1/codeq/tasks/<id>/result \
  -H 'Authorization: Bearer <worker-token>' \
  -H 'Content-Type: application/json' \
  -d '{"status":"COMPLETED","result":{"ok":true}}'

Specs and docs

Start here: docs/README.md

Key references:

• Getting Started Tutorial: docs/00-getting-started.md - Start here for your first experience with codeQ
• Overview: docs/01-overview.md - System goals and design principles
• Architecture: docs/03-architecture.md - System components and multi-tenant architecture
• Security: docs/09-security.md - Authentication, authorization, and tenant isolation
• HTTP API: docs/04-http-api.md - Complete API reference
• CLI Reference: docs/15-cli-reference.md - CLI command documentation
• SDK Integration: sdks/README.md - Official Java and Node.js SDKs
  • Java Integration - Spring Boot, Quarkus, Micronaut
  • Node.js Integration - Express, NestJS, React
• Examples: examples/ - Working examples with Java and Node.js frameworks
• Developer Guide: docs/21-developer-guide.md - Contributing and internal architecture
• Queue model: docs/05-queueing-model.md - Queue semantics
• Storage layout: docs/07-storage-kvrocks.md - KVRocks data structures
• Backoff and retries: docs/11-backoff.md - Retry logic
• Webhooks: docs/12-webhooks.md - Push notifications
• Configuration: docs/14-configuration.md - Config reference
• Operations: docs/10-operations.md - Metrics, health checks, and tracing
• Workflows: docs/16-workflows.md - GitHub Actions automation
• Performance: docs/17-performance-tuning.md - Optimization guide
• Load Testing: docs/26-load-testing.md - Load testing framework and benchmarks
• Testing: docs/19-testing.md - Test coverage and strategy
• Authentication Plugins: docs/20-authentication-plugins.md - Plugin system for custom auth
• Persistence Plugins: docs/26-persistence-plugin-system.md - Pluggable storage backends (Redis, Memory, and more)
• Design Documents:
  • docs/24-queue-sharding-hld.md - Queue sharding high-level design
  • docs/25-plugin-architecture-hld.md - Plugin architecture for persistence and auth
• Migration: docs/migration.md - Upgrade guide
• Contributing: CONTRIBUTING.md - Contribution guidelines

Repo layout

• pkg/: public packages (app, config, domain)
• internal/: controllers, middleware, services, repositories, providers
• helm/codeq: Helm chart
• docs/: full specification

License

MIT. See LICENSE.