Outboxify

Universal Polyglot Transactional Outbox Engine
Dual-Write Consistency with Sub-Millisecond Fast-Path Dispatch for Java, TypeScript, and Python.

---

Table of Contents

• Overview
• Key Features
• Architecture & How It Works
• Database Dialects Matrix
• Polyglot Quickstart
  • Java (Spring Boot 4.x / JDK 21+)
  • Node.js / TypeScript (NestJS, Prisma, TypeORM)
  • Python (FastAPI, SQLAlchemy)
• Configuration Reference
• Runnable Examples
• Building & Testing
• Contributing
• Security
• License

---

Overview

When microservices need to modify a database and emit events to a message broker (such as Apache Kafka), doing both in separate network calls creates the classic dual-write distributed consistency problem:

• If the database commit succeeds but the message broker publish fails (e.g. network partition), events are lost.
• If the broker publish succeeds before the database commits, downstream consumers process phantom data if the database transaction rolls back.

Outboxify solves dual-write consistency by persisting outbox records inside the same ACID transaction as your business data, guaranteeing at-least-once delivery with zero dual-write data loss.

Outboxify is designed from the ground up for modern enterprise workloads:

• Sub-Millisecond Fast-Path: Dispatches messages immediately upon transaction commit using post-commit synchronization hooks without waiting for poller intervals.
• Resilient Slow-Path Poller: A background poller recovers and dispatches any unprocessed messages after outages or node crashes.
• Autonomous Watchdog Reaper: Detects and un-sticks orphaned PROCESSING rows exceeding timeouts with exponential backoff and dead-letter handling.
• Polyglot & Framework-Free Core: Standardized state machines and JSON schemas with native integrations for Spring Boot, NestJS, Prisma, TypeORM, FastAPI, and SQLAlchemy.

---

Key Features

• ⚡ Zero-Latency Post-Commit Hook: Instant event dispatch upon transaction commit.
• 🛡️ Guaranteed Rollback Safety: Events are never dispatched if the transaction aborts.
• 🔒 Non-Blocking Row Locking: High-throughput database polling with FOR UPDATE SKIP LOCKED / READPAST.
• ☕ Spring KafkaTemplate Auto-Wiring: Automatically binds to Spring Boot's existing KafkaTemplate or routes across named beans per pipeline without duplicate broker definitions.
• 🗄️ Multi-Database Support: Oracle, PostgreSQL, MySQL, Microsoft SQL Server, and SQLite.
• 🚦 Multi-Pipeline Routing: Configure isolated outbox tables, polling intervals, retry policies, and Kafka broker destinations per event pipeline.
• 📦 Zero Mandatory Dependencies: Core libraries rely strictly on clean SPI interfaces (BrokerPublisher, OutboxRepository, DatabaseDialect).

---

Architecture & How It Works

sequenceDiagram
    autonumber
    actor Client
    participant App as Application Service
    participant DB as Relational Database
    participant Hook as Post-Commit Hook
    participant Broker as Apache Kafka
    participant Poller as Background Poller / Reaper

    Client->>App: Business Operation (e.g. Create Order)
    App->>DB: BEGIN Transaction & INSERT Entity + Outbox Row
    DB-->>App: Transaction Committed Successfully
    
    alt Fast-Path (Sub-Millisecond Dispatch)
        App->>Hook: Trigger Post-Commit Hook
        Hook->>Broker: Async Send Event
        Broker-->>Hook: ACK (Partition, Offset)
        Hook->>DB: UPDATE Outbox Status -> PROCESSED
        Hook-->>App: Publish Complete
    else Fallback Slow-Path (Failure / Crash Recovery)
        Poller->>DB: SELECT PENDING/FAILED FOR UPDATE SKIP LOCKED
        DB-->>Poller: Return Unprocessed Records
        Poller->>Broker: Batch Publish
        Broker-->>Poller: Batch ACK
        Poller->>DB: UPDATE Outbox Status -> PROCESSED
    end
    App-->>Client: Success Response (Order Confirmed)

Loading

---

Database Dialects Matrix

Outboxify encapsulates database-specific concurrency control behind a pluggable DatabaseDialect SPI:

Database	Row Locking Strategy	Paging Syntax	High-Precision Timestamp	Sparse Indexing
PostgreSQL (9.5+)	FOR UPDATE SKIP LOCKED	LIMIT :n	CURRENT_TIMESTAMP	Partial Index (WHERE status != 'PROCESSED')
Oracle (12c+)	FOR UPDATE SKIP LOCKED	FETCH FIRST :n ROWS ONLY	SYSTIMESTAMP	Function-Based Index
MySQL (8.0+)	FOR UPDATE SKIP LOCKED	LIMIT :n	NOW(6)	Composite Index
MS SQL Server (2019+)	WITH (UPDLOCK, READPAST, ROWLOCK)	TOP (:n)	SYSUTCDATETIME()	Filtered Index (WHERE status != 'PROCESSED')
SQLite (3.35+)	Single-Writer Transaction	LIMIT :n	CURRENT_TIMESTAMP	Partial Index

---

Polyglot Quickstart

1. Java (Spring Boot 4.x / JDK 21+)

Add Dependency

<dependency>
    <groupId>io.outboxify</groupId>
    <artifactId>outboxify-spring-boot-starter</artifactId>
    <version>0.1.0</version>
</dependency>

Application Code

@Service
public class OrderService {

    private final OutboxPublisher outboxPublisher;
    private final OrderRepository orderRepository;

    public OrderService(OutboxPublisher outboxPublisher, OrderRepository orderRepository) {
        this.outboxPublisher = outboxPublisher;
        this.orderRepository = orderRepository;
    }

    @Transactional
    public OrderResponse createOrder(CreateOrderRequest request) {
        // 1. Persist business entity
        Order order = orderRepository.save(new Order(request.customerId(), request.amount()));

        // 2. Stage outbox record in the same transaction
        outboxPublisher.publish("orders", OutboxPayload.builder()
                .topic("orders.v1")
                .partitionKey(order.getCustomerId())
                .payload("{\"orderId\":\"" + order.getId() + "\",\"amount\":" + order.getAmount() + "}")
                .header("traceId", UUID.randomUUID().toString())
                .header("eventType", "OrderCreated")
                .build());

        return new OrderResponse(order.getId(), "CONFIRMED");
    }
}

Spring KafkaTemplate Auto-Detection: If you have spring-kafka on your classpath or define a KafkaTemplate bean, Outboxify auto-detects it and dispatches records through your existing broker configuration.

---

2. Node.js / TypeScript (NestJS, Prisma, TypeORM)

Install Package

npm install @outboxify/core @outboxify/nestjs
# or for Prisma: npm install @outboxify/prisma
# or for TypeORM: npm install @outboxify/typeorm

NestJS Module Setup

import { Module } from '@nestjs/common';
import { OutboxifyModule, PipelineConfig, SqlOutboxRepository, KafkaBrokerPublisher } from '@outboxify/nestjs';

@Module({
  imports: [
    OutboxifyModule.forRoot({
      pipelines: {
        orders: new PipelineConfig({
          name: 'orders',
          tableName: 'ORDERS_OUTBOX',
          batchSize: 100,
          pollIntervalMs: 1000
        }),
      },
      repository: new SqlOutboxRepository(dbPool),
      brokerPublisher: new KafkaBrokerPublisher(kafkaProducer)
    })
  ]
})
export class AppModule {}

---

3. Python (FastAPI, SQLAlchemy)

Install Package

pip install "outboxify[sqlalchemy,fastapi,kafka]"

FastAPI Lifespan & Service

from fastapi import FastAPI, Depends
from sqlalchemy.orm import Session
from outboxify import OutboxPublisher, OutboxPayload, PipelineConfig, SqlAlchemyOutboxRepository, KafkaBrokerPublisher
from outboxify.fastapi import create_outboxify_lifespan, OutboxifyAppManager

pipelines = {"orders": PipelineConfig(name="orders", table_name="orders_outbox")}
app_manager = OutboxifyAppManager(pipelines, repo, broker)

app = FastAPI(lifespan=create_outboxify_lifespan(app_manager))

@app.post("/api/orders")
async def create_order(request: OrderCreate, db: Session = Depends(get_db)):
    # 1. Insert domain state
    order = Order(customer_id=request.customer_id, amount=request.amount)
    db.add(order)
    db.flush()

    # 2. Stage outbox record
    await app_manager.publisher.publish(
        "orders",
        OutboxPayload.of("orders.v1", {"orderId": order.id, "amount": order.amount}, partition_key=order.customer_id)
    )
    db.commit()
    return {"orderId": order.id, "status": "CONFIRMED"}

---

Configuration Reference

Example application.yml for multi-pipeline Spring Boot configuration:

outboxify:
  # Global defaults applied to all pipelines
  defaults:
    dialect: POSTGRESQL
    batch-size: 100
    poll-interval-ms: 1000
    processing-timeout-seconds: 60
    max-retries: 5
    immediate-send:
      enabled: true
      timeout-ms: 5000

  # Pipeline-specific definitions
  pipelines:
    orders:
      table-name: ORDERS_OUTBOX
      batch-size: 200

    payments:
      table-name: PAYMENTS_OUTBOX
      poll-interval-ms: 500
      processing-timeout-seconds: 15
      broker:
        # Route through a dedicated named KafkaTemplate bean
        kafka-template-ref: paymentsKafkaTemplate

---

Runnable Examples

Ready-to-run example microservices demonstrating fast-path execution, rollback protection, and database schema setups are available in examples/:

Example Project	Technology Stack	Description
order-service (Java)	Java 21, Spring Boot 4.x, H2	Dual-write order processing with transactional rollback API
spring-kafka-example (Java)	Java 21, Spring Kafka, H2	Custom KafkaTemplate injection & multi-template routing
order-service (Node.js)	Node.js 20+, Express, @outboxify/core	Node.js outbox dispatch with REST endpoints
order-service (Python)	Python 3.11+, FastAPI, SQLAlchemy	Async SQLAlchemy session hooks & lifespan integration

---

Building & Testing

Java

# Run unit tests
mvn clean test

# Run full integration tests with Testcontainers
mvn clean verify

Node.js

# Run test suite across all packages
npm test --prefix node

Python

# Run async test suite
pytest python/tests

---

Contributing

We welcome contributions from the community! Check out CONTRIBUTING.md for local setup instructions, code style guidelines, and pull request workflows. Maintainers can refer to RELEASING.md for the release process.

Please also review our Code of Conduct before participating.

---

Changelog

See CHANGELOG.md for release notes and migration guides.

---

Security

For vulnerability reporting and our security policy, please refer to SECURITY.md.

---

License

Outboxify is open-source software licensed under the Apache License, Version 2.0.