Transactional outbox for PostgreSQL, MySQL/MariaDB, SQL Server + Kafka/Redpanda — reliable event publishing for Node.js & TypeScript.
Write your data and your events in one transaction; eventferry ferries them to the broker — at‑least‑once, in order, with retries and dead‑lettering.
┌─────────────┐ one TX ┌────────────────────────┐ relay ┌───────────────┐
│ your code │ ───────────▶ │ outbox tbl │ ──────────▶ │ Kafka/Redpanda│
│ (order svc) │ (atomic) │ (Postgres / MySQL / │ publish │ topic │
│ │ │ MariaDB / SQL Server)│ │ │
└─────────────┘ └────────────────────────┘ └───────────────┘
eventferry implements the transactional outbox pattern so you don't have to hand-roll it. Instead of publishing to Kafka directly (which can lose events on a crash, or publish events for data that rolled back), you write the event into an outbox table in the same database transaction as your data. A background relay then reliably ships those rows to the broker. The database is the source of truth; nothing is lost or invented.
- ✅ Atomic — your data and its event commit together, or not at all. No lost or phantom events.
- 🗄️ Three shipped databases — PostgreSQL, MySQL / MariaDB, and SQL Server behind one
OutboxStorecontract. - 🔁 At-least-once delivery with idempotent producers (pair with idempotent consumers).
- 🔢 Strict per-aggregate ordering — events for the same entity arrive in order, even across many relays and retries.
- ♻️ Retries with fixed / linear / exponential backoff + jitter, and dead-letter routing for poison messages.
- 🏷️ Typed
errorKindtaxonomy —retriable/fatal/poison/backpressure/quota/fenced— so you can route DLQ records, alerts, and circuit-breakers without parsing error strings. - 🛟 Crash recovery — work orphaned by a dead relay is reclaimed automatically (visibility timeout).
- ⚡ Horizontal scale — run any number of relays against one table, lock-free (
FOR UPDATE SKIP LOCKEDon Postgres/MySQL,READPASTon SQL Server). - 🧷 Type-safe, schema-validated events (optional) — a bad payload can't reach the outbox. Any Standard Schema (Zod, Valibot, ArkType…).
- 🚀 Low-latency modes — Postgres
LISTEN/NOTIFYor WAL streaming; SQL Server Service Broker waker for sub-second wake on-prem + Azure SQL Managed Instance; CDC relay as a separate package. - 🧰 Two Kafka clients —
kafkajsand@confluentinc/kafka-javascriptbehind one API, plus an AWS MSK IAM helper. - 📦 Schema Registry — Avro / Protobuf / JSON Schema via Confluent Schema Registry.
- 🔭 Observability — metrics hooks + W3C / OpenTelemetry trace propagation on both sides (
publisher.tracer.injectat enqueue,extractTraceContextfor consumers). - 🪶 Zero runtime dependencies in the core; everything else is an optional peer.
npm i @eventferry/core @eventferry/postgres @eventferry/kafka pg kafkajsimport { Relay } from "@eventferry/core";
import { PostgresStore, createMigrationSql } from "@eventferry/postgres";
import { KafkaPublisher } from "@eventferry/kafka";
// 1. Create the outbox table (idempotent — safe to run on boot).
await pool.query(createMigrationSql("outbox"));
const store = new PostgresStore({ pool });
// 2. Write side: enqueue the event in the SAME transaction as your data.
const client = await pool.connect();
try {
await client.query("BEGIN");
await client.query("INSERT INTO orders (...) VALUES (...)");
await store.enqueue(client, {
topic: "orders.created",
aggregateType: "order",
aggregateId: order.id, // → Kafka partition key (preserves per-entity order)
payload: { orderId: order.id, total: order.total },
});
await client.query("COMMIT"); // order + event commit atomically
} catch (e) {
await client.query("ROLLBACK");
throw e;
} finally {
client.release();
}
// 3. Publish side (a separate worker): drain the outbox to Kafka.
const relay = new Relay({
store,
publisher: new KafkaPublisher({ brokers: ["localhost:19092"], idempotent: true }),
retry: { maxAttempts: 5, strategy: "exponential", baseMs: 200, maxMs: 30_000 },
dlq: { topic: "orders.dlq" },
});
await relay.start();
process.on("SIGTERM", () => relay.stop());That's the whole pattern. Everything below is optional power.
MySQL / MariaDB and SQL Server work the same way — swap
PostgresStoreforMySqlStorefrom@eventferry/mysqlorMssqlStorefrom@eventferry/mssql. Sameenqueue/RelayAPI, same ordering and crash-recovery semantics. See each package's README for the per-driver examples (mysql2for MySQL/MariaDB,mssqlfor SQL Server), and the 23-page Wiki for end-to-end recipes, ops guides, and the per-database capability matrix.
Most outbox libraries lock you to one Kafka client, only poll, and skip ordering, idempotency, or crash recovery. eventferry is a complete, production-grade toolkit:
| Concern | eventferry | Typical outbox lib |
|---|---|---|
| Databases | Postgres, MySQL/MariaDB, SQL Server behind one contract | usually one |
| Kafka client | kafkajs and confluent behind one API (+ MSK IAM helper) | one, hard-coded |
| Delivery | idempotent + optional transactional (EOS) | best-effort |
| Ordering | strict per-aggregate, across relays & retries | none / global only |
| Retries & DLQ | backoff + jitter → dead-letter topic, typed errorKind taxonomy |
basic / none |
| Crash recovery | visibility-timeout reaper | rows can get stuck |
| Latency | poll + LISTEN/NOTIFY + WAL streaming (Postgres), Service Broker waker + CDC relay (SQL Server) | poll only |
| Type safety | typed + schema-validated payloads | any |
| Serialization | JSON + Schema Registry (Avro/Proto/JSON) | JSON |
| Tracing | W3C / OpenTelemetry propagation on both producer and consumer | none |
| Footprint | zero-dep core; storage/broker pluggable | varies |
- Not a message broker or queue — it reliably bridges your DB to Kafka/Redpanda; the broker does delivery.
- Not a consumer framework — it publishes events. Consuming them is your job (a typed
decodehelper is provided, but no subscription loop). - Not an ORM or migration tool — you bring your own
pgpool and migration runner; eventferry hands you the SQL to run. - Not exactly-once end-to-end — at-least-once by default (with an optional EOS producer for the broker hop). Make consumers idempotent.
Use eventferry when:
- You write to PostgreSQL, MySQL / MariaDB, or SQL Server and publish to Kafka or Redpanda in the same business operation, and you can't afford the database and broker to disagree.
- You're building microservices with an event-driven architecture and need reliable event publishing without losing messages on crashes.
- You need strict per-aggregate ordering (e.g. all events for
order:42published in the order they happened) — even with many relays running. - You want at-least-once delivery with retries, backoff, jitter, and a dead-letter queue out of the box.
- You want a Node.js / TypeScript library and a small, focused dependency tree — not a JVM-heavy CDC platform.
- You need CDC-style throughput without operating Kafka Connect: eventferry can stream straight from Postgres WAL, and ships a separate SQL Server CDC relay (
@eventferry/mssql-cdc-relay) for sub-second waking on SQL Server.
Don't use eventferry when:
- You only publish events from one service to itself — a plain in-process queue is simpler.
- You don't have a transactional database write — there's no dual-write to solve.
- You need exactly-once end-to-end semantics across consumers — that's an application-layer concern (make consumers idempotent).
- You're already running Debezium / Kafka Connect at scale and it fits your operations — eventferry overlaps for the Postgres → Kafka path.
| Alternative | What it is | When eventferry is better |
|---|---|---|
| Debezium / Kafka Connect | JVM-based CDC platform that tails the database log and publishes changes to Kafka. | You want a Node.js library instead of a JVM cluster, no Kafka Connect to operate, and a typed event registry instead of raw row-change events. eventferry can still use Postgres WAL streaming or SQL Server CDC for similar throughput. |
| pg-boss / BullMQ | Background job queues backed by Postgres or Redis. | These are job queues, not outboxes — they're not designed for atomic dual-write with your business transaction or for publishing to Kafka. Use them for jobs, use eventferry for events. |
| Custom outbox table (DIY) | Hand-rolled outbox table + a cron / worker that polls and publishes. | You skip writing (and maintaining) the parts that are easy to get wrong: skip-locked claiming under concurrent relays, per-aggregate ordering, crash-recovery reaper, retry/backoff, DLQ routing, Schema Registry serialization, and CDC. |
| Publishing to Kafka after DB commit (naive) | await db.commit(); await kafka.send(...). |
This is exactly the dual-write problem eventferry exists to solve: a crash between the two calls drops the event. |
| Transactional Kafka producer alone | Kafka EOS without an outbox. | Doesn't help — your database write isn't inside the Kafka transaction. You still need an outbox to atomically persist the intent to publish. |
eventferry is a small set of focused packages. For everything in one shot, install the meta-package and your Kafka client:
npm i @eventferry/all pg kafkajs # pulls in core + postgres + kafka + schema-registryOr install only the pieces you use (smaller dependency tree) — the core, a storage adapter, a broker adapter, and your chosen Kafka client:
# core engine + Kafka publisher + ONE storage adapter (pick the one you need):
# PostgreSQL:
npm i @eventferry/core @eventferry/postgres @eventferry/kafka pg
# MySQL / MariaDB:
npm i @eventferry/core @eventferry/mysql @eventferry/kafka mysql2
# SQL Server (on-prem, Azure SQL Managed Instance, or Azure SQL DB):
npm i @eventferry/core @eventferry/mssql @eventferry/kafka mssql
# pick ONE Kafka client (both are optional peers):
npm i kafkajs # pure JS, zero native deps
npm i @confluentinc/kafka-javascript # librdkafka-backed, higher throughput
# optional add-ons:
npm i @eventferry/schema-registry @kafkajs/confluent-schema-registry # Avro/Proto/JSON
npm i pg-logical-replication # Postgres WAL streaming mode
npm i @eventferry/mssql-cdc-relay # SQL Server CDC-driven sub-second waker
npm i @eventferry/kafka-iam aws-msk-iam-sasl-signer-js # AWS MSK IAM auth helperNote Requires Node.js 18+, Kafka or Redpanda, and one of: PostgreSQL 13+, MySQL 8.0.1+ / MariaDB 10.6+, or SQL Server 2017+ (incl. Azure SQL DB and Managed Instance). The Postgres streaming relay also needs
wal_level = logicalon the server.
createMigrationSql returns idempotent DDL (table + indexes). Run it however you manage
schema — raw, Flyway, Prisma, node-pg-migrate, etc.
import { createMigrationSql } from "@eventferry/postgres/migrations";
await pool.query(createMigrationSql("outbox"));Declare topics once — aggregate type + payload schema — and get a typed, validated
enqueue plus a decode consumers can reuse. The schema is any Standard Schema; there is
no validator dependency in the package.
import { z } from "zod";
import { defineOutbox } from "@eventferry/core";
const registry = {
"orders.created": { aggregateType: "order", schema: z.object({ orderId: z.string(), total: z.number() }) },
"orders.shipped": { aggregateType: "order", schema: z.object({ orderId: z.string(), carrier: z.string() }) },
} as const;
const outbox = defineOutbox(registry, { store });
// Typed + validated before the row is inserted; a bad payload throws and your TX rolls back.
await outbox.enqueue(client, "orders.created", {
aggregateId: order.id,
payload: { orderId: order.id, total: order.total }, // ✓ typed from the schema
});
// In a consumer (same registry, no store): decode back to the typed, validated payload.
const event = await defineOutbox(registry).decode("orders.created", message.value);
// ^? { orderId: string; total: number }The claim query only takes a row when it is the head of its aggregate (no earlier
unfinished row for the same aggregateId). So at most one message per aggregate is
in-flight, a failed message blocks its successors until retried, and the broker never
sees same-key messages out of order — even across concurrent relays.
Each claim stamps claimed_at; a row left in processing longer than claimTimeoutMs
is reclaimed (its relay is presumed dead):
new PostgresStore({ pool, claimTimeoutMs: 60_000 }); // default 60sWarning Set
claimTimeoutMsabove your worst-case publish latency. If it fires while a slow- but-alive relay is still in flight, the row is republished — a duplicate that idempotent consumers absorb (this is an at-least-once system).
claimBatch uses SELECT ... FOR UPDATE SKIP LOCKED, so any number of relay instances
can run against the same table — each claims a disjoint set of rows, none block on the
others, and per-aggregate ordering still holds.
new KafkaPublisher({
driver: "confluent",
brokers: ["localhost:19092"],
idempotent: true,
transactional: true,
transactionalId: "eventferry-relay-1", // stable per relay instance
});kafkajs and @confluentinc/kafka-javascript are interchangeable behind one
KafkaPublisher — same options, same behavior. Pass a customDriver for testing or
unsupported clients.
new KafkaPublisher({ driver: "kafkajs", brokers }); // easy to deploy
new KafkaPublisher({ driver: "confluent", brokers }); // higher throughputPolling adds up to one pollIntervalMs of delay. Two faster modes:
Notify-driven (LISTEN/NOTIFY) — claims the instant a row commits; polling stays on
as a safety net. No extra dependency.
import { createNotifyTriggerSql } from "@eventferry/postgres/migrations";
import { PostgresNotifyWaker } from "@eventferry/postgres";
import { Client } from "pg";
await pool.query(createNotifyTriggerSql("outbox", "outbox")); // one-time trigger
new Relay({
store, publisher,
pollIntervalMs: 5_000, // now just a safety net
waker: new PostgresNotifyWaker({ connect: () => new Client(connStr), channel: "outbox" }),
});Streaming (logical replication) — publishes straight from the WAL with
PostgresStreamingRelay, with no claim query on the happy path (lower DB load). Failures
fall back to the claim-based retry/DLQ loop, so those guarantees are reused.
import { createPublicationSql } from "@eventferry/postgres/migrations";
import { PostgresStreamingRelay } from "@eventferry/postgres";
await pool.query(createPublicationSql("outbox", "outbox_pub"));
await new PostgresStreamingRelay({
store: new PostgresStore({ pool, claimFailedOnly: true }),
publisher,
replication: { connectionString: connStr, slot: "outbox_slot", publication: "outbox_pub", table: "outbox" },
dlq: { topic: "orders.dlq" },
}).start();Note Streaming is best-effort per-aggregate ordering (a retried failure can land after later same-aggregate rows). Use the polling relay if you need the strict guarantee.
Encode payloads in the Confluent wire format instead of JSON — a drop-in serializer:
import { SchemaRegistrySerializer } from "@eventferry/schema-registry";
const serializer = new SchemaRegistrySerializer({
host: "http://localhost:8081",
schemas: { "orders.created": { type: "AVRO", schema: orderCreatedAvsc } },
});
new Relay({ store, publisher, serializer }); // also works with PostgresStreamingRelayEnd-to-end W3C trace propagation works on both sides without pulling a tracing package into eventferry itself:
- Producer side — a
traceparentcaptured at enqueue rides to the published message on every path (polling relay, notify-driven, streaming, SQL Server Service Broker / CDC). You wire it in viapublisher.tracer.injector the store'stracing.inject. - Consumer side —
extractTraceContext(headers)from@eventferry/kafka/consumeparses thetraceparent/tracestateheaders your relay wrote, so a CONSUMER span can be started as the child of the producer's span.
import { propagation, context } from "@opentelemetry/api";
new PostgresStore({
pool,
tracing: { inject: (carrier) => propagation.inject(context.active(), carrier) },
});eventferry is publisher-only — your consumer is whatever you already use (kafkajs, @confluentinc/kafka-javascript, KafkaJS rebroadcast, Faust, …). Two helpers make the consumer side ergonomic:
decode(message, { decoder })from@eventferry/kafka/consumenormalizes the raw message shape (key, value, headers, offset, timestamp, partition) both kafkajs and confluent deliver. Built-in decoders:json(default),utf8,none, or a custom(bytes) => V.extractTraceContext(headers)parses the W3Ctraceparent/tracestateheaders your relay's tracing wrote at enqueue, so you can start a CONSUMER span as the child of the producer's span.defineOutbox(registry).decode(topic, bytes)from@eventferry/core— the same registry you used to enqueue, used in reverse: JSON-parse + validate + return the typed payload. ThrowsOutboxValidationErrorif the topic is unknown or the payload doesn't match the schema. Use this when the consumer lives in the same monorepo as the producer.
import { Kafka } from "kafkajs";
import { decode, extractTraceContext } from "@eventferry/kafka/consume";
import { defineOutbox } from "@eventferry/core";
import { registry } from "./outbox-registry";
const events = defineOutbox(registry); // no store — consumer side
const consumer = new Kafka({ brokers }).consumer({ groupId: "orders-worker" });
await consumer.connect();
await consumer.subscribe({ topic: "orders.created" });
await consumer.run({
eachMessage: async ({ message }) => {
// 1) Normalize the raw kafkajs/confluent shape.
const m = decode(message, { decoder: "utf8" });
// 2) Continue the W3C trace context the producer wrote (optional).
const trace = extractTraceContext(message.headers);
if (trace) startConsumerSpan(trace.traceId, trace.spanId);
// 3) Typed + validated payload via the same registry as the producer.
const event = await events.decode("orders.created", m.value!);
// ^? { orderId: string; total: number }
await handle(event);
},
});When a record exhausts retry.maxAttempts (or hits a fatal / poison error), the relay calls publishToDlq and the message lands on ${topic}.dlq (or your configured DLQ topic). The relay classifies every failure into a typed errorKind — one of retriable, fatal, poison, backpressure, quota, or fenced — which drives retry vs. DLQ vs. circuit-break decisions and lands on the DLQ headers. The DLQ message carries enriched headers:
dlq-reason—error.messagedlq-error-class—KafkaJSProtocolError,RecordTooLargeException, …dlq-error-classis good for alert routing without parsing the reason stringdlq-original-topic— the topic this record was originally destined fordlq-failed-at— ISO timestampdlq-attempts— how many tries the relay made before giving updlq-stack(optional, opt-in) — truncated UTF-8 stack
Minimal DLQ consumer:
const dlqConsumer = new Kafka({ brokers }).consumer({ groupId: "dlq-handler" });
await dlqConsumer.subscribe({ topic: "orders.created.dlq" });
await dlqConsumer.run({
eachMessage: async ({ message }) => {
const m = decode(message);
const reason = m.headers["dlq-reason"];
const errClass = m.headers["dlq-error-class"];
const failedAt = m.headers["dlq-failed-at"];
// Route poison records to a human queue; quota / transient to a retry queue, etc.
if (errClass === "KafkaJSProtocolError" && reason?.includes("MESSAGE_TOO_LARGE")) {
await ticket.create({ title: `Oversized DLQ record from ${m.headers["dlq-original-topic"]}`, body: reason });
} else {
await retryQueue.put({ payload: m.value, attemptsSoFar: Number(m.headers["dlq-attempts"] ?? "0") });
}
},
});new Relay({
hooks: {
onBatchClaimed: (n) => metrics.gauge("outbox.batch", n),
onPublished: (r) => metrics.increment("outbox.published"),
onDead: (rec, err) => alert(`dead message ${rec.id}: ${err.message}`),
},
});Each published message carries message-id, aggregate-type, aggregate-id,
content-type, and (when present) trace-id headers.
Published rows accumulate as done. Indexes exclude them so queries stay fast, but the
table grows; purgeDone batch-deletes old ones (run from your own cron):
await store.purgeDone({ olderThanMs: 7 * 24 * 60 * 60 * 1000 }); // older than 7 days| Package | What it is |
|---|---|
@eventferry/core |
DB- and broker-agnostic engine: relay loop, backoff, serializer, typed registry, errorKind taxonomy. Zero runtime deps. |
@eventferry/postgres |
PostgreSQL store, migration/trigger/publication SQL, LISTEN/NOTIFY waker, WAL streaming relay, retention. |
@eventferry/mysql |
MySQL 8.0.1+ / MariaDB 10.6+ store with SKIP LOCKED claim, strict per-aggregate ordering, and reaper. |
@eventferry/mssql |
SQL Server 2017+ / Azure SQL DB & Managed Instance store with READPAST claim, Service Broker waker for sub-second wake on-prem + Managed Instance, and reaper. |
@eventferry/mssql-cdc-relay |
Optional sub-second waker for SQL Server driven by Change Data Capture — shipped as a separate package so the core MSSQL store stays CDC-free. |
@eventferry/kafka |
Kafka/Redpanda publisher over kafkajs and confluent drivers, with DLQ routing, typed auth, and a cheap healthCheck() reachability probe. |
@eventferry/kafka-iam |
AWS MSK IAM helper — SASL/OAUTHBEARER auth provider you wire into the publisher when running against Amazon MSK. |
@eventferry/schema-registry |
Confluent Schema Registry serializer (Avro / Protobuf / JSON Schema), with typed basic + bearer auth. |
@eventferry/all |
Meta-package — installs & re-exports all of the above. npm i @eventferry/all for everything in one import. |
Yes. eventferry takes a pg pool / client; it doesn't replace your ORM. Pass the underlying pg.Client your ORM is using inside its transaction to store.enqueue(client, ...) — the event is written in the same transaction as your ORM's writes.
That's the dual-write problem: a crash, a network blip, or a Kafka outage between the commit and the publish drops the event silently. The database thinks the order was placed; downstream services never hear about it. The outbox pattern (and eventferry) closes that window by writing the intent to publish atomically with the business data.
Debezium is a JVM-based CDC platform that tails the database log and publishes row changes to Kafka via Kafka Connect. It's powerful but operationally heavy: you run a Connect cluster, your events are row-level (not domain-level), and integration with Node.js services is indirect. eventferry is a Node.js library with a typed event registry — you publish domain events you define, no Kafka Connect to operate. For high throughput, eventferry can stream directly from Postgres WAL just like Debezium does.
To the broker, yes — you can enable a transactional (EOS) producer so a batch lands atomically in Kafka. End-to-end exactly-once is impossible without idempotent consumers: design consumers to dedupe on messageId.
Yes. Redpanda is wire-compatible with the Kafka protocol; both kafkajs and @confluentinc/kafka-javascript drivers work unchanged.
The relay is database-agnostic — only the OutboxStore adapter is per-database. PostgreSQL, MySQL / MariaDB, and SQL Server ship today (@eventferry/postgres, @eventferry/mysql, @eventferry/mssql). MongoDB is the next adapter on the roadmap, followed by CockroachDB, SQLite, Oracle, and DynamoDB.
One database table (outbox), one background process (the relay — runs in your existing Node.js service, no extra deployment), and a periodic retention job. The relay is horizontally scalable: run N instances, lock-free claim guarantees no double-publish.
Yes. eventferry ships with strict per-aggregate ordering, a crash-recovery reaper, retries with backoff + jitter, DLQ routing with a typed errorKind taxonomy, Schema Registry support, W3C trace propagation on both producer and consumer sides, and an integration test suite running against real PostgreSQL, MySQL, MariaDB, SQL Server, and Redpanda via Testcontainers — including an adversarial bug-hunt suite for the Postgres + MySQL stores. It is MIT-licensed and used in production.
eventferry is built around a small, database-agnostic OutboxStore contract, so
every new database is just a new adapter. PostgreSQL ✅, MySQL / MariaDB ✅,
and SQL Server ✅ ship today; MongoDB is next-up, with CockroachDB, SQLite,
Oracle, and DynamoDB on the horizon.
See ROADMAP.md for the full plan — architecture diagrams, the per-database capability matrix, and phase-by-phase checklists — and the 23-page Wiki for ops guides, end-to-end recipes, and per-database tuning notes.
pnpm install
pnpm build
pnpm typecheck
pnpm test:run # unit tests (fakes, no infra)
pnpm test:integration # real Postgres / MySQL / MariaDB / SQL Server + Redpanda via Testcontainers (needs Docker)Issues and pull requests are welcome. Run pnpm test:run and pnpm typecheck before
opening a PR; add a changeset for any
user-facing change.
MIT © Samet GOKTEPE