Dead Letter Queue (DLQ) for Connectors

[kriti-sc]

Overview

When a connector fails to process one or more messages from a batch, those messages are currently lost forever. The following is a proposal for a Dead Letter Queue (DLQ) that routes failed messages to a dedicated Iggy topic so they can be inspected, debugged, and replayed.

---

Outline

1. Design defines the DLQ semantics and the high-level design for routing failures to DLQ.
2. Connector to DLQ Message Path describes the control flow to route a failed message to the DLQ.
3. ABI Impact evaluates alternatives for implementing the required changes to the iggy_sink_open FFI.

---

Design

DLQ Semantics

1. DLQ to contain failing messages only, not the entire batch

When a batch partially fails, the connector must separate successful messages from failed ones and only send the failed messages to the DLQ. The sucessful messages must be written to the target.

Writing the entire batch to DLQ on any single failure wastes storage, makes debugging and replay harder, and obscures the actual failure rate.

2. DLQ is an Iggy topic

Failed messages are published to a user-defined Iggy topic. These are fully user-defined in the sink config. The runtime reads them at startup and sets up the DLQ producer accordingly.

High-level Design

The design of the DLQ follows directly from the principles above.

1. Since DLQ operates at message granularity and only the connector has visibility into message-level failures, it is the connector's resposibility to route messages to the DLQ. This also means that DLQ is opt-in per connector. If a connector does not implement DLQ, failed messages are silently lost.

2. DLQ writes are bridged via a DlqCallback function pointer injected into the plugin at open() time, following the same pattern as LogCallback. This is needed because connectors do not have access to the runtime's IggyClient.

---

Connector to DLQ Message Path

This section describes the flow of a failed message from detection in the sink/source to its eventual persistence in the DLQ.

Sink

1. The runtime injects a DlqWriter into the sink at open() time via a callback
2. During consume(), the sink segregates messages into two structures — successful and failed — as it processes each message
3. At the end of consume(), successful messages are written to the target in a single batch write
4. The sink calls dlq.send(topic_metadata, partition_id, msg, reason) for each failed message — the SDK constructs the DlqEnvelope, serializes it to bytes, and invokes the callback
5. The runtime's callback receives the serialized bytes and produces them to the DLQ stream and topic specified in config

Source

1. The runtime injects a DlqWriter into the source at open() time via a callback
2. During poll(), the source fetches a batch from the external system and converts each record into a ProducedMessage
3. Records that fail conversion or transformation are sent to DLQ via dlq.send() with the failure reason
4. Successful records are returned in ProducedMessages — the runtime writes them to the target Iggy topic
5. The runtime's callback receives the serialized bytes and produces them to the DLQ stream and topic specified in config

---

ABI Impact

Introducing DlqCallback requires changing the iggy_sink_open (and iggy_source_open) FFI signature — the entrypoint symbol every plugin must export. This is a breaking ABI change: any plugin compiled against the old signature and loaded by a new runtime will misinterpret the call arguments, regardless of whether the plugin uses DLQ. A forced rebuild of all plugins is required.

Three approaches to handle this:

Option 1: New versioned entrypoint symbol

Add iggy_sink_open_v2 with the new signature alongside the existing iggy_sink_open. The runtime checks for iggy_sink_open_v2 first (dlopen symbol lookup), and falls back to iggy_sink_open for older plugins. Old plugins continue to work without recompile; new plugins opt in by exporting the v2 symbol.

• Pro: fully backwards compatible, no forced rebuild
• Con: runtime must carry fallback logic indefinitely; adds complexity to the plugin loading path

Option 2: Separate iggy_sink_set_dlq_callback symbol (recommended)

Keep iggy_sink_open unchanged. Add a new optional symbol iggy_sink_set_dlq_callback(id, dlq_callback) that the runtime calls after open() if DLQ is configured for that sink. Plugins that don't export this symbol simply don't get DLQ wired up.

// new optional symbol — only connectors that support DLQ need to export this
unsafe extern "C" fn iggy_sink_set_dlq_callback(id: u32, dlq_callback: DlqCallback) -> i32

• Pro: purely additive, no signature change, no versioning logic, cleanly expresses DLQ as optional capability
• Con: two-step initialization (open, then set callback); runtime must handle the case where the symbol is absent

Option 3: Accept the forced rebuild

Since all official sink and source plugins live in the same repository, a coordinated rebuild across all of them is low friction. The only plugins affected without a rebuild would be third-party plugins compiled outside the repo.

• Pro: simplest — one clean signature, no compatibility shims
• Con: breaks any out-of-tree plugins; requires discipline to rebuild everything together