Skip to content

[Fix][Connector-V2] Close JDBC source read transactions after each split - #1

Open
brownie-cake wants to merge 2 commits into
devfrom
fix/jdbc-source-read-transaction-cleanup
Open

[Fix][Connector-V2] Close JDBC source read transactions after each split#1
brownie-cake wants to merge 2 commits into
devfrom
fix/jdbc-source-read-transaction-cleanup

Conversation

@brownie-cake

Copy link
Copy Markdown

Purpose of this pull request

Fix JDBC source sessions that can remain idle in transaction after a source split has finished reading or has failed during initialization.

PostgreSQL and Redshift cursor reads disable auto-commit in their dialect-specific prepared-statement creation. JdbcInputFormat.close() currently closes only the ResultSet and PreparedStatement; closing those resources does not end a manual JDBC transaction. Because the source connection is reused across splits, the database backend can retain the old transaction snapshot while the reader is waiting for another split or while the rest of the job is being cleaned up.

On PostgreSQL-compatible databases this is visible as sessions with:

state           = idle in transaction
wait_event_type = Client
wait_event      = ClientRead

The retained transaction can hold backend_xmin, delay vacuum/tuple cleanup, retain transaction-scoped locks and consume connection slots.

This patch closes the transaction at the JDBC source split lifecycle boundary:

  • capture the statement connection before closing JDBC resources;
  • close and clear the result set and statement even when either close operation fails;
  • roll back an open read transaction after a split is consumed or aborted;
  • restore the user-configured auto-commit mode after a dialect temporarily changes it;
  • clean up SQL and runtime failures raised while opening a split;
  • discard the cached connection if statement creation fails before the statement can expose its connection;
  • physically close and invalidate a connection when rollback or auto-commit restoration fails;
  • make repeated close() calls idempotent;
  • always close the splitter connection from closeInputFormat() through finally.

rollback() is intentional. JDBC source processing is read-only from the connector's perspective. Rolling back ends the server-side cursor transaction, releases its snapshot and also clears a transaction that entered an aborted state. Cursor-based incremental fetching remains enabled; the patch does not force PostgreSQL/Redshift reads into auto-commit mode while the result set is active.

Does this PR introduce any user-facing change?

Yes, as a bug fix.

Before this change, PostgreSQL-compatible JDBC source sessions could remain in idle in transaction after a split completed, until the whole reader connection was eventually closed.

After this change, the read transaction ends immediately at split cleanup. The configured auto-commit mode is restored before the connection is reused.

There is no configuration-key, default-value or public-API change. JDBC Sink, XA and exactly-once commit semantics are not modified.

How was this patch tested?

Added JdbcInputFormatTest coverage for:

  1. PostgreSQL-style cursor mode: runtime auto-commit changes from configured true to false; cleanup rolls back and restores true.
  2. Explicit auto_commit=false: cleanup rolls back while preserving manual-commit mode.
  3. Native auto-commit connections: no unnecessary rollback or mode change.
  4. Result-set and statement close failures: transaction cleanup still executes.
  5. Rollback failure: the physical connection is closed and the cached provider connection is invalidated.
  6. Failure to obtain the connection from the statement: the cached connection is closed instead of being reused in an unknown transaction state.
  7. Prepared-statement creation failure before assignment: the cached connection is discarded.
  8. Query execution failure after statement creation: statement cleanup, rollback, mode restoration and provider cleanup all execute.
  9. Input-format shutdown: the splitter connection is always closed through finally.
  10. Already-closed connections and repeated close() calls: cleanup is safe and idempotent.

The production and test sources were also compiled against Java 8-compatible type stubs, and the transaction state transitions were exercised with a standalone JDBC Connection proxy harness. The repository's connector-JDBC unit, formatting and static checks are delegated to this PR's GitHub CI.

Check list

  • No new Jar binary package is added.
  • No connector option, plugin mapping or distribution dependency is added.
  • No incompatible configuration or public API change is introduced.
  • Unit tests cover normal, exceptional and idempotent cleanup paths.
  • Documentation changes are not required because this restores the existing bounded-source lifecycle contract.

…esense and Assert connector docs (apache#11751)

Co-authored-by: DanielCarter-stack <danielcarter-stack@users.noreply.github.com>
@brownie-cake
brownie-cake force-pushed the fix/jdbc-source-read-transaction-cleanup branch from 054f94f to 7e92698 Compare August 12, 2026 10:59
End cursor/manual transactions when a JDBC source split closes, restore the configured auto-commit mode, and discard only connections whose transaction state cannot be cleaned safely. Add regression coverage for clean failure recovery and multi-split connection reuse.
@brownie-cake
brownie-cake force-pushed the fix/jdbc-source-read-transaction-cleanup branch from 7e92698 to 7f70804 Compare August 12, 2026 11:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants