Clamp bulk_update batches to the dialect bind-parameter ceiling#21
Merged
Conversation
bulk_update sized batches purely by batch_size, unlike bulk_create which already clamps by dialect.max_bind_params. Each row costs 2*F + 1 bind parameters (a pk/value pair per field in the CASE arms plus one pk slot in the WHERE IN list), so on SQL Server the default batch_size=500 with two or more fields exceeded the 2100-parameter ceiling and failed at runtime. Also lower SqliteDialect.max_bind_params to 32766, SQLite's actual SQLITE_MAX_VARIABLE_NUMBER default (3.32+); the inherited 65535 allowed bulk statements past it. Tests: a SQL-shape guard asserting one WHERE / one IN clause per statement (one CASE per field), and a clamp test verifying batches split under a lowered ceiling with all rows still updated.
Oracle treats '' as NULL, so the field's empty-string default violated the NOT NULL constraint on insert (ORA-01400) and both new tests failed on the oracle backend in CI.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
bulk_updatesized batches purely bybatch_size, unlikebulk_createwhich already clamps bydialect.max_bind_params. Each row costs2*F + 1bind parameters (a pk/value pair per field in the CASE arms plus one pk slot in theWHERE ... INlist), so on SQL Server the defaultbatch_size=500with two or more fields exceeded the 2100-parameter ceiling and failed at runtime.Changes
Model.bulk_updatenow clamps the effective batch size:min(batch_size, max(1, max_bind_params // (2*F + 1)))— same approachbulk_createuses.SqliteDialect.max_bind_paramslowered to 32766, SQLite's actualSQLITE_MAX_VARIABLE_NUMBERdefault (3.32+); the inherited 65535 allowed bulk statements past it.Tests
WHEREand oneINclause per statement and oneCASEper written field (tripwire against the per-field WHERE duplication pattern that makes PostgreSQL reject the PK index and seq-scan).Full sqlite suite (1223 tests) and
make lintpass.