Implement our own version of normalize#41
Open
sgrif wants to merge 1 commit into
Open
Conversation
This introduces a reimplementation of normalize using our primitives instead of the implementation in libpg_query. In addition to reducing our reliance on that library, this implementation takes advantage of the fact that for our use case we will already have a parsed AST to pass in. libpg_query takes a string, parses it, records the location of any constants, then copies and manipulates that string to replace the constants. In contrast, this implementation just copies the AST, walks it, replaces any constants with parameters, and changes the numbers on any parameters it encounters. In an effort to do this change in a single iteration over the parse tree, we don't preserve any bind parameters that were in the original query, which you can see reflected in the test that was updated. On an extremely large query, the new implementation is 4.5x faster. If you include deparse time it is about twice as fast, and including parse + deparse time it is roughly 10% slower. While it isn't ideal that the full parse + normalize + deparse flow is slightly slower, in pgdog we'd only be using the normalize + deparse flow, so this results in a significant performance win as well as a reduction in unsafe code.
levkk
reviewed
Jul 16, 2026
| [dev-dependencies] | ||
| criterion = "0.8.2" | ||
| pg_query = { version = "6.1.1", git = "https://github.com/pgdogdev/pg_query.rs.git" } | ||
| # pg_query = { version = "6.1.1", git = "https://github.com/pgdogdev/pg_query.rs.git" } |
Contributor
|
Could you also include a |
Collaborator
Author
If you'd reviewed the code you would have seen that it's already there ;) https://github.com/pgdogdev/pg_raw_parse/pull/41/changes#diff-7d7ec379288d03f370e79b3210ce515b7e24f37e5e17248d1e9ce6fd557e1bfbR26 |
meskill
approved these changes
Jul 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This introduces a reimplementation of normalize using our primitives instead of the implementation in libpg_query. In addition to reducing our reliance on that library, this implementation takes advantage of the fact that for our use case we will already have a parsed AST to pass in.
libpg_query takes a string, parses it, records the location of any constants, then copies and manipulates that string to replace the constants.
In contrast, this implementation just copies the AST, walks it, replaces any constants with parameters, and changes the numbers on any parameters it encounters. In an effort to do this change in a single iteration over the parse tree, we don't preserve any bind parameters that were in the original query, which you can see reflected in the test that was updated.
On an extremely large query, the new implementation is 4.5x faster. If you include deparse time it is about twice as fast, and including parse