Skip to content

Apply v1.0.19#178

Merged
nettrash merged 4 commits intomasterfrom
v1.0.19
May 3, 2026
Merged

Apply v1.0.19#178
nettrash merged 4 commits intomasterfrom
v1.0.19

Conversation

@nettrash
Copy link
Copy Markdown
Owner

@nettrash nettrash commented May 3, 2026

Bug fixes:

  • Refactored comment-related query construction into dedicated
    helper methods on Dump:
    build_tables_standalone_query,
    build_regular_views_query,
    build_materialized_views_query,
    build_view_column_comments_query,
    build_sequences_standalone_query,
    and on Table: build_indexes_bulk_query.
  • Kept pg_description joins scoped to relation comments by
    filtering on pg_description.classoid = 'pg_class'::regclass
    and pg_description.objsubid = 0 to avoid catalog
    collisions.
  • Normalized SQL formatting in the new helper query strings.
  • Comparer::compare_grants no longer runs a duplicate
    iter().find(...) over from_cols for every TO column
    whose own ACL is empty. The first scan was discarded
    and the second was redone unconditionally; both are
    now replaced by a single per-table
    HashMap<&str, &[String]> lookup, which also reduces
    the column-grants emission from O(C^2) to O(C) on
    wide tables.
  • Comparer::mark_serial_columns now keys
    serial_columns by (schema, table, column) tuple
    instead of a format!("{}.{}.{}", ...) string that
    was parsed back via splitn(3, '.'). The string
    form silently misparsed any identifier containing a
    literal . (legal in PostgreSQL when quoted),
    leaving the corresponding column unmarked.
  • Comparer::compare_sequences had a dead
    dropped_sequences: HashSet<String> that was
    checked and updated on every iteration of a loop
    over self.from.sequences. Sequences are unique by
    (schema, name) so the dedupe could never fire;
    the set and its per-iteration format!/contains/
    insert calls are removed.
  • Comparer::compare_routines and
    Comparer::compare_routines_and_views no longer
    clone every Routine (each carries the full
    source_code string). The drop path now holds
    routines_to_drop: Vec<&Routine> borrowing into
    self.from.routines. The create/update path —
    previously forced to clone because
    process_target_routine took &mut self and so
    conflicted with any borrow into self.from /
    self.to — is unblocked by refactoring that
    method into a free associated function
    Self::emit_routine_diff(&mut script, use_drop, routine, from_routine). Disjoint-field split
    borrows now allow &mut self.script to coexist
    with &Routine borrows out of the dump fields,
    removing the per-emit clones. Same pattern as the
    pre-existing Self::emit_drop.

Performance:

  • Dump::process no longer materializes the entire
    serialized dump as a String before handing it to
    the zip writer. The JSON payload is now streamed
    into ZipWriter via serde_json::to_writer
    through a 256 KiB BufWriter, bounding peak
    memory at the buffer plus zlib's internal state
    rather than ~2x the uncompressed dump size. The
    BufWriter is required for speed: to_writer
    emits one write per JSON token, and feeding those
    straight into the deflate stream paid a per-call
    cost on every one (an early unbuffered version of
    this change made dumps roughly 10x slower). The
    write path is exposed as Dump::write_to_file,
    mirroring the existing Dump::read_from_file and
    making the round-trip directly testable.

Tests:

  • Added regression tests proving every new query
    builder includes the pg_class classoid filter:
    build_tables_standalone_query_filters_by_pg_class,
    build_regular_views_query_filters_by_pg_class,
    build_materialized_views_query_filters_by_pg_class,
    build_view_column_comments_query_filters_by_pg_class,
    build_sequences_standalone_query_filters_by_pg_class,
    and build_indexes_bulk_query_filters_by_pg_class.
  • write_to_file_round_trips_via_read_from_file
    builds a Dump with schemas, extensions, tables,
    views, sequences, and routines, writes it via the
    new streaming path, reads it back, and asserts
    every collection size and a few content fields
    match.
  • compare_column_grants_dispatches_per_column_acl_correctly
    builds a table with three columns whose effective
    from_acl differs (kept / revoked / newly
    granted) and verifies that the per-table
    column-ACL HashMap dispatches each column to the
    right diff outcome — guarding against off-by-one
    mistakes that single-column tests would miss.
  • mark_serial_columns_handles_dotted_identifier_names
    drives the mark_serial_columns path with a
    schema, table, and column name that all contain a
    literal ., asserting that the new tuple key
    still locates the target column. The pre-fix
    splitn(3, '.') parser would have failed on this
    input.

iBarBuba and others added 3 commits May 1, 2026 20:42
* Fix pg_description joins with classoid

* Add regression tests for pg_class classoid joins

* Fix remaining pg_class-backed description joins

* chore: remove trailing spaces from classoid filter query

* Fix style of sequence/index pg_class query builders

* chore(changelog): add v1.0.19 bug fix entry for pr 177

---------

Co-authored-by: iBarBuba <350579+iBarBuba@users.noreply.github.com>
@nettrash nettrash added this to the v1.0.19 milestone May 3, 2026
@nettrash nettrash self-assigned this May 3, 2026
Copilot AI review requested due to automatic review settings May 3, 2026 11:57
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR applies v1.0.19, focusing on fixing PostgreSQL catalog comment join collisions, improving dump-write performance via streaming ZIP serialization, and reducing comparer overhead on grants/serial detection/routine handling.

Changes:

  • Scoped pg_description joins to pg_class (classoid + objsubid) and refactored several SQL builders into helper methods with normalized formatting.
  • Streamed dump serialization directly into ZipWriter through a buffered writer to reduce peak memory and improve throughput.
  • Reduced comparer overhead (column grants, serial column keys, routines/sequences handling) and added regression tests for the above.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
app/src/dump/table.rs Adds pg_class scoping for column/index comment joins; extracts index query builder + regression test.
app/src/dump/core.rs Streams dump JSON into ZIP via BufWriter; extracts multiple query builders + adds regression tests.
app/src/comparer/core.rs Optimizes grants dispatch, serial column keying, and reduces routine/sequences overhead.
app/src/comparer/core_tests.rs Adds regression tests for column-grants dispatch and dotted identifiers in serial detection.
app/Cargo.toml Bumps crate version to 1.0.19.
CHANGELOG Documents v1.0.19 release notes.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread app/src/dump/core.rs
Comment thread app/src/comparer/core.rs
Comment thread app/src/dump/core.rs Outdated
@nettrash nettrash merged commit 0be4a6a into master May 3, 2026
16 checks passed
@nettrash nettrash deleted the v1.0.19 branch May 3, 2026 12:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants