Skip to content

fix(sanitize): anchor sandbox-comment regex to avoid dropping real rows - #110

Merged
ddtcorex merged 1 commit into
masterfrom
fix/sql-sanitize-sandbox-false-positive
Aug 5, 2026
Merged

fix(sanitize): anchor sandbox-comment regex to avoid dropping real rows#110
ddtcorex merged 1 commit into
masterfrom
fix/sql-sanitize-sandbox-false-positive

Conversation

@ddtcorex

@ddtcorex ddtcorex commented Aug 5, 2026

Copy link
Copy Markdown
Owner

Summary

  • sync --db and db import --stream-db were silently dropping rows because
    SanitizeSQLDump's sandbox-comment regex (.*999999.*sandbox.*) matched any
    dump line containing both substrings, not just the specific mysqlbinlog artifact
    comment (/*!999999\- enable the sandbox mode */) it was meant to strip.
  • Since mysqldump --extended-insert (the default here) puts a whole table's rows
    on one line, real data containing "999999" (e.g. an ID) and "sandbox"
    (e.g. a PayPal/Braintree environment setting) anywhere on that line caused the
    entire INSERT statement to be dropped with no error.
  • db import -f <file> doesn't sanitize file-based imports at all, which is why it
    imported full data while the streaming paths (sync --db, db import --stream-db)
    came up short — same root symptom, same root cause.
  • Anchored sqlSandboxPattern to the real /*!999999 ... */ comment delimiters, and
    changed the fast-path substring gate from "sandbox" to the distinctive
    "/*!999999" marker.

Closes #109

Test plan

  • Added regression test reproducing the false-positive drop on a synthetic
    --extended-insert-style line (tests/sql_sanitize_test.go)
  • Updated the existing "Remove Sandbox lines" test to use the real comment
    format (/*!999999\- enable the sandbox mode */) and confirmed it's still
    dropped
  • go build ./..., go vet ./..., go test ./... all pass
  • gofmt -s -l . shows no drift on changed files

The sandbox-comment sanitizer regex (`.*999999.*sandbox.*`) matched any
line containing both substrings anywhere, not just the mysqlbinlog
"sandbox mode" artifact comment it was meant to strip. Since mysqldump
uses --extended-insert by default, a whole table's rows can share one
dump line, so real data containing "999999" (e.g. an ID) and "sandbox"
(e.g. a PayPal/Braintree environment setting) anywhere on that line
caused SanitizeSQLDump to silently drop the entire INSERT statement.

This explained why `govard sync --db` (which sanitizes the stream)
imported less data than `db dump` + `db import -f` (which doesn't
sanitize file-based imports).

Anchor the pattern to the actual `/*!999999 ... */` comment delimiters
so it only matches the intended artifact.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@ddtcorex
ddtcorex merged commit 5d6181d into master Aug 5, 2026
7 checks passed
@ddtcorex
ddtcorex deleted the fix/sql-sanitize-sandbox-false-positive branch August 5, 2026 17:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sync --db / stream-db import silently drops rows via over-broad sandbox sanitize regex

1 participant