Refactor: bug fixes, resource safety, shared utilities, logging, validation, and performance

[vjsingh1984]

Fixes #3

Summary

This PR is a comprehensive refactoring of the DR sync scripts, addressing 6 confirmed bugs, warehouse resource leaks, massive code duplication, zero observability, no safety nets, and performance bottlenecks — all while preserving full backward compatibility.

Existing users editing common.py and running python sync_*.py will not be disrupted.

The changes are organized into 8 atomic commits (reviewable independently):

#	Commit	What it does
1	Fix 6 confirmed bugs	Fixes crash bug in sync_grs_ext.py, data corruption in sync_views.py, operator precedence bug, missing f-strings, example list bug
2	Fix warehouse resource leaks	managed_warehouse() context manager guarantees cleanup; sys.exit() replaced with RuntimeError
3	Add dr_sync/ shared utilities	Extracts ~400 lines of duplicated code into reusable modules (sql_utils, csv_mapping, thread_utils, workspace, exceptions)
4	Add DRSyncConfig	Dataclass with from_common_module() (backward compat) and from_env() (secure tokens via env vars)
5	Add structured logging	Replaces all 76 print() calls with logger.info/warning/error across 9 scripts
6	Add validation and --dry-run	CSV validation, --dry-run flag, --log-level CLI support for all scripts
7	Improve performance	O(1) dict lookups, Spark SQL filter pushdown, parallel schema creation
8	Apply code formatting	black formatting, ruff config for Databricks builtins, import order fixes

---

Why Each Change Matters

Bug Fixes (Commit 1)

The most critical bug is in sync_grs_ext.py: loaded_table_types is initialized as an empty list but never populated in the loop body, causing a DataFrame column length mismatch that crashes the script when writing the status table. The operator precedence bug in sync_views.py can cause system tables to be synced or user tables to be skipped.

Warehouse Leaks (Commit 2)

Every run of sync_tables.py leaks 2 serverless warehouses. Three other scripts leak 1 each. Over repeated DR runs, this accumulates orphaned warehouses that incur cost and clutter. The new managed_warehouse() context manager wraps creation in try/finally — even if the script crashes, the warehouse is deleted.

Shared Utilities (Commit 3)

The SQL polling loop was copied 8 times across 4 files. When the sync_grs_ext.py copy drifted (missing f-string), only that copy broke — a classic duplication hazard. Consolidating into dr_sync/sql_utils.py means one implementation to maintain, test, and improve (e.g., the shared version adds timeout protection and statement cancellation that none of the originals had).

Configuration (Commit 4)

DRSyncConfig.from_common_module() reads the existing common.py — zero disruption. But from_env() provides a secure alternative: DR_SYNC_SOURCE_TOKEN as an env var instead of a checked-in file. Scripts auto-detect which to use.

Logging (Commit 5)

DR operations are high-stakes. When something fails at 3 AM, operators need timestamps, log levels, and the ability to grep for ERROR — not undifferentiated print() output. All 76 calls are now properly leveled (info for progress, warning for skips, error for failures).

Dry-Run (Commit 6)

python sync_tables.py --dry-run shows every table that would be cloned, every warehouse that would be created, every external table that would be dropped — without executing anything. This lets operators validate their configuration before committing to a DR sync. CSV validation catches mapping file errors (duplicates, missing columns, empty values) before execution begins.

Performance (Commit 7)

For workspaces with many credentials/locations, the O(n)→O(1) lookup improvement is meaningful. Pushing SQL filters down to Spark eliminates transferring the entire system.information_schema.tables to Python. Parallel schema creation speeds up the catalog/schema sync phase.

---

New Files

File	Purpose
dr_sync/__init__.py	Package init with public API re-exports
dr_sync/sql_utils.py	execute_statement_sync(), managed_warehouse(), drop_table_if_exists()
dr_sync/config.py	DRSyncConfig dataclass
dr_sync/csv_mapping.py	CSV loading, validation, safe lookups
dr_sync/thread_utils.py	parallel_map() with as_completed(), ProgressCounter
dr_sync/workspace.py	create_client() factory
dr_sync/exceptions.py	DRSyncError hierarchy
dr_sync/log.py	setup_logging()
.env.example	Environment variable template
ruff.toml	Ruff config for Databricks builtins

How to Verify

# Pre-commit checks
pre-commit run --all-files  # black, ruff, mypy all pass

# Dry-run any script
python sync_tables.py --dry-run
python sync_creds_and_locs.py --dry-run --log-level DEBUG

# Existing workflow (backward compatible)
# Edit common.py as before, then:
python sync_catalogs_and_schemas.py

# New env-var workflow
export DR_SYNC_SOURCE_HOST=...
export DR_SYNC_SOURCE_TOKEN=...
python sync_tables.py --dry-run

Test Plan

• pre-commit run --all-files passes (black, ruff, mypy)
• Each script runs with --dry-run and logs planned operations
• sync_creds_and_locs.py creates credentials/locations in a test workspace
• sync_catalogs_and_schemas.py creates catalogs/schemas in a test workspace
• sync_tables.py deep clones tables end-to-end
• No orphaned warehouses remain after script completion
• common.py workflow: edit common.py, run script, verify it reads config
• Env-var workflow: set DR_SYNC_* vars, run script, verify it uses them