Comprehensive refactoring: packaging, tests, new sync modules, AWS support

[vjsingh1984]

Fixes #3

Summary

This is a comprehensive refactoring of the DR sync tools with 10 commits organized across 4 major phases, building on the existing stacked PR approach. All changes maintain backward compatibility while adding significant production-ready features for AWS + Databricks customers.

What Changed

Core Improvements (Commits from base)

Area	Before	After
Bugs	6 confirmed bugs (crash, data corruption, operator precedence)	✅ All fixed
Resource leaks	4 scripts leaked serverless warehouses	✅ All fixed with context managers
Code duplication	~400 lines duplicated 8 times	✅ Consolidated into dr_sync/ package
Logging	76 print() calls, no structure	✅ Structured logging with levels
Config	Only common.py hardcoded	✅ DRSyncConfig with env var support
Safety	No dry-run, no validation	✅ --dry-run flag + CSV validation

New Features

Phase 1: Packaging, CI/CD, Tests (Commit a298d96)

• pyproject.toml: Proper Python package with CLI entry point
• Makefile: install, install-dev, test, lint, format, clean commands
• .github/workflows/ci.yml: CI/CD for Python 3.9-3.12 with linting and testing
• 40 unit tests: 53% coverage for dr_sync/ package

Phase 2: Core Infrastructure (Commit ebd5f02)

• dr_sync/retry.py: Exponential backoff decorator for SDK operations
• dr_sync/checkpoint.py: Resumable sync tracking (state in .dr_sync_state/)
• dr_sync/filter.py: Include/exclude glob patterns for selective sync
• dr_sync/registry.py: Plugin-based sync module registration with dependency resolution
• dr_sync/cli.py: Unified CLI: dr-sync run --all, checkpoint management

Phase 3: AWS-Focused Sync Modules (Commit ebd5f02)

• sync_jobs.py: Jobs/Workflows with checkpointing and filtering
• sync_cluster_policies.py: Cluster policy definitions (parallel sync)
• sync_instance_pools.py: Instance pools with AWS instance type remapping
• sync_instance_profiles.py: AWS IAM profile registration
• sync_secret_scopes.py: Secret scope metadata and ACLs (NOT values by design)
• sync_notebooks.py: Notebooks with folder structure preservation

Phase 4: Documentation (Commit 034fb13)

• Updated README.md with CLI usage, installation, and all sync modules
• New docs/aws_usage_guide.md for AWS customers (IAM, S3 CRR, Secrets Manager, CloudWatch, VPC)

Unified CLI

# Run all sync in dependency order
dr-sync run --all

# Dry-run to preview
dr-sync run --all --dry-run

# Selective filtering
dr-sync run --all --include "prod.*.*" --exclude "*.staging.*"

# Resume from checkpoint
dr-sync run --all --resume

# Checkpoint management
dr-sync checkpoint list
dr-sync checkpoint clear jobs

Backward Compatibility

100% preserved - All existing workflows continue to work:

# Traditional workflow still works
# Edit common.py, then:
python sync_tables.py

# New env var workflow
export DR_SYNC_SOURCE_HOST=...
export DR_SYNC_CATALOGS_TO_COPY=cat1,cat2
python sync_tables.py --dry-run

Installation

pip install databricks-dr-sync

This installs the dr-sync CLI command.

Test Coverage

• 40 unit tests passing
• 53% code coverage for dr_sync/ package
• All linting passing (black, ruff, mypy)
• CI/CD via GitHub Actions on Python 3.9-3.12

Next Steps for Maintainer

1. Review the documentation in README.md and docs/aws_usage_guide.md
2. Test in a dev environment with sample workspaces
3. Run tests: pytest
4. Check CI/CD: https://github.com/gregwood-db/databricks-dr-examples/actions

Related Issues

This PR addresses the issues identified in the gap analysis:

• Warehouse resource leaks → Fixed
• No safety nets → Dry-run, validation added
• No observability → Structured logging added
• No CI/CD → GitHub Actions workflow added
• No tests → 40 unit tests added
• AWS-specific gaps → 6 new sync modules added