Data migrations framework + new site-type transfers, Typer CLI, and NMA notes backfill#464
Data migrations framework + new site-type transfers, Typer CLI, and NMA notes backfill#464
Conversation
jirhiker
requested review from
Copilot,
kbighorse and
ksmuczynski
and removed request for
Copilot
There was a problem hiding this comment.
Pull request overview
This PR introduces a structured data migrations framework tied to Alembic revisions, migrates the CLI from Click to Typer, and implements a data migration to move Location.nma_location_notes into the polymorphic Notes table. It also adds site-type transfer functions for non-well locations.
Changes:
- Implemented a repeatable data migration framework with registry, runner, history tracking, and CLI commands
- Added transfer functions for six new site types (rock sample, diversion, lake/pond, soil gas, outfall, other)
- Migrated CLI from Click to Typer with new
data-migrationscommand group andalembic-upgrade-and-datacommand
Reviewed changes
Copilot reviewed 14 out of 15 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
tests/test_thing_transfer.py |
Tests for new site-type transfer helper functions |
tests/test_data_migrations_cli.py |
Unit tests for data migration CLI commands (list, run, run-all) |
tests/test_data_migrations.py |
Tests for NMA location notes migration including idempotency checks |
tests/test_cli_commands.py |
Updated import from Click to Typer testing utilities |
pyproject.toml |
Added typer dependency |
data_migrations/runner.py |
Core runner logic with history table, status tracking, and migration execution |
data_migrations/registry.py |
Auto-discovery and registration of data migrations |
data_migrations/migrations/_template.py |
Template for creating new data migrations |
data_migrations/migrations/__init__.py |
Package initialization file |
data_migrations/migrations/20260205_0001_move_nma_location_notes.py |
Migration to backfill legacy location notes |
data_migrations/base.py |
Base DataMigration dataclass definition |
data_migrations/__init__.py |
Package initialization file |
cli/cli.py |
Refactored from Click to Typer with new data migration commands |
AGENTS.MD |
Added documentation on data migration idempotency requirements |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 060e7d56e9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
jirhiker
changed the title
| MIGRATION = DataMigration( | ||
| id="YYYYMMDD_0000", | ||
| alembic_revision="REVISION_ID", | ||
| name="Short migration name", | ||
| description="Why this data migration exists.", | ||
| run=run, | ||
| is_repeatable=False, | ||
| ) |
There was a problem hiding this comment.
I like the template a lot! It'll help moving forward. Is alembic_revision the ID of the alembic head script at the moment the migration is defined? So that we can identify at what stage in the DB's life the migration was made
There was a problem hiding this comment.
yep thats it exactly.
jacob-a-brown
left a comment
jacob-a-brown
left a comment
There was a problem hiding this comment.
Overall I think it looks good! And like the data migration tracking. If nma_location_notes is being moved to the polymorphic Notes table should that field be removed from the Location table?
|
Nma_location_notes will be removed in a subsequent alembic migration |
2 participants
Summary
Location.nma_location_notesinto the polymorphicNotestable.Key Changes
Data migrations
data_migrations/framework with registry, runner, history table, and Typer CLI.data_migrations/migrations/_template.py.20260205_0001_move_nma_location_notes.Transfers
LU_SiteTypeentries.CLI
data-migrationscommand group andalembic-upgrade-and-data.Tests
Behavioral Notes
nma_location_notes.How to Test
source .venv/bin/activatepytest tests/test_thing_transfer.py tests/test_data_migrations.py tests/test_data_migrations_cli.py tests/test_cli_commands.pyFollow-ups