Merged
Conversation
Use PointID as the location description field to satisfy the NOT NULL constraint on location.description. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
When TRANSFER_LIMIT was set to -1 (meaning "no limit"), the condition `if limit and i >= limit` evaluated to True on first iteration because -1 is truthy and 0 >= -1. This caused transfers to exit immediately without processing any records. Changed condition to `if limit > 0 and i >= limit` (or with explicit None check) so that -1 and 0 correctly mean "no limit". 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Different wells (things) can have observation blocks with the same review_status, parameter_id, and overlapping time ranges. The previous unique constraint on (review_status, parameter_id, start_datetime, end_datetime) caused duplicate key errors. Changes: - Add thing_id foreign key to TransducerObservationBlock - Add Thing relationship for navigation - Update unique constraint to include thing_id - Update transfer code to extract thing_id from deployment before creating blocks 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Legacy data from MS Access may not have descriptions for all locations. Making this field nullable allows the transfer to proceed without requiring a description value. Changes: - Update Location model to set nullable=True on description - Add migration to alter the column constraint 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Configure SQLAlchemy connection pooling with configurable pool_size and max_overflow settings via environment variables. This enables concurrent database connections required for parallel transfer operations. - Add DB_POOL_SIZE and DB_MAX_OVERFLOW env vars (defaults: 10, 20) - Enable pool_pre_ping to verify connections before use - Apply to both CloudSQL and local PostgreSQL engines 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Restructure transfer pipeline to execute independent transfers concurrently using ThreadPoolExecutor. Transfers are organized into phases: Phase 1: Foundational (AquiferSystems, GeologicFormations) - parallel Phase 2: Wells - supports parallel mode via TRANSFER_PARALLEL_WELLS Phase 3: Group 1 (Screens, Contacts, WaterLevels, etc.) - parallel Phase 4: Sensors - sequential (required before continuous water levels) Phase 5: Group 2 (Pressure, Acoustic) - parallel Add helper functions for thread-safe transfer execution with timing. Retain sequential mode via TRANSFER_PARALLEL=0 for compatibility. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Implement row-level parallelization for wells transfer using batch processing with ThreadPoolExecutor. Key optimizations: - transfer_parallel(): Splits wells into batches across workers - _step_parallel_complete(): Creates well + ALL dependent objects (notes, status, provenances, measuring points, formation zone) in a single pass, eliminating the sequential after_hook bottleneck - Thread-safe aquifer handling with minimal lock contention - Pre-load formations per batch to avoid race conditions Performance: 9,887 wells in ~2 minutes (vs ~56 minutes with after_hook) Throughput: ~21 wells/sec with 4 workers Enable via TRANSFER_PARALLEL_WELLS=1, configure workers with TRANSFER_WORKERS (default: 4). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…hod for compatibility
…ion-ms-access-jir Feature/admin location ms access jir
…taging abd testing
…leaner configuration
…eamline upgrade process
…ily table and make location.description nullable
…odel for migration
…itional table creation
…ls-continuous-pressure-daily Implement backfill for legacy WaterLevelsContinuous_Pressure_Daily table
…ls-continuous-pressure-daily feat: reorder Google Cloud authentication step in staging workflow
…ls-continuous-pressure-daily migration/waterlevels-continuous-pressure-daily
fix: comment out pytest hook in pre-commit configuration
…hy.csv into NMA_Stratigraphy
BDMS-471,472,473
BDMS-471-472-473
BDMS-471-472-473
BDMS-471-472-473
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR implements a test release (version 0.1) by adding comprehensive legacy data transfer functionality, improving logging, and enhancing the CLI with water level bulk upload capabilities.
Changes:
- Added multiple new legacy data transfer modules for chemistry, surface water, soil/rock results, and NGWMN views
- Enhanced logging configuration and improved transfer completion tracking across existing transfer modules
- Implemented CLI water level bulk upload functionality with CSV parsing and validation
Reviewed changes
Copilot reviewed 167 out of 184 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| transfers/thing_transfer.py | Added logging for transfer start/completion and improved limit condition checking |
| transfers/surface_water_photos.py | New transferer for SurfaceWaterPhotos legacy data |
| transfers/surface_water_data.py | New transferer for SurfaceWaterData legacy table |
| transfers/stratigraphy_transfer.py | Updated logging format and improved limit condition |
| transfers/stratigraphy_legacy.py | New transferer for NMA_Stratigraphy legacy table |
| transfers/soil_rock_results.py | New transferer for Soil_Rock_Results legacy data |
| transfers/sensor_transfer.py | Added meteorological/environmental sensor types to mapping |
| transfers/seed.py | Added prerequisite checks, skip-if-exists flag, and improved import organization |
| transfers/radionuclides.py | New transferer for Radionuclides legacy data with extensive documentation |
| transfers/permissions_transfer.py | Added logging and record counting for permissions transfer |
| transfers/ngwmn_views.py | New transferer for NGWMN view tables (well construction, water levels, lithology) |
| transfers/minor_trace_chemistry_transfer.py | New transferer for MinorTraceChemistry with ChemistrySampleInfo linkage |
| transfers/metrics.py | Added metrics functions for new legacy tables |
| transfers/major_chemistry.py | New transferer for MajorChemistry legacy data |
| transfers/logger.py | Enhanced logger configuration with force=True and explicit INFO level |
| transfers/hydraulicsdata.py | New transferer for HydraulicsData legacy table |
| transfers/geologic_formation_transfer.py | Updated logging format and improved limit condition |
| transfers/chemistry_sampleinfo.py | New transferer for ChemistrySampleInfo with Thing relationship validation |
| transfers/backfill/backfill.py | New backfill orchestration module for CD workflows |
| transfers/backfill/init.py | Empty init file for backfill package |
| transfers/associated_data.py | New transferer for AssociatedData legacy table |
| transfers/asset_transfer.py | Refactored to use new upload_and_associate helper function |
| transfers/aquifer_system_transfer.py | Updated logging format and improved limit condition |
| tests/test_*.py | Multiple new test files for legacy models and functionality |
| tests/features/*.feature | New and updated feature files for BDD tests |
| tests/features/steps/*.py | New step implementations for CLI and water level features |
| tests/conftest.py | Major test configuration improvements with session-scoped DB setup |
| tests/init.py | Refactored parameter lookup to use cached function |
| services/water_level_csv.py | New service for water level CSV bulk upload |
| services/util.py | Enhanced HTTP utilities with retry logic and improved error handling |
| services/ngwmn_helper.py | New helper for NGWMN XML response generation |
| services/geospatial_helper.py | Import reorganization |
| services/gcs_helper.py | Added bucket parameter and extracted blob name/URI generation |
| services/asset_helper.py | New helper for asset upload and association |
| showcase_pointids | New file listing showcase point IDs |
| docker-compose.yml | Added POSTGRES_PORT environment variable |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…handling of missing points
feat: improve data handling in util.py by grouping and sorting measurements
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
This PR addresses the following problem / context:
How
Implementation summary - the following was changed / added / removed:
Notes
Any special considerations, workarounds, or follow-up work to note?