Draft
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
alexrichey
force-pushed
the
ar-dbtify-pluto-staging-models
branch
3 times, most recently
from
e6cb9d5 to
eb97be2
Compare
all our scripts are expecting the command with the prefix, so...
I keep mine outside the data engineering dir
See comment for rationale
- Created 4 additional staging models: * stg__pluto_input_research.sql * stg__pluto_pts.sql * stg__dcp_mappluto.sql * stg__previous_pluto.sql - Fixed remaining non-stg__ references in 13 SQL files - All source tables now consistently use staging models - Total staging models: 40 (up from 36)
- Created 01a_dbt_staging.sh to run dbt staging models - Script runs between data load and legacy SQL build - Materializes 40 staging models before 02_build.sh runs - Added pluto_build/README.md documenting build sequence - Legacy SQL can now reference stg__ tables
- Moved 9 CSV files from pluto_build/data/ to seeds/ - Configured seeds in dbt_project.yml (+quote_columns, +schema: public) - Documented all seeds in seeds/_seeds.yml - Updated 01a_dbt_staging.sh to run 'dbt seed' before staging models - Deleted 01_load_local_csvs.sh (replaced by dbt seed) - Deleted sql/_create.sql (replaced by dbt seed) - Updated README.md with seed documentation - No SQL changes needed - seeds create same table names
- Update GitHub workflow to call 01a_dbt_staging.sh instead of removed 01_load_local_csvs.sh - Remove duplicate dbt seed call from 07_custom_qaqc.sh to avoid reloading seeds - Seeds are now loaded exactly once via 01a_dbt_staging.sh Closes data-engineering-n58.3
- Add --profiles-dir . to all dbt commands in 01a_dbt_staging.sh and 07_custom_qaqc.sh - Move 'cd ..' before dbt deps/debug in 01a_dbt_staging.sh - Fix schema config deprecation in dbt_project.yml (add + prefix to tests.schema) - Ensures dbt uses local profiles.yml in GHA workflows
- Removed duplicate pluto_pts entry (was in recipe_sources twice) - Removed duplicate dcp_zoningdistricts entry (was in recipe_sources and build_sources) Fixes dbt compilation error about duplicate source names
- Add column_types config for ignored_bbls_for_unit_count_test (bbl, pluto_version as text) - Add column_types config for pluto_input_research (bbl as text) - Add column_types config for pluto_input_condolot_descriptiveattributes (condno, parid as text) - Remove incorrect column_types from zoning_district_class_descriptions - Fixes 'integer out of range' errors when loading BBL values
- Change condno -> CondNO and parid -> PARID to match CSV header - Fixes integer out of range error
- Change seeds schema from 'public' to BUILD_ENGINE_SCHEMA to match build scripts - Update stg__pluto_input_research to reference seed with ref() instead of source() - Ensures build scripts can find seed tables in the correct schema
- Seeds were loading to doubled schema (target_schema + custom_schema) - dbt automatically uses BUILD_ENGINE_SCHEMA from profiles.yml as target - Removing +schema config fixes: ar_dbtify_pluto_staging_models_ar_dbtify_pluto_staging_models -> ar_dbtify_pluto_staging_models - Matches green_fast_track pattern Closes data-engineering-n58.5
alexrichey
force-pushed
the
ar-dbtify-pluto-staging-models
branch
from
11b8e89 to
26e2b59
Compare
alexrichey
force-pushed
the
ar-dbtify-pluto-staging-models
branch
from
dc48cde to
27253fb
Compare
- Use SCRIPT_DIR to find bash/config.sh relative to script location - Remove 'cd ..' and 'cd pluto_build' navigation - Fix column name case in _seeds.yml (CondNO -> condno, PARID -> parid) - Script now runs successfully from products/pluto directory
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.