Add maildat columns to default.vw_pin_address#980
Add maildat columns to default.vw_pin_address#980wrridgeway wants to merge 11 commits intomasterfrom
default.vw_pin_address#980Conversation
|
@ccao-jardine Are you okay with the taxpayer/owner dichotomy I've gone with here? |
|
@wrridgeway, yes! This is much clearer, provides frequently requested data, and provides more information while also remaining transparent about ongoing data quality issues. |
jeancochrane
left a comment
There was a problem hiding this comment.
Looking great! A few small suggestions, plus one request to clean up the docs. I'll give this a final look once that's done.
|
|
||
| data_tests: | ||
| - unique_combination_of_columns: | ||
| name: iasworld_maildat_unique_by_parid_taxyr |
There was a problem hiding this comment.
[Nitpick, optional] For extra clarity:
| name: iasworld_maildat_unique_by_parid_taxyr | |
| name: iasworld_maildat_unique_by_parid_taxyr_mailseq |
| - Taxpayer mailing addresses are not necessarily the same as property owner | ||
| mailing addresses. |
There was a problem hiding this comment.
[Nitpick, optional] A pointer might help future readers:
| - Taxpayer mailing addresses are not necessarily the same as property owner | |
| mailing addresses. | |
| - Taxpayer mailing addresses are not necessarily the same as property owner | |
| mailing addresses. For property owner addresses, see `iasworld.owndat`. |
| - Property owner mailing addresses are not necessarily the same as taxpayer | ||
| mailing addresses. |
There was a problem hiding this comment.
[Nitpick, optional] Same here:
| - Property owner mailing addresses are not necessarily the same as taxpayer | |
| mailing addresses. | |
| - Property owner mailing addresses are not necessarily the same as taxpayer | |
| mailing addresses. For taxpayer addresses, see `iasworld.maildat`. |
| data_tests: | ||
| - unique_combination_of_columns: | ||
| name: iasworld_maildat_unique_by_parid_taxyr | ||
| combination_of_columns: | ||
| - parid | ||
| - taxyr | ||
| - mailseq | ||
| additional_select_columns: | ||
| - column: who | ||
| alias: who | ||
| agg_func: array_agg | ||
| - column: wen | ||
| alias: wen | ||
| agg_func: array_agg | ||
| config: &unique-conditions | ||
| where: | | ||
| CAST(taxyr AS int) BETWEEN {{ var('data_test_iasworld_year_start') }} AND {{ var('data_test_iasworld_year_end') }} | ||
| AND cur = 'Y' | ||
| AND deactivat IS NULL | ||
| meta: | ||
| description: maildat should be unique by parid and taxyr |
There was a problem hiding this comment.
[Thought, non-blocking] Just a flag that this test won't run as part of our weekly data integrity test suite, since tests on iasWorld models get filtered into a manual process that follows the town open/close cycle. I doubt the stakeholders that review our iasWorld tests will pay attention to this error, unfortunately. I don't mind leaving the test in as a no-op for the sake of explaining the table's uniqueness condition to readers, but if we want to be proactively notified of failures, we should think of a way to move it to a downstream view so that it will get picked up by our weekly data integrity tests.
|
|
||
| columns: | ||
| - name : addrtype | ||
| description: Address type code (N, S, F, R, C) |
There was a problem hiding this comment.
[Suggestion, required] It seems like many of these columns are shared with owndat, and they already have docs definitions in dbt/models/iasworld/columns.md. Do you mind taking a minute to replace these descriptions with {{ docs() }} calls wherever documentation already exists in columns.md, or wherever a column exists on owndat such that its description can be moved to columns.md?
This PR adds
iasworld.maildattodefault.vw_pin_addressas the new source ofmail_columns. We keep columns fromiasworld.owndat, but we rename these columns with anowner_prefix. This allows us to avoid committing entirely to one table or the other and provide as much information to users as possible while being clear about the limitations of both sources of address data.The only real nuance here is that
maildatis not unique by pin and year whencur = 'Y' and deactivat is nulllike most iasworld tables. Instead we also need to depend onmailseqand choosing which value ofmailseqto privilege in order to makemaildatunique by pin and year (we take the max value) boiled down to trying to match up with what's displayed on the treasurer's website. Two example pins are below:Updates have not changed row count, as expected:
returns nothing. and
mail_address_fullis largely non-null: