feat: issue with relation.schema in the S3 path when a custom location is set

[akmalsoliev]

resolves #667

Description

Please refer to #667 for more information.

This PR also includes simplification of location construction, certain sections of the code were hard coding the location, now it refers to _build_location

I rant this on personal AWS and works like a charm.

Checklist

• I have signed the CLA
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• I have updated the CHANGELOG.md and added information about my change to the "dbt-glue next" section.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[akmalsoliev]

Hey @yotahk can we kick CI for this one?

[yotahk]

@akmalsoliev Added one comment, would you check before kicking CI?

[akmalsoliev]

@akmalsoliev Added one comment, would you check before kicking CI?

✅ Done

[akmalsoliev]

Hey @yotahk, I noticed that the fail had nothing to do with my PR, a simple indentation error, want me to fix it?

UPDATE: Just realized that git blame is @akmalsoliev 🤫 Don't tell the author that he introduced this error 🤣. I'll fix this.

[yotahk]

Oof... appreciate the cleanup :)

[akmalsoliev]

Oof... appreciate the cleanup :)

Fixed now.

[yotahk]

Looks like CI is still failing with IndentationError. Do you have pre-commit configured in your local environment? We should be able to capture linter/formatter errors before committing.
https://pre-commit.com/

[akmalsoliev]

Looks like CI is still failing with IndentationError. Do you have pre-commit configured in your local environment? We should be able to capture linter/formatter errors before committing. https://pre-commit.com/

sadly I'm unable to reproduce the error locally, so attempting couple of fixes, if this time doesn't work I will just create a temporary aws account for this. I hope the issue was caused by jinja's if else indentation, let's give it a last chance and if doesn't work will go with fresh aws account.

[akmalsoliev]

[akmalsoliev]

hey @yotahk, started with a fresh account, ran all the test, nothing fails. Could you check on your end if arn and s3 location DBT_GLUE_ROLE_ARN, DBT_S3_LOCATION isn't expired (github secrets)?

[yotahk]

Investigation:

• ci: use default checkout ref in PR workflows #678 changed the CI workflow to checkout main branch (default ref) - after this change, CI for integration.yml python-models.yml s3-tables.yml started referencing main branch, not PR branch
• fix: incremental append and partition_by support for Iceberg Python #662 introduced the problematic syntax. However the CI in the PR couldn't detect syntax error since it's referencing main branch and the problematic syntax was merged to main
• feat: issue with relation.schema in the S3 path when a custom location is set  #683 (this PR) some CI is also checking out from main, causing syntax failure in unit test

@akmalsoliev the changes in this PR looks good, however the CI configuration change introduced in #678 seems to be the root of unit test failure. I'll check the background and update again.