Skip to content

Add ingestion workflow logic#505

Open
vish-cs wants to merge 1 commit into
datacommonsorg:masterfrom
vish-cs:dcp
Open

Add ingestion workflow logic#505
vish-cs wants to merge 1 commit into
datacommonsorg:masterfrom
vish-cs:dcp

Conversation

@vish-cs
Copy link
Copy Markdown
Contributor

@vish-cs vish-cs commented May 11, 2026

Code under workflow and Terraform folders is copied as-is from the data repo as part of the migration.
Changes are made to pom.xml and SpannerClient.java to read the schema file from the ingestion-helper.

@vish-cs vish-cs requested a review from gmechali May 11, 2026 18:01
@vish-cs vish-cs marked this pull request as draft May 11, 2026 18:01
@codacy-production

This comment was marked as outdated.

gemini-code-assist[bot]

This comment was marked as outdated.

@vish-cs vish-cs marked this pull request as ready for review May 14, 2026 09:44
@vish-cs vish-cs changed the title WIP: add ingestion workflow logic Add ingestion workflow logic May 14, 2026
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements a comprehensive import automation and Spanner ingestion pipeline. Key additions include Terraform configurations for GCP infrastructure, Cloud Workflows for process orchestration, and two Python-based helper services for managing ingestion tasks and Pub/Sub events. The code review identified several high-priority concerns, including a violation of the principle of least privilege in IAM role assignments and a security vulnerability in the JWT verification logic. Additionally, the reviewer noted the use of placeholder SQL in the aggregation utility, non-idiomatic flag parsing in Cloud Functions, and recommended enabling deletion protection for the Spanner database.

Comment thread pipeline/terraform/main.tf
Comment thread pipeline/workflow/ingestion-helper/import_utils.py
Comment thread pipeline/terraform/main.tf
Comment thread pipeline/workflow/ingestion-helper/aggregation_utils.py
Comment thread pipeline/workflow/ingestion-helper/main.py
Code is being migrated from the data repo to import repo
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant