Skip to content

feat(billing): GCP cloud billing via BigQuery export#237

Open
PatrickCmd wants to merge 14 commits into
mainfrom
feat/gcp-cost-billing
Open

feat(billing): GCP cloud billing via BigQuery export#237
PatrickCmd wants to merge 14 commits into
mainfrom
feat/gcp-cost-billing

Conversation

@PatrickCmd

Copy link
Copy Markdown
Contributor

Summary

Adds GCP cost/usage billing as a second provider under the existing Cloud category on the admin billing dashboard (/admin/billing), alongside AWS. GCP has no on-demand cost API, so data comes from the Cloud Billing export in BigQuery (gcp_billing_export_v1_*), queried with parameterized SQL, reported as net cost (cost + credits), broken down by service + SKU. Slots into the same provider-agnostic BillingRecord → aggregation → caching → router pipeline built for AWS/Vast.ai — GCP is just a new provider mapped to the cloud category.

Third and final provider in the billing suite: Inference (Runpod + Modal)Training (Vast.ai)Cloud (AWS + GCP).

What's included

  • GcpAnalyticsProvider (app/integrations/billing/gcp.py) — parameterized BigQuery query grouped by time-bucket + service.description + sku.description; net cost incl. credits; HOUR/DAY truncation (no 14-day limit — that was AWS-specific); blocking client run via asyncio.to_thread.
    • Security/safety: table name comes only from trusted config, validated (^[A-Za-z0-9_-]+\.[A-Za-z0-9_]+\.[A-Za-z0-9_]+$) and backtick-quoted; date bounds are bound ScalarQueryParameters; granularity is a fixed internal literal. All BigQuery/auth errors are wrapped as generic ProviderUnavailableno credentials, table, or project ever leak into responses/warnings/logs.
  • cloud category now returns [aws, gcp]; service registers gcp; active_services counts AWS + GCP. SKU is stored in metadata.usage_type, so the existing Service↔Usage-type breakdown toggle works for GCP unchanged.
  • Frontend Cloud tab combines AWS + GCP with an All / AWS / GCP provider filter; distinct pie colours per cloud provider; AWS + GCP explainer paragraphs.
  • GCP_BILLING_LOCATION config — sets the BigQuery job location. Required for single-region export datasets (e.g. europe-west1), otherwise the query defaults to US and fails; leave unset for US/EU multi-region (client auto-detects).
  • Config: GCP_BILLING_BQ_TABLE, GCP_BILLING_PROJECT_ID (falls back to GCP_PROJECT_ID), GCP_BILLING_LOCATION. Dependency google-cloud-bigquery==3.42.1. Docs updated (docs/billing-analytics.md).

Setup prerequisite (one-time, per billing account)

The gcp_billing_export_v1_* table is not created by this code — enable Cloud Billing → Billing export → Standard usage cost → BigQuery into a dataset you create; Google auto-populates the table (first rows within a few hours). Grant the ADC service account BigQuery Data Viewer + Job User. Until then the provider fails gracefully (Cloud tab still shows AWS, warning surfaced).

Testing

  • Backend suite: 1194 passed / 4 pre-existing unrelated test_config.py GA failures (present on main). Zero regressions.
  • GcpAnalyticsProvider: 9 unit tests (normalization, net-cost incl. credits, HOUR/DAY truncation, table validation, generic error wrapping, zero/blank-row skip, job-location used/defaulted).
  • Frontend tsc && vite build: clean.
  • Executed via subagent-driven development (implementer + independent spec/quality review per task) + a final opus whole-branch review: verdict "Ready to merge", no Critical/Important.

Deferred (non-blocking follow-ups)

  • Cloud cost-over-time chart is per-provider (matches pre-existing AWS behaviour); per-service lines are a later enhancement.
  • Pre-existing: AWS provider interpolates {exc} into its user-facing message (aws.py:148) — separate hardening ticket.
  • Detailed/resource-level GCP export (gcp_billing_export_resource_v1_*) and a Heroku cloud provider remain future additions.

🤖 Generated with Claude Code

PatrickCmd and others added 14 commits July 2, 2026 22:54
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… dep)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Single-region export datasets (e.g. europe-west1) need an explicit job
location or the query defaults to US and fails. Add GCP_BILLING_LOCATION
(default None => client auto-detects for US/EU multi-region).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant