feat(billing): GCP cloud billing via BigQuery export#237
Open
PatrickCmd wants to merge 14 commits into
Open
Conversation
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… dep) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Single-region export datasets (e.g. europe-west1) need an explicit job location or the query defaults to US and fails. Add GCP_BILLING_LOCATION (default None => client auto-detects for US/EU multi-region). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…on tests and checks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds GCP cost/usage billing as a second provider under the existing Cloud category on the admin billing dashboard (
/admin/billing), alongside AWS. GCP has no on-demand cost API, so data comes from the Cloud Billing export in BigQuery (gcp_billing_export_v1_*), queried with parameterized SQL, reported as net cost (cost+ credits), broken down by service + SKU. Slots into the same provider-agnosticBillingRecord→ aggregation → caching → router pipeline built for AWS/Vast.ai — GCP is just a new provider mapped to thecloudcategory.Third and final provider in the billing suite: Inference (Runpod + Modal) → Training (Vast.ai) → Cloud (AWS + GCP).
What's included
GcpAnalyticsProvider(app/integrations/billing/gcp.py) — parameterized BigQuery query grouped by time-bucket +service.description+sku.description; net cost incl.credits;HOUR/DAYtruncation (no 14-day limit — that was AWS-specific); blocking client run viaasyncio.to_thread.^[A-Za-z0-9_-]+\.[A-Za-z0-9_]+\.[A-Za-z0-9_]+$) and backtick-quoted; date bounds are boundScalarQueryParameters; granularity is a fixed internal literal. All BigQuery/auth errors are wrapped as genericProviderUnavailable— no credentials, table, or project ever leak into responses/warnings/logs.cloudcategory now returns[aws, gcp]; service registersgcp;active_servicescounts AWS + GCP. SKU is stored inmetadata.usage_type, so the existing Service↔Usage-type breakdown toggle works for GCP unchanged.GCP_BILLING_LOCATIONconfig — sets the BigQuery job location. Required for single-region export datasets (e.g.europe-west1), otherwise the query defaults to US and fails; leave unset for US/EU multi-region (client auto-detects).GCP_BILLING_BQ_TABLE,GCP_BILLING_PROJECT_ID(falls back toGCP_PROJECT_ID),GCP_BILLING_LOCATION. Dependencygoogle-cloud-bigquery==3.42.1. Docs updated (docs/billing-analytics.md).Setup prerequisite (one-time, per billing account)
The
gcp_billing_export_v1_*table is not created by this code — enable Cloud Billing → Billing export → Standard usage cost → BigQuery into a dataset you create; Google auto-populates the table (first rows within a few hours). Grant the ADC service account BigQuery Data Viewer + Job User. Until then the provider fails gracefully (Cloud tab still shows AWS, warning surfaced).Testing
test_config.pyGA failures (present onmain). Zero regressions.GcpAnalyticsProvider: 9 unit tests (normalization, net-cost incl. credits, HOUR/DAY truncation, table validation, generic error wrapping, zero/blank-row skip, job-location used/defaulted).tsc && vite build: clean.Deferred (non-blocking follow-ups)
{exc}into its user-facing message (aws.py:148) — separate hardening ticket.gcp_billing_export_resource_v1_*) and a Heroku cloud provider remain future additions.🤖 Generated with Claude Code