Skip to content

Migrate Opower data reads from REST to GraphQL#177

Open
loganrosen wants to merge 4 commits intotronikos:mainfrom
loganrosen:graphql-bill-costs
Open

Migrate Opower data reads from REST to GraphQL#177
loganrosen wants to merge 4 commits intotronikos:mainfrom
loganrosen:graphql-bill-costs

Conversation

@loganrosen
Copy link
Copy Markdown

@loganrosen loganrosen commented Mar 1, 2026

Summary

Migrates Opower data reads from REST to GraphQL for:

  • Bill-level cost/usage reads
  • Interval usage/cost reads (day/hour/half-hour/quarter-hour)
  • Realtime usage reads

REST fallback for bill reads has been removed.

Key implementation changes

src/opower/opower.py

  • Added GraphQL bill-cost path via _async_get_bill_cost_reads()
  • Added GraphQL interval path via _async_discover_service_point() + _async_get_graphql_interval_reads()
  • Added client-side interval aggregation (_aggregate_interval_reads())
  • Interval reads pipeline uses CostRead internally, extracting monetaryAmount from GraphQL when available (defaults to 0 when null)
  • Updated public APIs:
    • async_get_cost_reads() — returns CostRead for both bill and sub-bill aggregate types
    • async_get_usage_reads() — returns UsageRead (converts from internal CostRead)
    • async_get_realtime_usage_reads() — returns UsageRead
  • Removed obsolete REST read paths for interval/realtime data

src/opower/__main__.py

  • CLI output includes usage_charges and current_amount for bill-level reads

tests/test_opower.py

  • Added GraphQL fixture tests for bill parsing/filtering
  • Added interval discovery/fetch/cache tests
  • Added batching/discovery-failure/non-bill-cost tests
  • Added timezone bucketing regression test
  • Added monetaryAmount summation test for interval cost reads

Validation

  • Unit tests: 41 passed
  • pre-commit: clean on changed files
  • Live ConEd smoke tests passed for day, hour, and bill
  • Live-tested monetaryAmount field against ConEd — accepted by server but returns null (ConEd does not populate sub-bill cost data)

Known limitations

  • Gas interval filter values are currently best-effort (units: [CCF], serviceQuantityIdentifier: [DELIVERED]).
  • Electric/ConEd path is live-verified; gas GraphQL interval variants have not yet been live-validated.
  • monetaryAmount is null for ConEd interval reads. Other utilities may populate it — we carry it through when available.

Part of #176

@loganrosen loganrosen marked this pull request as draft March 1, 2026 23:02
@loganrosen loganrosen changed the title Add GraphQL bill-level cost data (Phase 1 of REST→GraphQL migration) Migrate Opower data reads from REST to GraphQL Mar 2, 2026
loganrosen and others added 2 commits March 1, 2026 22:19
Many utilities (ConEd, PSE, SCL, etc.) return providedCost=0 from
the REST DataBrowser API. The GraphQL API's bills query returns
actual cost data including usageCharges (energy only) and
currentAmount (total bill incl. delivery + taxes).

Changes:
- Add usage_charges and current_amount fields to CostRead dataclass
- Add _async_get_bill_cost_reads() using GraphQL bills query
- For AggregateType.BILL, try GraphQL first, fall back to REST
- Add variables support to _async_post_graphql()
- Update CLI tool to display new cost fields

Ref: tronikos#176

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace all REST-based data fetching (DataBrowser-v1, real-time-ami-v1)
with Opower's GraphQL API for interval reads.

Key changes:
- Add _async_discover_service_point() for GraphQL service point/register
  discovery with per-account caching
- Add _async_get_graphql_interval_reads() with 24-hour batching
  (API enforces max 24h per request)
- Add _parse_interval_reads_response() for response parsing
- Add _aggregate_interval_reads() for client-side aggregation
  (DAY/HOUR/HALF_HOUR from raw quarter-hour meter data)
- Rewrite async_get_cost_reads() to use GraphQL for all aggregate types
- Rewrite async_get_usage_reads() to use GraphQL
- Rewrite async_get_realtime_usage_reads() with 2-day GraphQL lookback
- Remove dead REST code: _async_get_dated_data, _async_fetch,
  _async_get_meters, self.meters

GraphQL API requirements discovered during testing:
- onlyUnverifiedStreams: true is required on intervalReads field
- serviceQuantityIdentifier is required (ELEC: NET_USAGE, GAS: DELIVERED)
- Time intervals must be UTC format (yyyy-mm-ddThh:mm:ssZ)
- No matching filter on serviceAgreementsConnection (returns empty)
- singlePremise parameter needed on billingAccountByAuthContext

Bill-level costs (from _async_get_bill_cost_reads) provide real cost data.
Sub-bill intervals only have usage data; cost derivation from bill rates
would be a separate enhancement.

Tested live with ConEd: day, hour, and bill aggregation all working.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@loganrosen loganrosen force-pushed the graphql-bill-costs branch from a668e64 to 7a19965 Compare March 2, 2026 03:21
@loganrosen loganrosen marked this pull request as ready for review March 2, 2026 03:21
@loganrosen loganrosen force-pushed the graphql-bill-costs branch from 5731eeb to bad1d86 Compare March 2, 2026 03:37
Add monetaryAmount { value currency } to interval reads GraphQL query.
Change internal pipeline to use CostRead (instead of UsageRead) for
interval reads, carrying provided_cost through parsing and aggregation.

For utilities that populate monetaryAmount on ServiceQuantityRead (not
ConEd currently), cost data now flows through to async_get_cost_reads
at sub-bill resolution. Public async_get_usage_reads still returns
UsageRead by converting at the boundary.

Add test for monetaryAmount summation in cost reads.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@loganrosen loganrosen force-pushed the graphql-bill-costs branch from bad1d86 to 094d3fb Compare March 2, 2026 03:39
@tronikos
Copy link
Copy Markdown
Owner

I haven't looked at the code. I just ran it for my utility PG&E and have several issues:

  • I got empty usage_charges and current_amount in the output.
  • The provided_cost is always 0 for me which is not the case when calling the REST API.
  • The consumption gives weird values. With aggregation set to day the REST API returns -10.1933 (I have solar and I'm returning to the grid) while the new API returns 6.9. With aggregation set to hour the consumption seem to match during the night (e.g. 0.5545 vs 0.554, note the less precision from the new API) but not during the day when I return to the grid. REST API has negative values while the new API has 0.
  • When the daylight saving time began on March 8 I get no results from 4am to 6pm, that's a gap of 14 hours. With the REST API I have no gap.

Address PR feedback from tronikos testing with PG&E:

- Include interval reads with null measuredAmount as consumption=0
  instead of skipping them (fixes data gaps including DST transitions)
- Use local time with offset for interval query batches instead of UTC,
  matching the bill query format (fixes DST-related server issues)
- Bill provided_cost falls back to usageCharges when currentAmount is
  null (fixes provided_cost always showing 0 for some utilities)
- usage_charges and current_amount fields are now None (not 0) when the
  GraphQL response doesn't include them
- CLI only shows usage_charges/current_amount columns for bill
  aggregation (they're always None for interval reads)
- Extract CLI output helpers to reduce _main() complexity
- Added tests for null measuredAmount handling and bill cost fallback

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@loganrosen
Copy link
Copy Markdown
Author

loganrosen commented Mar 29, 2026

Thanks for testing with PG&E, @tronikos! I've pushed fixes for all four issues:

  1. Empty usage_charges/current_amount — these columns are now only shown for bill-level reads (they're always None for interval reads, so showing them was confusing).

  2. provided_cost always 0 — for bill reads, provided_cost now falls back to usageCharges when currentAmount is null. Previously it was hardcoded to currentAmount which was null for some utilities.

  3. Wrong consumption values (no negatives, different daily totals) — the parser was skipping interval reads where measuredAmount was null instead of including them as 0. This was dropping data points. They're now included with consumption=0. (Note: if PG&E's GraphQL endpoint genuinely doesn't return negative values for solar net metering, that would be a server-side difference from the REST API — I can't fix that on the client side. Worth checking if the raw GraphQL response contains negatives.)

  4. 14-hour DST gap — two fixes here: (a) null reads are no longer dropped (see point 3 above), and (b) interval query batches now use local time with UTC offset instead of converting to UTC, matching the format used by the bill query. This avoids ambiguity at DST boundaries.

I've verified all three aggregation types (bill/day/hour) against ConEd, including across the March 8 DST transition — no gaps.

Would be great if you could re-test with PG&E to confirm these fix the issues on your end!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants