Migrate Opower data reads from REST to GraphQL#177
Migrate Opower data reads from REST to GraphQL#177loganrosen wants to merge 4 commits intotronikos:mainfrom
Conversation
Many utilities (ConEd, PSE, SCL, etc.) return providedCost=0 from the REST DataBrowser API. The GraphQL API's bills query returns actual cost data including usageCharges (energy only) and currentAmount (total bill incl. delivery + taxes). Changes: - Add usage_charges and current_amount fields to CostRead dataclass - Add _async_get_bill_cost_reads() using GraphQL bills query - For AggregateType.BILL, try GraphQL first, fall back to REST - Add variables support to _async_post_graphql() - Update CLI tool to display new cost fields Ref: tronikos#176 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace all REST-based data fetching (DataBrowser-v1, real-time-ami-v1) with Opower's GraphQL API for interval reads. Key changes: - Add _async_discover_service_point() for GraphQL service point/register discovery with per-account caching - Add _async_get_graphql_interval_reads() with 24-hour batching (API enforces max 24h per request) - Add _parse_interval_reads_response() for response parsing - Add _aggregate_interval_reads() for client-side aggregation (DAY/HOUR/HALF_HOUR from raw quarter-hour meter data) - Rewrite async_get_cost_reads() to use GraphQL for all aggregate types - Rewrite async_get_usage_reads() to use GraphQL - Rewrite async_get_realtime_usage_reads() with 2-day GraphQL lookback - Remove dead REST code: _async_get_dated_data, _async_fetch, _async_get_meters, self.meters GraphQL API requirements discovered during testing: - onlyUnverifiedStreams: true is required on intervalReads field - serviceQuantityIdentifier is required (ELEC: NET_USAGE, GAS: DELIVERED) - Time intervals must be UTC format (yyyy-mm-ddThh:mm:ssZ) - No matching filter on serviceAgreementsConnection (returns empty) - singlePremise parameter needed on billingAccountByAuthContext Bill-level costs (from _async_get_bill_cost_reads) provide real cost data. Sub-bill intervals only have usage data; cost derivation from bill rates would be a separate enhancement. Tested live with ConEd: day, hour, and bill aggregation all working. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
a668e64 to
7a19965
Compare
5731eeb to
bad1d86
Compare
Add monetaryAmount { value currency } to interval reads GraphQL query.
Change internal pipeline to use CostRead (instead of UsageRead) for
interval reads, carrying provided_cost through parsing and aggregation.
For utilities that populate monetaryAmount on ServiceQuantityRead (not
ConEd currently), cost data now flows through to async_get_cost_reads
at sub-bill resolution. Public async_get_usage_reads still returns
UsageRead by converting at the boundary.
Add test for monetaryAmount summation in cost reads.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
bad1d86 to
094d3fb
Compare
|
I haven't looked at the code. I just ran it for my utility PG&E and have several issues:
|
Address PR feedback from tronikos testing with PG&E: - Include interval reads with null measuredAmount as consumption=0 instead of skipping them (fixes data gaps including DST transitions) - Use local time with offset for interval query batches instead of UTC, matching the bill query format (fixes DST-related server issues) - Bill provided_cost falls back to usageCharges when currentAmount is null (fixes provided_cost always showing 0 for some utilities) - usage_charges and current_amount fields are now None (not 0) when the GraphQL response doesn't include them - CLI only shows usage_charges/current_amount columns for bill aggregation (they're always None for interval reads) - Extract CLI output helpers to reduce _main() complexity - Added tests for null measuredAmount handling and bill cost fallback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Thanks for testing with PG&E, @tronikos! I've pushed fixes for all four issues:
I've verified all three aggregation types (bill/day/hour) against ConEd, including across the March 8 DST transition — no gaps. Would be great if you could re-test with PG&E to confirm these fix the issues on your end! |
Summary
Migrates Opower data reads from REST to GraphQL for:
REST fallback for bill reads has been removed.
Key implementation changes
src/opower/opower.py_async_get_bill_cost_reads()_async_discover_service_point()+_async_get_graphql_interval_reads()_aggregate_interval_reads())CostReadinternally, extractingmonetaryAmountfrom GraphQL when available (defaults to 0 when null)async_get_cost_reads()— returnsCostReadfor both bill and sub-bill aggregate typesasync_get_usage_reads()— returnsUsageRead(converts from internalCostRead)async_get_realtime_usage_reads()— returnsUsageReadsrc/opower/__main__.pyusage_chargesandcurrent_amountfor bill-level readstests/test_opower.pymonetaryAmountsummation test for interval cost readsValidation
41 passedday,hour, andbillmonetaryAmountfield against ConEd — accepted by server but returns null (ConEd does not populate sub-bill cost data)Known limitations
units: [CCF],serviceQuantityIdentifier: [DELIVERED]).monetaryAmountis null for ConEd interval reads. Other utilities may populate it — we carry it through when available.Part of #176