Making Data Sources more useful

[nhoening]

FlexMeasures has data sources linked to each belief. We are encountering a few pain points while we begin to rely on them in some projects.

Let's revisit the goals in using them, current pain points (critical path) and what the roadmap can look like.

Goals of tracking data sources

1. Per sensor, different sources can add beliefs (for example: market prices, price forecasts, but also individual users)

• Who believed what?
• Choose whose data you want to use.

2. Trace back how beliefs was created. Data sources can store details on this, for example data sources have model & version columns, and reporters store their config in the attributes.
3. Treat data differently based on the type of source (for example schedules vs forecasts) - we do this already in the UI.

Current pain points (critical path)

• Partners start to re-use data in their customer-facing layers, and want to select certain beliefs by source. They don't want to track several source/model IDs to have the most recent (simple example: prices are pulled from market platform, but also forecasted by Seita as a fallback, or uploaded by a user). This also affects Seita as we are creating more and more live reports [ACTION -> give data sources an account_id & allow filtering for those stable IDs - users trust accounts to help them get beliefs in place]
• The status page is only getting most recent entries for a hard-coded list of data source types, but we are adding others (gateway, market) [ACTION: centralise list of known types]
• For developers, the mental concept of creating new data sources when the model or any configuration changed is creating a barrier to entry [ACTION: design proposal to keep number of sources robust, and move model/version/attribute updates to a data source audit log]
• The sensor page graphs and statistic widget can diverge in which sources are used (and also which time range) which confuses users.

Suggested stories

Transparent data source & types

• Default data source types centrally defined (and add two new ones)
• GET endpoint for data sources (accessible ones) and defined types
• GET endpoint for sensor data gets source type filter (it already has a source ID filter)

Data sources linked to account

• add account_id field
• datamigration: add account_id where user_id was used
• POST sensor data endpoint stores account_id on data source
• Weather plugin automatically registers an account for its forecasting service (source type: forecaster)
• ENTSOE plugin makes account (type: market)
• Reporters/schedulers/forecasters can filter by source account
• GET endpoint for sensor data gets source account filter

Sensor page: less confusion, explain data model

• Statistics default to given time range, add toggle to switch to "all time" (idea:
• Statistics show a table with one column per data source in the given time range
• Warn better when statistics diverge from graph (e.g. multiple sources in chart for given time range, or statistics shown for all history)
• Allow to select source IDs for graph (not critical)
• Statistics per default show only most recent beliefs.
• Allow to select "All belief(horizon)s per event" by a toggle (not critical)

[Flix6x]

@nhoening I edited the issue description with my suggestions.

[Flix6x]

I think that for the critical path we want whatever option is the least amount of work that still gets rid of the disparity between chart and statistics in case of multiple sources. In my mind showing statistics for all sources wasn't the option requiring the least amount of work, but I may be wrong. Let's explore.

What does it mean to show statistics for all sources?

1. Show statistics per source, for all sources (e.g. in a table), or
2. Show statistics for all sources, where each single statistic is based on data from all the sources

Option 1: I initially thought of this as more work than implementing the warning and picking the most prominent source in the given time range, but when I think about it we do already have all the required data loaded on the page and I guess showing a table instead of a single column isn't that much work.

Option 2: Seems the fastest to implement, but I don't think these statistics are actually useful. In fact, I think they would paint a very obscure picture.

[nhoening]

Thanks for thinking that through. I recall you expect seldom more than 4 or 5 sources. Then, such a table makes sense, but it could get slow if we need to look for more. How did you come up with the insight that "we do already have all the required data loaded on the page"?

[nhoening]

And: I found another dimension that matters: horizons.

I had already made this issue a while ago: #1290 ("Sensor page shows latest forecasts visually, but stats use all horizons") Here also, the statistics was showing different values (e.g. max) which the graph didn't have.

So we have now three:

• Date range vs all time
• Sources (all or one)
• Horizons (all or most recent)

To come back to the critical path, but minimal effort, for horizons I propose we could simply do most recent horizon. Then of course make that clear on the page (statistics use most recent, graphs also, but graphs show others if you hit playback). We can also promise a toggle for statistics to include the other horizons, but not start work on that yet.

Overall, a big gain next to less confusion can be that the user will hopefully learn a bit more about our beliefs data model when using this page - new useful info:

• we have more than 1 data source per sensor
• there is more than one belief per event (-> belief horizons)

[Flix6x]

Nice catch. I agree.

Just to note, that also for multi-horizon statistics imo it does not make sense to report a single statistic based on all horizons, but rather they would have to be presented separately for different horizons. That then could open up a discussion about which exact horizons to show, and how, but rather than go into that, I will do the exact opposite and would argue that multi-horizon support is among the ideas furthest from the critical path on this issue. :) That is, I would not even put it on the roadmap right now, let alone promise it.

[Ragnar-the-mighty]

I think the cardinality of sources for one sensor is fairly low, the total number for sources can be much higher.

sources are understandable, horizons are harder since they are almost invisible in the beginning. I agree the multi-horizon least critical.

[nhoening]

Thanks for all comments. The stories have been made, discussion can continue there:

• Transparent data source & types #2042
• Data sources linked to account #2046
• Sensor page: less confusion between statistics & graph #2053