Create Quick start documentation section

[jgc31416]

• ElHub Python SDK version: ALL
• Python version: NA
• Operating System: NA

Description

Describe clearly how to start whit this Python library.

[hakontro]

Big ups on this one! We really want to use the SDK, but I'm not really able to get started using it.. I can run tests and write code, however I'm not able to do stuff like reliably reading actual meter values from Elhub production by polling

[jgc31416]

@hakontro We had a talk with @SimonAtEida about how to deploy and what is the place and rol of the SDK in a bigger system, is that kind of documentation what you could need?

[hakontro]

That sounds great, but what I'm actually looking for is much simpler (I hope): How can I use the SDK to ask for some meter data, poll for it until I get the correct response, and access it from the XML-object. The existing tests return response, but I'm currently unable to fetch out the actual meter data.. it's probably very simple, and I'm probably very close, but it's a little bit confusing.

[sundstei]

I think those things are rather down to the understanding of the ElHub architecture. First you make a request, which gives you a status result. X amount of time later you have to do a polling for the results which will be waiting on your queue at ElHub.

[hakontro]

Well yes, but I understand that and am still unable to use the SDK for what I want to do, even after quite a bit of trying. Seems to me the code parses XML to a string and returns it, but I'm guessing you're doing some XML parsing somewhere to actually access data in a structured way?

To be clear: I'm sure this is just down to me not really grasping zeep or something, but a simple example in the quickstart would be a really helpful way to understand what's going on 🙏

[sundstei]

Let us know exactly what you want to do :) Maybe some example of how you are using it in code.

Zeep parses the result from Elhub to an custom structure, which can be used (but the designers of the API here has decided not to return this to the client). Reading the last returned XML from the history plugin to a string lets you at least parse it with whatever XML lib you are comfy with using.

In our use, we simply dump this string onto an external queue for processing by another service.

[hakontro]

Right, that makes sense. I see, I suppose we could do that (but would be very intersted in the parsing code, hehe).

Here's what I'm working on:

def test_meter_reading_test():
    """
    Request data from meter, then poll indefinitely
    """
    meter_identificator = "7070575000XXXXXXX"
    start = datetime.datetime(2023, 7, 1)
    end = start + datetime.timedelta(days=1)

    query_client, history = APIClient.get_client(
        environment=ElHubEnvironment.PRODUCTION,
        service=ElHubService.QUERY,
        key_file=KEY_FILE,
        cert_file=CERT_FILE,
    )

    print("#1 Requesting consumption")

    response = request_consumption(
        query_client, history, meter_identificator, sender_gsn=THIRD_PARTY_GSN_EXA, start=start, end=end
    )
    assert response != False
    print("#2 Requested consumption")
    # print(response)

    poll_client, history = APIClient.get_client(
        environment=ElHubEnvironment.PRODUCTION,
        service=ElHubService.POLL_METERING_VALUES,
        key_file=KEY_FILE,
        cert_file=CERT_FILE,
    )
    while True:
        print("#3 Polling consumption")
        response = poll_consumption(poll_client, history, THIRD_PARTY_GSN_EXA)
        print(response)
        print("#4 Polling done")
        time.sleep(1)

Elhub inconsistently returns various different responses to that polling, sometimes this payload

<ns5:PayloadResponseEvent>
            <ns6:StatusType listAgencyIdentifier="6">41</ns6:StatusType>
            <ns6:ResponseReasonType listAgencyIdentifier="89">EH011</ns6:ResponseReasonType>
            <ns6:ResponseReasonType listAgencyIdentifier="89">EH025</ns6:ResponseReasonType>
            <ns6:OriginalBusinessDocumentReference>7f0488d0-547d-4bee-a92a-a37bc99190b6</ns6:OriginalBusinessDocumentReference>
          </ns5:PayloadResponseEvent>

and other times really, really crazy amounts of XML (several tens of thousands of rows), containing the meter data in there.

Ideally I'd like to have a function that requests data from Elhub, starts polling, and retries either until a) it actually gets the data returned, or b) times out after some determined time. Would ideally like to have the data returned in a structured format like numpy or pandas.

Any tips on how to go about building that?

[jgc31416]

@hakontro I understand what you ask for, but having such a function couples too many things that I think they should be isolated, mainly the pooling with the parsing and perhaps the persistence (as you might be tempted to persist just only what you have parsed)

Here are some tips from the design we have at production now:

• I think that a major task as @sundstei mentions is to decouple pooling ElHub from processing the retrieved document. We have a process that just pools and writes the retrieved XML doc into cloud storage. Then we have a second process that listens to the changes in the storage and parses and stores the information the way we need. This design allows us to reprocess files easily if something goes wrong in the second step, and we don't need to ask ElHub for that info again.
• We parse the XML returned ignoring the namespaces and applying the business logic we need, this is why parsing is left out of the SDK, as the amount of information in the XML is pretty large and the pieces you might need can be different.

And here some ElHub tips:

• ElHub makes the delivery of the consumption information mostly during the morning, till 8.30, then it will go almost quiet the rest of the day, perhaps delivering some updated information. They will send you replies with no data most of the time.
• If you ask for historical consumption about a meter, ElHub will put that information into that same pooling endpoint, and then you can receive a few MB in one request. Make sure the code handling the XML can deal with these large files.

[hakontro]

Right, thanks for the info! The design makes a lot of sense given the description of Elhub you make here.

ElHub makes the delivery of the consumption information mostly during the morning, till 8.30

So, do I understand correctly - if we request consumption after 0830 we'll probably have to wait until the next morning for that data to be available on the polling queue?

[sundstei]

Typically you would subscribe to the consumption data for meters. This would be available on the queue as soon as these numbers are imported from the provider (electricity company).

[hakontro]

So, could something like this make sense?

1. Early in the morning request all the data you need for the day (for example consumption of all meters you want information about, and possible new meters we want historical data for).
2. Poll for data for some time (a couple of hours?), for example 06.00 - 08.30 or similar, write all results to some external storage
3. Parse those XMLs with some other tool

[hakontro]

Riight, that makes sense. Thanks a lot, you guys are the real Elhub technical support 🙏⭐️

[sundstei]

I am unsure about how you are using ElHub, but a normal approach for us would be :

• A new customer is added (we get permissions for his meters), so we request the base data for him and his meter.
• Subscribe for future updates, including consumption.

On a daily basis we then poll ElHub every 5-10 min for updates. This queue includes all data from all customers we have subscriptions for.

This is then put on an external queue as "string" for parsing and processing. This part is is the actual work ;-)

[sundstei]

Riight, that makes sense. Thanks a lot, you guys are the real Elhub technical support 🙏⭐️

We have suffered the pain of ElHub already :)