IT: Problem with operaitng period

[ue71603]

Parameters for testrunner: ./tool_scripts/script_regression_v1.1.txt reg.log lombardiatrainitalia

relevant block:

       {
    "block": "lombardiatrainitalia",
    "url":"https://www.cciss.it/nap/mmtis/public/catalog/Dataset/5486",
    "download_url":"https://github.com/user-attachments/files/18585826/IT-ITC4-TRENITALIA_126_L1.xml.gz",
    "scripts": [
        {"step":1,"script": "clean_tmp", "args": "%%dir%%"},
        {"step":2,"script": "download_input_file", "args": "%%dir%%"},
        {"step":3,"script": "conv.netex_to_db", "args": "[%%inputfilepath%%] %%dir%%/02-import.lmdb "},
        {"step":4,"script": "#tools.netex_check_assertions", "args": "./scripts/general-netex-assertions.txt %%inputfilepath%%"},
        {"step":5,"script": "#tools.netex_stats", "args": "%%inputfilepath%%"},
        {"step":6,"script": "tools.xml_validator", "args": "%%inputfilepath%% schema/netex/xsd/NeTEx_publication.xsd"},
        {"step":7,"script": "conv.epip_db_to_db", "args": "%%dir%%/02-import.lmdb %%dir%%/07-netex-database.lmdb"},
        {"step":8,"script": "conv.epip_db_to_xml", "args": "%%dir%%/07-netex-database.lmdb %%dir%%/08-%%block%%-netex.xml.gz"},
        {"step":9,"script": "conv.netex_to_db", "args": "[%%dir%%/08-%%block%%-netex.xml.gz] %%dir%%/08-gtfs.lmdb "},
        {"step":10,"script": "#related_explorer", "args": "%%dir%%/08-gtfs.lmdb Line random %%dir%%/09-line.xml.gz "},
        {"step":11,"script": "conv.gtfs_db_to_db", "args": "%%dir%%/08-gtfs.lmdb %%dir%%/10-gtfs.lmdb"},
        {"step":12,"script": "conv.gtfs_db_to_gtfs", "args": "%%dir%%/10-gtfs.lmdb %%dir%%/11-%%block%%-gtfs.zip"},
        {"step":13,"script": "tools.gtfs_validator", "args": "%%dir%%/11-%%block%%-gtfs.zip %%dir%%"},
        {"step":14,"script": "tools.gtfs_map_visualisation", "args": "%%dir%%/11-%%block%%-gtfs.zip %%dir%%/13-%%block%%-map.html 1"}
         ]
  },

occuring error:

2025-08-05 18:47:15,803 - script_runner - INFO - lombardiatrainitalia - step: 7: conv.epip_db_to_db d:/aux_testing_processing/lombardiatrainitalia/02-import.lmdb d:/aux_testing_processing/lombardiatrainitalia/07-netex-database.lmdb
2025-08-05 18:47:16,018 - script_runner - INFO - Copy all tables as-is 
Database Operator does not exist, creating it.
2025-08-05 18:47:16,028 - script_runner - INFO - Copy lines, in EPIP style 
epip_line_memory
Database Line does not exist, creating it.
2025-08-05 18:47:16,067 - script_runner - INFO - Fix Quay / StopPlace locations 
Database ScheduledStopPoint does not exist, creating it.
2025-08-05 18:47:16,189 - script_runner - INFO - Investigate this site frame step 
epip_site_frame_memory
Database StopPlace does not exist, creating it.
2025-08-05 18:47:16,226 - script_runner - INFO - Service journeys 
epip_service_journey_generator
2025-08-05 18:47:16,227 - script_runner - INFO - Indexing RoutePoint to ScheduledStopPoint 
2025-08-05 18:47:16,250 - script_runner - INFO - Service journeys for now 
Database ServiceJourney does not exist, creating it.
Database ServiceJourneyPattern does not exist, creating it.
2025-08-05 18:47:19,693 - script_runner - INFO - Calendars 
2025-08-05 18:47:19,694 - script_runner - INFO - Calendar creation...
2025-08-05 18:47:19,741 - script_runner - ERROR - 'IT:ITC4:UicOperatingPeriod:TRENITALIA:102056' Traceback (most recent call last):
  File "D:\development\github\badger\tools\script_runner.py", line 454, in <module>
    main(
    ~~~~^
        args.script_file,
        ^^^^^^^^^^^^^^^^^
    ...<5 lines>...
        this_step=args.this_step,
        ^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "D:\development\github\badger\tools\script_runner.py", line 394, in main
    result = load_and_run(script_name, script_args)
  File "D:\development\github\badger\tools\script_runner.py", line 113, in load_and_run
    result = main_function(*args1)
  File "D:\development\github\badger\conv\epip_db_to_db.py", line 114, in main
    epip_service_calendar(source_db, target_db, generator_defaults)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\development\github\badger\transformers\epip.py", line 826, in epip_service_calendar
    operating_periods[dta.uic_operating_period_ref_or_operating_period_ref_or_operating_day_ref_or_date.ref]
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'IT:ITC4:UicOperatingPeriod:TRENITALIA:102056'

I guess it is the same as for SNCF.

[skinkie]

This is the issue:

              <DayTypeAssignment id="IT:ITC4:DayTypeAssignment:TRENITALIA:1_102056" version="1602" order="1">
                <OperatingPeriodRef ref="IT:ITC4:UicOperatingPeriod:TRENITALIA:102056" version="1602"></OperatingPeriodRef>
                <DayTypeRef ref="IT:ITC4:DayType:TRENITALIA:1_102056" version="1602"></DayTypeRef>
              </DayTypeAssignment>

So it is referencing an OperatingPeriod without any indication it is referencing to a UicOperatingPeriod. So it should have nameOfRefClass="UicOperatingPeriod" in order for our implementation to find it (currently). We can do all kind of fantastisch alternative and costly resolve techniques by object inheritance, but in essence this is broken in the source.

[ue71603]

This is your understanding of how it should have been done. But this is not how NeTex was done and works and they therefore don't need to change it. The relevant PR is not finalised and not accepted into 2.0. It did not make it intothe specification document for 2.0, therefore it probably is pushed back to 3.0
NeTEx-CEN/NeTEx#870

The italian implementation is valid and badger in my view has to deal with it.
That an OperatingPeriod often is a UicOperatingPeriod at least is a common issue (SNCF and Italy). So fixing this isolated may solve several countries.

Or what are you suggesting: A preprocessing step? for all such occurances?

[skinkie]

The EPIP document does not have an XML-example in the PDF. There is no 'right' from EPIP-perspective.

The original NeTEx Part 1 defines clearly that it fixed for each entity type. Hence the entity type for OperatingPeriodRef is OperatingPeriod, not UicOperatingPeriod, otherwise UicOperatingPeriodRef should have been used. This is the mistake made in EPIP, which defines OperatingPeriodRef, with UicOperatingPeriod as its instance.

Only EPIP defines that all object id's must be (document) unique...

...this is not the case for NeTEx in general. It is impossible to know if an ref="ABC" would refer to OperatingPeriod/@id='ABC' or UicOperatingPeriod/@id='ABC'. Sure one or the other may exist, but it is ambigious.

If both are unique individually;
        <xsd:unique name="OperatingPeriod_UniqueBy_Id_Version">
            <xsd:annotation>
                <xsd:documentation>Every [OperatingPeriod Id + Version] must be unique within document.</xsd:documentation>
            </xsd:annotation>
            <xsd:selector xpath=".//netex:OperatingPeriod"/>
            <xsd:field xpath="@id"/>
            <xsd:field xpath="@version"/>
        </xsd:unique>
        <xsd:unique name="UicOperatingPeriod_UniqueBy_Id_Version">
            <xsd:annotation>
                <xsd:documentation>Every [UicOperatingPeriod Id + Version] must be unique within document.</xsd:documentation>
            </xsd:annotation>
            <xsd:selector xpath=".//netex:UicOperatingPeriod"/>
            <xsd:field xpath="@id"/>
            <xsd:field xpath="@version"/>
            <!-- =====UicOperatingPeriod Key ========================== -->
        </xsd:unique>

This will never be a generic a unique reference:
        <xsd:keyref name="OperatingPeriod_KeyRef" refer="netex:OperatingPeriod_AnyVersionedKey">
            <xsd:selector xpath=".//netex:OperatingPeriodRef"/>
            <xsd:field xpath="@ref"/>
            <xsd:field xpath="@version"/>
        </xsd:keyref> 
        <xsd:key name="OperatingPeriod_AnyVersionedKey">
            <xsd:selector xpath=".//netex:OperatingPeriod | .//netex:UicOperatingPeriod"/>
            <xsd:field xpath="@id"/>
            <xsd:field xpath="@version"/>
        </xsd:key>

That we can do workarounds to handle poor implementations is obvious, that this implementation does not even validate the NeTEx XSD should be obvious too. Not failing on this example btw, since it is 'correct' according to the XSD validation.

There are two and half options:

1. If you know that the profile never defines an OperatingPeriod, then preprocessing is likely the best option.
2. Our loader does not care what comes in, and it creates a reference table. After loading, by use of the reference table we can figure out which keys are unresolved. Then we have the option to do an MRO search going alternative classes. For an OperatingPeriod it would be UicOperatingPeriod, for a TimingPoint it would be ScheduledStopPoint and FareScheduledStopPoint. First id match would cause the object that has the broken reference to be updated. The path within the object is known from the reference. The object is updated with the missing nameOfRefClass, and written to disk. Causing the references to be updated again.

Option 2 would effectively give a very fast constraint validator for free.

[ue71603]

solved