Skip to content

adaptions to test scripts#74

Merged
skinkie merged 19 commits intobinary_relation_serializerfrom
update_2025-12-23
Jan 8, 2026
Merged

adaptions to test scripts#74
skinkie merged 19 commits intobinary_relation_serializerfrom
update_2025-12-23

Conversation

@ue71603
Copy link
Collaborator

@ue71603 ue71603 commented Dec 23, 2025

@skinkie Can you check, why
the script ./tool_scripts/script_regression_basic.txt mylog.log nl_filter

produces this error:


2025-12-23 22:03:57,348 - script_runner - INFO - nl_filter - step: 6: conv.filter_db_to_db d:/badger_work/nl_filter/04.mdbx d:/badger_work/nl_filter/05.mdbx Line NL:WPD:Line:HA  [Line Route ServiceJourneyPattern]
[Errno 38] Filename too long
2025-12-23 22:10:03,763 - script_runner - ERROR - [Errno 38] Filename too long
Traceback (most recent call last):
  File "C:\Users\ue71603\MG_Daten\github\badger\tools\script_runner.py", line 404, in main
    result = load_and_run(script_name, script_args)
  File "C:\Users\ue71603\MG_Daten\github\badger\tools\script_runner.py", line 115, in load_and_run
    result = main_function(*args1)
  File "C:\Users\ue71603\MG_Daten\github\badger\conv\filter_db_to_db.py", line 114, in main
    filter_db_to_db(source_path, Path(target), clazz, object_filter, inward_classes)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ue71603\MG_Daten\github\badger\conv\filter_db_to_db.py", line 46, in filter_db_to_db
    db_write.insert_any_object_on_queue(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        txn_write,
        ^^^^^^^^^^
        db_read.load_references_by_object_values_dfs(txn, full_key, inward_classes),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "C:\Users\ue71603\MG_Daten\github\badger\storage\mdbx\core\implementation.py", line 235, in insert_any_object_on_queue
    for obj in objects:
               ^^^^^^^
  File "C:\Users\ue71603\MG_Daten\github\badger\storage\mdbx\core\implementation.py", line 422, in load_references_by_object_values_dfs
    for _referenced_full_key in self.load_references_by_clazz_full_key(txn, full_key, False):
                                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ue71603\MG_Daten\github\badger\storage\mdbx\core\implementation.py", line 354, in load_references_by_clazz_full_key
    yield from self._load_references_by_fullkey(txn, full_key)
  File "C:\Users\ue71603\MG_Daten\github\badger\storage\mdbx\core\implementation.py", line 315, in _load_references_by_fullkey
    for it in cursor.iter_dupsort_rows(start_key=full_key):
              ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ue71603\MG_Daten\github\badger\.venv\Lib\site-packages\mdbx\mdbx.py", line 3313, in __next__
    k, v = self.cur.get_full(None, op)
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "C:\Users\ue71603\MG_Daten\github\badger\.venv\Lib\site-packages\mdbx\mdbx.py", line 3068, in get_full
    raise make_exception(ret)
OSError: [Errno 38] Filename too long
2025-12-23 22:10:03,952 - script_runner - INFO - Execution time: 366.6 seconds for nl_filter - step: 6: conv.filter_db_to_db d:/badger_work/nl_filter/04.mdbx d:/badger_work/nl_filter/05.mdbx Line NL:WPD:Line:HA  [Line Route ServiceJourneyPattern]

2025-12-23 22:10:03,953 - script_runner - ERROR - Script conv.filter_db_to_db returned an error. Terminating the block of scripts: nl_filter

Process finished with exit code 1

something wrong with the definition?


    {
       "block": "nl_filter",
       "url":"https://data.ndovloket.nl/netex/wpd/",
       "download_url":"https://data.ndovloket.nl/netex/wpd/NeTEx_WPD_WPD_20251124_20251124.xml.gz",
        "description":"Primary filter test based on NL",
        "scripts": [
        {"script": "clean_tmp", "args": "%%dir%%"},
        {"script": "set_defaults", "args": "authority='SKI+'; time_zone='Europe/Berlin'; feed_publisher_name='SBB SKI+'; feed_publisher_url='https://opentransportdata.swiss'"},
        {"script": "download_input_file", "args": "%%dir%%"},
        {"script": "conv.netex_to_db", "args": "[%%inputfilepath%%] %%dir%%/03.mdbx"},
        {"script": "conv.epip_db_to_db", "args": "%%dir%%/03.mdbx %%dir%%/04.mdbx"},
        {"script": "conv.filter_db_to_db", "args": "%%dir%%/04.mdbx %%dir%%/05.mdbx Line NL:WPD:Line:HA  [Line Route ServiceJourneyPattern]" },
        {"script": "conv.epip_db_to_xml", "args": "%%dir%%/05.mdbx %%dir%%/%%block%%-line.xml"},
        {"script": "conv.filter_db_to_db", "args": "%%dir%%/04.mdbx %%dir%%/06.mdbx Operator NL:WPD:Operator:WPD   [Line Route ServiceJourneyPattern]" },
        {"script": "conv.epip_db_to_xml", "args": "%%dir%%/06.mdbx %%dir%%/%%block%%-operator.xml"},
        {"script": "conv.filter_db_to_db", "args": "%%dir%%/04.mdbx %%dir%%/07.mdbx ServiceJourney NL:WPD:ServiceJourney:AMHO-060000   [Line Route ServiceJourneyPattern]" },
        {"script": "conv.epip_db_to_xml", "args": "%%dir%%/07.mdbx %%dir%%/%%block%%-servicejourney.xml"},
        {"script": "tools.xml_validator.py", "args": "%%dir%%/%%block%%-netex.xml domain/schema/netex/xsd/NeTEx_publication.xsd"}

        ]
    },

@skinkie
Copy link
Contributor

skinkie commented Dec 23, 2025

@ue71603 the fundamental problem that I had running the code was that the path towards the XML Schema was wrong. I don't get any error afterwards.

@skinkie
Copy link
Contributor

skinkie commented Dec 23, 2025

Maybe you can print the path around line 114?

@ue71603
Copy link
Collaborator Author

ue71603 commented Dec 24, 2025

@skinkie Thanks for fixing the path. However, I still get the error. Do I need to regenerate schema and model again. Also perhaps I don't have the excact mdbx as you do or a library not installed.
It seems to me also that the toml is not up to date: no mention of mdbx. If you update it, I try to figure out, if I have installed the right versions.

@skinkie
Copy link
Contributor

skinkie commented Dec 24, 2025

You get an error about the path length, please print the path before the error.

@ue71603
Copy link
Collaborator Author

ue71603 commented Dec 24, 2025

It obviously is not the length ot the paths.. I added the code and it does show

2025-12-24 17:19:07,813 - script_runner - INFO - nl_filter - step: 6: conv.filter_db_to_db d:/badger_work/nl_filter/04.mdbx d:/badger_work/nl_filter/05.mdbx Line NL:WPD:Line:HA  [Line Route ServiceJourneyPattern]
the arguments ['d:/badger_work/nl_filter/04.mdbx', 'd:/badger_work/nl_filter/05.mdbx', 'Line', 'NL:WPD:Line:HA', ['Line', 'Route', 'ServiceJourneyPattern']]
source_path: d:\badger_work\nl_filter\04.mdbx
target_path: d:\badger_work\nl_filter\05.mdbx
[Errno 38] Filename too long
2025-12-24 17:19:08,075 - script_runner - ERROR - [Errno 38] Filename too long
Traceback (most recent call last):
  File "C:\Users\ue71603\MG_Daten\github\badger\tools\script_runner.py", line 405, in main
    result = load_and_run(script_name, script_args)
  File "C:\Users\ue71603\MG_Daten\github\badger\tools\script_runner.py", line 116, in load_and_run
    result = main_function(*args1)
  File "C:\Users\ue71603\MG_Daten\github\badger\conv\filter_db_to_db.py", line 116, in main
    filter_db_to_db(source_path, Path(target), clazz, object_filter, inward_classes)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ue71603\MG_Daten\github\badger\conv\filter_db_to_db.py", line 46, in filter_db_to_db
    db_write.insert_any_object_on_queue(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        txn_write,
        ^^^^^^^^^^
        db_read.load_references_by_object_values_dfs(txn, full_key, inward_classes),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "C:\Users\ue71603\MG_Daten\github\badger\storage\mdbx\core\implementation.py", line 235, in insert_any_object_on_queue
    for obj in objects:
               ^^^^^^^
  File "C:\Users\ue71603\MG_Daten\github\badger\storage\mdbx\core\implementation.py", line 422, in load_references_by_object_values_dfs
    for _referenced_full_key in self.load_references_by_clazz_full_key(txn, full_key, False):
                                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ue71603\MG_Daten\github\badger\storage\mdbx\core\implementation.py", line 354, in load_references_by_clazz_full_key
    yield from self._load_references_by_fullkey(txn, full_key)
  File "C:\Users\ue71603\MG_Daten\github\badger\storage\mdbx\core\implementation.py", line 315, in _load_references_by_fullkey
    for it in cursor.iter_dupsort_rows(start_key=full_key):
              ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ue71603\MG_Daten\github\badger\.venv\Lib\site-packages\mdbx\mdbx.py", line 3313, in __next__
    k, v = self.cur.get_full(None, op)
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "C:\Users\ue71603\MG_Daten\github\badger\.venv\Lib\site-packages\mdbx\mdbx.py", line 3068, in get_full
    raise make_exception(ret)
OSError: [Errno 38] Filename too long
2025-12-24 17:19:08,083 - script_runner - INFO - Execution time: 0.2 seconds for nl_filter - step: 6: conv.filter_db_to_db d:/badger_work/nl_filter/04.mdbx d:/badger_work/nl_filter/05.mdbx Line NL:WPD:Line:HA  [Line Route ServiceJourneyPattern]

2025-12-24 17:19:08,083 - script_runner - ERROR - Script conv.filter_db_to_db returned an error. Terminating the block of scripts: nl_filter

Does python -m tools.script_runner ./tools/tool_scripts/script_regression_basic.txt mylog.log nl_filter work with you?

I added some more code to show what the keys are. ChatGPT suggest that the problem lies deeper and then the error is created later.

C:\Users\ue71603\MG_Daten\github\badger\.venv\Scripts\python.exe -X pycache_prefix=C:\Users\ue71603\AppData\Local\JetBrains\PyCharmCE2024.3\cpython-cache "C:/Program Files/JetBrains/PyCharm Community Edition 2024.1.4/plugins/python-ce/helpers/pydev/pydevd.py" --multiprocess --qt-support=auto --port 29781 --file C:\Users\ue71603\MG_Daten\github\badger\tools\script_runner.py ./tool_scripts/script_regression_basic.txt mylog.log nl_filter --begin_step=6 
2025-12-24 17:31:22,807 - script_runner - INFO - nl_filter - step: 6: conv.filter_db_to_db d:/badger_work/nl_filter/04.mdbx d:/badger_work/nl_filter/05.mdbx Line NL:WPD:Line:HA  [Line Route ServiceJourneyPattern]
the arguments ['d:/badger_work/nl_filter/04.mdbx', 'd:/badger_work/nl_filter/05.mdbx', 'Line', 'NL:WPD:Line:HA', ['Line', 'Route', 'ServiceJourneyPattern']]
source_path: d:\badger_work\nl_filter\04.mdbx
target_path: d:\badger_work\nl_filter\05.mdbx
b'S\x00\x00\x00\xe4\x02\x00\x00' 8
b'#\x00\x00\x00\x87\x03\x00\x00' 8
b'B\x00\x00\x009\x05\x00\x00' 8
b'W\x00\x00\x00>\x05\x00\x00' 8
b';\x00\x00\x00\xff\x04\x00\x00' 8
b'\x0f\x00\x00\x00\xff\x04\x00\x00' 8
b'\xeb\x00\x00\x00\x80\x01\x00\x00' 8
b'7\x00\x00\x00{\x01\x00\x00' 8
b'\x10\x00\x00\x009\x05\x00\x00' 8
b'e\x00\x00\x00>\x05\x00\x00' 8
b'\xec\x00\x00\x00\x80\x01\x00\x00' 8
b' \x00\x00\x00{\x01\x00\x00' 8
b'G\x00\x00\x00\xd2\x04\x00\x00' 8
b'!\x00\x00\x00\xd9\x04\x00\x00' 8
b'\x1c\x00\x00\x00\xd9\x04\x00\x00' 8
b'\x1e\x00\x00\x00\xd6\x04\x00\x00' 8
b'\x0b\x00\x00\x00\x85\x03\x00\x00' 8
b'\x01\x00\x00\x00\xd2\x04\x00\x00' 8
b'\x00\x00\x00\x00\xd6\x04\x00\x00' 8
b'\xc9\x00\x00\x004\x05\x00\x00' 8
b'4\x00\x00\x00h\x07\x00\x00' 8
here we are
[Errno 38] Filename too long
2025-12-24 17:36:07,804 - script_runner - ERROR - [Errno 38] Filename too long
Traceback (most recent call last):
  File "C:\Users\ue71603\MG_Daten\github\badger\tools\script_runner.py", line 405, in main
    result = load_and_run(script_name, script_args)
  File "C:\Users\ue71603\MG_Daten\github\badger\tools\script_runner.py", line 116, in load_and_run
    result = main_function(*args1)
  File "C:\Users\ue71603\MG_Daten\github\badger\conv\filter_db_to_db.py", line 116, in main
    filter_db_to_db(source_path, Path(target), clazz, object_filter, inward_classes)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ue71603\MG_Daten\github\badger\conv\filter_db_to_db.py", line 46, in filter_db_to_db
    db_write.insert_any_object_on_queue(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        txn_write,
        ^^^^^^^^^^
        db_read.load_references_by_object_values_dfs(txn, full_key, inward_classes),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "C:\Users\ue71603\MG_Daten\github\badger\storage\mdbx\core\implementation.py", line 235, in insert_any_object_on_queue
    for obj in objects:
               ^^^^^^^
  File "C:\Users\ue71603\MG_Daten\github\badger\storage\mdbx\core\implementation.py", line 425, in load_references_by_object_values_dfs
    for _referenced_full_key in self.load_references_by_clazz_full_key(txn, full_key, False):
                                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ue71603\MG_Daten\github\badger\storage\mdbx\core\implementation.py", line 357, in load_references_by_clazz_full_key
    yield from self._load_references_by_fullkey(txn, full_key)
  File "C:\Users\ue71603\MG_Daten\github\badger\storage\mdbx\core\implementation.py", line 318, in _load_references_by_fullkey
    for it in cursor.iter_dupsort_rows(start_key=full_key):
              ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ue71603\MG_Daten\github\badger\.venv\Lib\site-packages\mdbx\mdbx.py", line 3313, in __next__
    k, v = self.cur.get_full(None, op)
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "C:\Users\ue71603\MG_Daten\github\badger\.venv\Lib\site-packages\mdbx\mdbx.py", line 3068, in get_full
    raise make_exception(ret)
OSError: [Errno 38] Filename too long
2025-12-24 17:36:07,859 - script_runner - INFO - Execution time: 285.0 seconds for nl_filter - step: 6: conv.filter_db_to_db d:/badger_work/nl_filter/04.mdbx d:/badger_work/nl_filter/05.mdbx Line NL:WPD:Line:HA  [Line Route ServiceJourneyPattern]

2025-12-24 17:36:07,859 - script_runner - ERROR - Script conv.filter_db_to_db returned an error. Terminating the block of scripts: nl_filter

Process finished with exit code 1

Which libmdbx do you use?
image

@skinkie
Copy link
Contributor

skinkie commented Dec 24, 2025

I was running libmdbx 0.2.1, upgrade to 0.3.1.

This is my output:

(badger) skinkie@thinkpad /mnt/storage/home/skinkie/Sources/badger $ python -m tools.script_runner ./tools/tool_scripts/script_regression_basic.txt mylog.log nl_filter
2025-12-24 18:16:14,561 - script_runner - INFO - nl_filter - step: 1: clean_tmp d:/aux_testing_processing/nl_filter
2025-12-24 18:16:14,562 - script_runner - INFO - Command 'clean_tmp' executed for folder: d:/aux_testing_processing/nl_filter

2025-12-24 18:16:14,562 - script_runner - INFO - nl_filter - step: 2: set_defaults authority='SKI+'; time_zone='Europe/Berlin'; feed_publisher_name='SBB SKI+'; feed_publisher_url='https://opentransportdata.swiss'
2025-12-24 18:16:14,562 - script_runner - INFO - Command 'set_defaults' executed for: authority='SKI+'; time_zone='Europe/Berlin'; feed_publisher_name='SBB SKI+'; feed_publisher_url='https://opentransportdata.swiss'

2025-12-24 18:16:14,563 - script_runner - INFO - nl_filter - step: 3: download_input_file d:/aux_testing_processing/nl_filter
2025-12-24 18:16:14,563 - script_runner - INFO - File: d:/aux_testing_processing/nl_filter/NeTEx_WPD_WPD_20251124_20251124.xml.gz exists already.Will use that one
2025-12-24 18:16:14,563 - script_runner - INFO - Download from: https://data.ndovloket.nl/netex/wpd/NeTEx_WPD_WPD_20251124_20251124.xml.gz
2025-12-24 18:16:14,692 - script_runner - INFO - File downloaded successfully.
2025-12-24 18:16:14,693 - script_runner - INFO - Command 'download_input_file' executed for url: https://data.ndovloket.nl/netex/wpd/NeTEx_WPD_WPD_20251124_20251124.xml.gz

2025-12-24 18:16:14,693 - script_runner - INFO - nl_filter - step: 4: conv.netex_to_db [d:/aux_testing_processing/nl_filter/NeTEx_WPD_WPD_20251124_20251124.xml.gz] d:/aux_testing_processing/nl_filter/03.mdbx
the arguments [['d:/aux_testing_processing/nl_filter/NeTEx_WPD_WPD_20251124_20251124.xml.gz'], 'd:/aux_testing_processing/nl_filter/03.mdbx']
2025-12-24 18:16:25,815 - script_runner - INFO - Execution time: 11.1 seconds for nl_filter - step: 4: conv.netex_to_db [d:/aux_testing_processing/nl_filter/NeTEx_WPD_WPD_20251124_20251124.xml.gz] d:/aux_testing_processing/nl_filter/03.mdbx

2025-12-24 18:16:25,816 - script_runner - INFO - nl_filter - step: 5: conv.epip_db_to_db d:/aux_testing_processing/nl_filter/03.mdbx d:/aux_testing_processing/nl_filter/04.mdbx
the arguments ['d:/aux_testing_processing/nl_filter/03.mdbx', 'd:/aux_testing_processing/nl_filter/04.mdbx']
epip_line_generator
2025-12-24 18:16:26,508 - script_runner - INFO - Indexing RoutePoint to ScheduledStopPoint 
2025-12-24 18:16:26,509 - script_runner - INFO - Service journeys for now 
2025-12-24 18:16:26,674 - script_runner - INFO - Calendar creation...
epip_service_journey_interchange
2025-12-24 18:16:27,050 - script_runner - INFO - Execution time: 1.2 seconds for nl_filter - step: 5: conv.epip_db_to_db d:/aux_testing_processing/nl_filter/03.mdbx d:/aux_testing_processing/nl_filter/04.mdbx

2025-12-24 18:16:27,051 - script_runner - INFO - nl_filter - step: 6: conv.filter_db_to_db d:/aux_testing_processing/nl_filter/04.mdbx d:/aux_testing_processing/nl_filter/05.mdbx Line NL:WPD:Line:HA  [Line Route ServiceJourneyPattern]
the arguments ['d:/aux_testing_processing/nl_filter/04.mdbx', 'd:/aux_testing_processing/nl_filter/05.mdbx', 'Line', 'NL:WPD:Line:HA', ['Line', 'Route', 'ServiceJourneyPattern']]
source_path: d:/aux_testing_processing/nl_filter/04.mdbx
target_path: d:/aux_testing_processing/nl_filter/05.mdbx
b'S\x00\x00\x00\xe4\x02\x00\x00' 8
b'\x1f\x00\x00\x00\x87\x03\x00\x00' 8
b'>\x00\x00\x009\x05\x00\x00' 8
b'W\x00\x00\x00>\x05\x00\x00' 8
b'+\x00\x00\x00\xff\x04\x00\x00' 8
b'\x1c\x00\x00\x00\xff\x04\x00\x00' 8
b'\xeb\x00\x00\x00\x80\x01\x00\x00' 8
b'I\x00\x00\x00{\x01\x00\x00' 8
b'\n\x00\x00\x009\x05\x00\x00' 8
b'e\x00\x00\x00>\x05\x00\x00' 8
b'\xec\x00\x00\x00\x80\x01\x00\x00' 8
b'=\x00\x00\x00{\x01\x00\x00' 8
b'<\x00\x00\x00\xd2\x04\x00\x00' 8
b'\x13\x00\x00\x00\xd9\x04\x00\x00' 8
b'\x04\x00\x00\x00\xd9\x04\x00\x00' 8
b' \x00\x00\x00\xd6\x04\x00\x00' 8
b'E\x00\x00\x00\x85\x03\x00\x00' 8
b'%\x00\x00\x00\xd2\x04\x00\x00' 8
b'\x0e\x00\x00\x00\xd6\x04\x00\x00' 8
b'\xc9\x00\x00\x004\x05\x00\x00' 8
b'/\x00\x00\x00h\x07\x00\x00' 8
b'\xc6\x00\x00\x00=\x01\x00\x00' 8
b'\xc8\x00\x00\x004\x05\x00\x00' 8
b'\xc4\x00\x00\x004\x05\x00\x00' 8
b'\xc1\x00\x00\x00=\x01\x00\x00' 8
b'\xc3\x00\x00\x004\x05\x00\x00' 8
b'\xbf\x00\x00\x004\x05\x00\x00' 8
b'\xbc\x00\x00\x00=\x01\x00\x00' 8
b'\xbe\x00\x00\x004\x05\x00\x00' 8
b'\xad\x00\x00\x004\x05\x00\x00' 8
b'\xaa\x00\x00\x00=\x01\x00\x00' 8
b'\xac\x00\x00\x004\x05\x00\x00' 8
b'\xa8\x00\x00\x004\x05\x00\x00' 8
b'\xa5\x00\x00\x00=\x01\x00\x00' 8
b'\xa7\x00\x00\x004\x05\x00\x00' 8
b'\xa1\x00\x00\x004\x05\x00\x00' 8
b'\x97\x00\x00\x00=\x01\x00\x00' 8
b'\x9a\x00\x00\x004\x05\x00\x00' 8
b'\x90\x00\x00\x004\x05\x00\x00' 8
b'r\x00\x00\x00=\x01\x00\x00' 8
b'\x8a\x00\x00\x004\x05\x00\x00' 8
b'\x87\x00\x00\x004\x05\x00\x00' 8
b'\x84\x00\x00\x004\x05\x00\x00' 8
b'\x80\x00\x00\x004\x05\x00\x00' 8
b'}\x00\x00\x004\x05\x00\x00' 8
b'z\x00\x00\x004\x05\x00\x00' 8
b't\x00\x00\x004\x05\x00\x00' 8
b'h\x00\x00\x004\x05\x00\x00' 8
b'Y\x00\x00\x00=\x01\x00\x00' 8
b'[\x00\x00\x004\x05\x00\x00' 8
2025-12-24 18:16:27,371 - script_runner - INFO - Execution time: 0.3 seconds for nl_filter - step: 6: conv.filter_db_to_db d:/aux_testing_processing/nl_filter/04.mdbx d:/aux_testing_processing/nl_filter/05.mdbx Line NL:WPD:Line:HA  [Line Route ServiceJourneyPattern]

2025-12-24 18:16:27,372 - script_runner - INFO - nl_filter - step: 7: conv.epip_db_to_xml d:/aux_testing_processing/nl_filter/05.mdbx d:/aux_testing_processing/nl_filter/nl_filter-line.xml
the arguments ['d:/aux_testing_processing/nl_filter/05.mdbx', 'd:/aux_testing_processing/nl_filter/nl_filter-line.xml']
2025-12-24 18:16:27,578 - script_runner - INFO - Execution time: 0.2 seconds for nl_filter - step: 7: conv.epip_db_to_xml d:/aux_testing_processing/nl_filter/05.mdbx d:/aux_testing_processing/nl_filter/nl_filter-line.xml

2025-12-24 18:16:27,578 - script_runner - INFO - nl_filter - step: 8: conv.filter_db_to_db d:/aux_testing_processing/nl_filter/04.mdbx d:/aux_testing_processing/nl_filter/06.mdbx Operator NL:WPD:Operator:WPD   [Line Route ServiceJourneyPattern]
the arguments ['d:/aux_testing_processing/nl_filter/04.mdbx', 'd:/aux_testing_processing/nl_filter/06.mdbx', 'Operator', 'NL:WPD:Operator:WPD', ['Line', 'Route', 'ServiceJourneyPattern']]
source_path: d:/aux_testing_processing/nl_filter/04.mdbx
target_path: d:/aux_testing_processing/nl_filter/06.mdbx
b'\x1f\x00\x00\x00\x87\x03\x00\x00' 8
2025-12-24 18:16:27,764 - script_runner - INFO - Execution time: 0.1 seconds for nl_filter - step: 8: conv.filter_db_to_db d:/aux_testing_processing/nl_filter/04.mdbx d:/aux_testing_processing/nl_filter/06.mdbx Operator NL:WPD:Operator:WPD   [Line Route ServiceJourneyPattern]

2025-12-24 18:16:27,765 - script_runner - INFO - nl_filter - step: 9: conv.epip_db_to_xml d:/aux_testing_processing/nl_filter/06.mdbx d:/aux_testing_processing/nl_filter/nl_filter-operator.xml
the arguments ['d:/aux_testing_processing/nl_filter/06.mdbx', 'd:/aux_testing_processing/nl_filter/nl_filter-operator.xml']
2025-12-24 18:16:27,846 - script_runner - INFO - Execution time: 0.0 seconds for nl_filter - step: 9: conv.epip_db_to_xml d:/aux_testing_processing/nl_filter/06.mdbx d:/aux_testing_processing/nl_filter/nl_filter-operator.xml

2025-12-24 18:16:27,846 - script_runner - INFO - nl_filter - step: 10: conv.filter_db_to_db d:/aux_testing_processing/nl_filter/04.mdbx d:/aux_testing_processing/nl_filter/07.mdbx ServiceJourney NL:WPD:ServiceJourney:AMHO-060000   [Line Route ServiceJourneyPattern]
the arguments ['d:/aux_testing_processing/nl_filter/04.mdbx', 'd:/aux_testing_processing/nl_filter/07.mdbx', 'ServiceJourney', 'NL:WPD:ServiceJourney:AMHO-060000', ['Line', 'Route', 'ServiceJourneyPattern']]
source_path: d:/aux_testing_processing/nl_filter/04.mdbx
target_path: d:/aux_testing_processing/nl_filter/07.mdbx
b'[\x00\x00\x004\x05\x00\x00' 8
b'/\x00\x00\x00h\x07\x00\x00' 8
b'>\x00\x00\x009\x05\x00\x00' 8
b'W\x00\x00\x00>\x05\x00\x00' 8
b'+\x00\x00\x00\xff\x04\x00\x00' 8
b'\x1c\x00\x00\x00\xff\x04\x00\x00' 8
b'S\x00\x00\x00\xe4\x02\x00\x00' 8
b'\x1f\x00\x00\x00\x87\x03\x00\x00' 8
b'\xeb\x00\x00\x00\x80\x01\x00\x00' 8
b'I\x00\x00\x00{\x01\x00\x00' 8
b'Y\x00\x00\x00=\x01\x00\x00' 8
b'\n\x00\x00\x009\x05\x00\x00' 8
b'e\x00\x00\x00>\x05\x00\x00' 8
b'\xec\x00\x00\x00\x80\x01\x00\x00' 8
b'=\x00\x00\x00{\x01\x00\x00' 8
b'\xc8\x00\x00\x004\x05\x00\x00' 8
b'\xc6\x00\x00\x00=\x01\x00\x00' 8
b'\xc3\x00\x00\x004\x05\x00\x00' 8
b'\xc1\x00\x00\x00=\x01\x00\x00' 8
b'\xbe\x00\x00\x004\x05\x00\x00' 8
b'\xbc\x00\x00\x00=\x01\x00\x00' 8
b'\xac\x00\x00\x004\x05\x00\x00' 8
b'\xaa\x00\x00\x00=\x01\x00\x00' 8
b'\xa7\x00\x00\x004\x05\x00\x00' 8
b'\xa5\x00\x00\x00=\x01\x00\x00' 8
b'\x9a\x00\x00\x004\x05\x00\x00' 8
b'\x97\x00\x00\x00=\x01\x00\x00' 8
b'\x8a\x00\x00\x004\x05\x00\x00' 8
b'r\x00\x00\x00=\x01\x00\x00' 8
b'\x84\x00\x00\x004\x05\x00\x00' 8
b'}\x00\x00\x004\x05\x00\x00' 8
b't\x00\x00\x004\x05\x00\x00' 8
b'<\x00\x00\x00\xd2\x04\x00\x00' 8
b'\x13\x00\x00\x00\xd9\x04\x00\x00' 8
b'\x04\x00\x00\x00\xd9\x04\x00\x00' 8
b' \x00\x00\x00\xd6\x04\x00\x00' 8
b'E\x00\x00\x00\x85\x03\x00\x00' 8
b'%\x00\x00\x00\xd2\x04\x00\x00' 8
b'\x0e\x00\x00\x00\xd6\x04\x00\x00' 8
b'\xc9\x00\x00\x004\x05\x00\x00' 8
b'\xc4\x00\x00\x004\x05\x00\x00' 8
b'\xbf\x00\x00\x004\x05\x00\x00' 8
b'\xad\x00\x00\x004\x05\x00\x00' 8
b'\xa8\x00\x00\x004\x05\x00\x00' 8
b'\xa1\x00\x00\x004\x05\x00\x00' 8
b'\x90\x00\x00\x004\x05\x00\x00' 8
b'\x87\x00\x00\x004\x05\x00\x00' 8
b'\x80\x00\x00\x004\x05\x00\x00' 8
b'z\x00\x00\x004\x05\x00\x00' 8
b'h\x00\x00\x004\x05\x00\x00' 8
2025-12-24 18:16:28,177 - script_runner - INFO - Execution time: 0.3 seconds for nl_filter - step: 10: conv.filter_db_to_db d:/aux_testing_processing/nl_filter/04.mdbx d:/aux_testing_processing/nl_filter/07.mdbx ServiceJourney NL:WPD:ServiceJourney:AMHO-060000   [Line Route ServiceJourneyPattern]

2025-12-24 18:16:28,178 - script_runner - INFO - nl_filter - step: 11: conv.epip_db_to_xml d:/aux_testing_processing/nl_filter/07.mdbx d:/aux_testing_processing/nl_filter/nl_filter-servicejourney.xml
the arguments ['d:/aux_testing_processing/nl_filter/07.mdbx', 'd:/aux_testing_processing/nl_filter/nl_filter-servicejourney.xml']
2025-12-24 18:16:28,372 - script_runner - INFO - Execution time: 0.1 seconds for nl_filter - step: 11: conv.epip_db_to_xml d:/aux_testing_processing/nl_filter/07.mdbx d:/aux_testing_processing/nl_filter/nl_filter-servicejourney.xml

2025-12-24 18:16:28,373 - script_runner - INFO - nl_filter - step: 12: tools.xml_validator.py d:/aux_testing_processing/nl_filter/nl_filter-netex.xml domain/netex/schema/xsd/NeTEx_publication.xsd
the arguments ['d:/aux_testing_processing/nl_filter/nl_filter-netex.xml', 'domain/netex/schema/xsd/NeTEx_publication.xsd']
2025-12-24 18:16:51,354 - script_runner - INFO - Execution time: 22.9 seconds for nl_filter - step: 12: tools.xml_validator.py d:/aux_testing_processing/nl_filter/nl_filter-netex.xml domain/netex/schema/xsd/NeTEx_publication.xsd

@skinkie
Copy link
Contributor

skinkie commented Dec 24, 2025

@ue71603 you have hardcoded somewhere in the code that the logging should go do "d:". No clue where this happens yet.

So that is configuration.py. Validate that if a non existing Microsoft Windows drive is specified that the code won't break.

@skinkie
Copy link
Contributor

skinkie commented Dec 24, 2025

Can repoduce the "filename too long" error on Windows.

@skinkie
Copy link
Contributor

skinkie commented Dec 24, 2025

@ue71603

In your local code open mdbx.py (for example by clicking on TXN). Search for MDBX_ENODATA, replace the list with this one below.

    MDBX_ENODATA = 38
    MDBX_EINVAL = 87
    MDBX_EACCES = 5
    MDBX_ENOMEM = 14
    MDBX_EROFS = 6008
    MDBX_ENOSYS = 50
    MDBX_EIO = 29
    MDBX_EPERM = 1
    MDBX_EINTR = 1223
    MDBX_ENOFILE = 2
    MDBX_EREMOTE = 4352

Pull request upstream: wtdcode/mdbx-py#14

)
from domain.netex.model.name_of_class_operating_period_ref_structure import NameOfClassOperatingPeriodRefStructure
#TODO remove as an correct
from domain.netex.model.name_of_class_operating_period_ref_structure_type import NameOfClassOperatingPeriodRefStructureType as NameOfClassOperatingPeriodRefStructure
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please elaborate why the renaming is done.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@skinkie it seems that now all are _type in the end. In this file they were not. So i changed the import, but not yet the content of gtfs.py. At least in my version. Or is something wrong with my model again (regenerated it before I started)

# TODO: ResponsibilitySet to Operator/Authority should be a separate function
responsibility_set = cast(
ResponsibilitySet, db_read.load_object_by_reference(txn, getFakeRef(line.responsibility_set_ref_attribute, ResponsibilitySetRef))
ResponsibilitySet, db_read.load_object_by_reference(txn, getFakeRef(line.responsibility_set_ref_attribute, ResponsibilitySetRef,line.version))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

black would have saved the humilation!

Suggested change
ResponsibilitySet, db_read.load_object_by_reference(txn, getFakeRef(line.responsibility_set_ref_attribute, ResponsibilitySetRef,line.version))
ResponsibilitySet, db_read.load_object_by_reference(txn, getFakeRef(line.responsibility_set_ref_attribute, ResponsibilitySetRef, line.version))

But I wonder why we give responsibilityset a version at all.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know, but it wants to get a version parameter.

ue71603 and others added 4 commits December 28, 2025 19:59
sorry forgot to remove it

Co-authored-by: Stefan de Konink <stefan@konink.de>
Co-authored-by: Stefan de Konink <stefan@konink.de>
…eType in epip.py

- fixed getId in two files,
- fixed NameOfClassOperatingPEriodRefStructureType in epip.py
@skinkie
Copy link
Contributor

skinkie commented Dec 30, 2025

I'll merge when I am done with the branch.

as route_desc is hard-coded in the select, it would break otherwise later in the next step. However, you need to check if this is ok with your code.
tuxalp and others added 3 commits January 7, 2026 12:32
* SKIPLUS-1116: remove repleated lines.

* SKIPLUS-1116: adapt example about how to run xml_validator from command line.
# Conflicts:
#	transformers/gtfs.py
@skinkie
Copy link
Contributor

skinkie commented Jan 8, 2026

@ue71603 ready to merge this branch?

@ue71603
Copy link
Collaborator Author

ue71603 commented Jan 8, 2026

@skinkie we can merge this into binary... The other problem I am still analysing but we recreate the branch....

@skinkie skinkie merged commit 10444d9 into binary_relation_serializer Jan 8, 2026
@skinkie skinkie deleted the update_2025-12-23 branch January 8, 2026 18:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants