Collaborative Launch: 1751385600 (Tuesday, July 1, 16:00:00 UTC)

[jurraca]

Next proposed collaborative launch date is 1751385600 (Tuesday, July 1, 16:00:00 UTC).

Last time we had several runs get different RPKI repository data, which resulted in different output hashes for the ASmap file produced. We've tried a few things to remediate this (asmap/kartograf#78, asmap/kartograf#82) and haven't come up with a proper solution; the RPKI repos should be able to handle many more simultaneous connections than what we're doing. However, it's been over 3 months since our last run, so we need to produce a new ASmap for general usage, and confirm whether the issue persists.

You can run the following command to set kartograf to wait and run at that time:

./run map -w=1751385600 -irr -rv

The latest kartograf release is 0.4.10 "Ptolemy".
The requirements are Python 3.11 and rpki-client 9.5. We recommend you use Nix and run nix develop to enter an environment with all the required dependencies included. Otherwise, see the setup instructions for your platform in the README.

If people would rather we push it to a different date, please let us know below.

@fjahr @0xB10C @dunxen @Sjors @luisschwab @Emzy @jonatack @freerko @sr-gi

[0xB10C]

I'll start it, but won't be able to post my results immediately.

[fjahr]

I got 824e5336642ddd5007511acec61c3b2ceaccda0eb6490c46e0953cac506dafe0

logs

$ ./run map -w=1751385600 -irr -rv

--- Start Kartograf ---

Kartograf version: 0.4.10
Using rpki-client version 9.5 (recommended).
Coordinated launch mode: Waiting until 1751385600 (2025-07-01 18:00:00 CEST) to launch mapping process.
Countdown: 0 second(s)
Starting...
The epoch for this run is: 1751385600 (2025-07-01 16:00:00 UTC, local: 2025-07-01 18:00:00 CEST)

--- Fetching RPKI ---

Downloaded TAL for AFRINIC to /Users/FJ/projects/python/kartograf/data/1751385600/rpki/tals/afrinic.tal, file hash: 2838ef30ea27ce5705abf5f5adb131d8c35b1f50858338a2f3c84bb207c2fa35
Downloaded TAL for APNIC to /Users/FJ/projects/python/kartograf/data/1751385600/rpki/tals/apnic.tal, file hash: 472e551f7c551c2e999e582b7c9437d3bee4900fe53afff62aeb28d4940ade94
Downloaded TAL for ARIN to /Users/FJ/projects/python/kartograf/data/1751385600/rpki/tals/arin.tal, file hash: 1f8bdb03bcc30a3b8e11fd9a87102fba250c22137a3c8baa9c81b139cb412639
Downloaded TAL for LACNIC to /Users/FJ/projects/python/kartograf/data/1751385600/rpki/tals/lacnic.tal, file hash: d44bb9394ab009c8b53e5efebf2a1c9450bab61a27efe00de5a3e4587a3a2f6a
Downloaded TAL for RIPE to /Users/FJ/projects/python/kartograf/data/1751385600/rpki/tals/ripe.tal, file hash: 59ca27ef93f23682749fcefe7c6d70fbc723343549ff9e4d3996acaff79817fb
Downloading RPKI Data, this may take a while.
Downloaded RPKI Data, hash sum: 3e9229cab41e7053b0888fe586dd95842bb67f3bc1df7df176e76fae78afca75
...finished in 0:04:35.939191

--- Fetching IRR ---

Downloading afrinic.db.gz
Downloaded afrinic.db.gz, file hash: 65881e9aee7b3f26a56a7f67d4cbd875968d6b5682939e3df1b4f84729fff3e4
Downloading apnic.db.route.gz
Downloaded apnic.db.route.gz, file hash: 794d6fb922548c44817c54e693b521a71932d1bb287ce896abe5ec3f43cc30af
Downloading apnic.db.route6.gz
Downloaded apnic.db.route6.gz, file hash: 40ff09ff1d32c0619b58908f3f028c4dfa478b753170deed7108cd8ec7bf1fe6
Downloading arin.db.gz
Downloaded arin.db.gz, file hash: 38b793fd1eb1fc959bdcbdc8396d7c7bd5f358007f5af2d6bf1828bd190dd191
Downloading lacnic.db.gz
Downloaded lacnic.db.gz, file hash: a9ae17cd7e5dec5a6d09f6b61d3cc6e063df46f9f6590e7db30ba59b5cfc1563
Downloading ripe.db.route.gz
Downloaded ripe.db.route.gz, file hash: 76c9db4e7a3bb6b547db0a0717631524126d6556166ccd5fe0f4ad05eabded14
Downloading ripe.db.route6.gz
Downloaded ripe.db.route6.gz, file hash: 9bb28f7409e83a343c5d4a003b2c971866b47a55ebb61826edcdb763524675bb
...finished in 0:00:27.766238

--- Fetching Routeviews pfx2as ---

The page at https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2025/07/ couldn't be fetched. Trying the previous month.
Downloading from https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2025/06/routeviews-rv2-20250629-1200.pfx2as.gz
Downloaded /Users/FJ/projects/python/kartograf/data/1751385600/collectors/routeviews_pfx2asn_ip4.txt.gz, file hash: c4204182bab19ac1eadaf41269f270e48a95f63c31ba70690833014d67ffe9f6
The page at https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2025/07/ couldn't be fetched. Trying the previous month.
Downloading from https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2025/06/routeviews-rv6-20250630-2000.pfx2as.gz
Downloaded /Users/FJ/projects/python/kartograf/data/1751385600/collectors/routeviews_pfx2asn_ip6.txt.gz, file hash: f7d0c19711cf54bf76f144ad40b3441990308062da0a4717a346dbd5c5509db0
...finished in 0:00:08.579822

--- Validating RPKI ---

Validating RPKI ROAs
317660 raw RKPI ROA files found.
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1271/1271 [00:43<00:00, 29.05it/s]
317660 RKPI ROAs validated and saved to /Users/FJ/projects/python/kartograf/out/1751385600/rpki/rpki_raw.json, file hash: 5db02c777b4645b9f6a0c97cd299d195f6729a746734d91f2e0591efd4552431
...finished in 0:01:01.839327

--- Parsing RPKI ---

Parsing 317660 ROAs
Result entries written: 634280
Duplicates found: 92304
Invalids found: 2275
Incompletes: 0
Non-ROA files: 0
...finished in 0:00:13.863882

--- Parsing IRR ---

Extracting afrinic.db.gz
Extracting apnic.db.route.gz
Extracting apnic.db.route6.gz
Extracting arin.db.gz
Extracting lacnic.db.gz
Extracting ripe.db.route.gz
Extracting ripe.db.route6.gz
Parsing /Users/FJ/projects/python/kartograf/out/1751385600/irr/lacnic.db
Found in this file: 15280
Parsing /Users/FJ/projects/python/kartograf/out/1751385600/irr/ripe.db.route
Found in this file: 386802
Parsing /Users/FJ/projects/python/kartograf/out/1751385600/irr/apnic.db.route6
Found in this file: 632639
Parsing /Users/FJ/projects/python/kartograf/out/1751385600/irr/apnic.db.route
Found in this file: 611242
Parsing /Users/FJ/projects/python/kartograf/out/1751385600/irr/arin.db
Found in this file: 107461
Parsing /Users/FJ/projects/python/kartograf/out/1751385600/irr/afrinic.db
Found in this file: 116768
Parsing /Users/FJ/projects/python/kartograf/out/1751385600/irr/ripe.db.route6
Found in this file: 149461
Found valid, unique entries: 2019653
...finished in 0:00:51.088134

--- Merging RPKI and IRR data ---

Parse base file to dictionary
Parse extra file to Pandas DataFrame
Merging extra prefixes that were not included in the base file.
Finished merging extra prefixes.
Finished filtering! Originally 2019653 entries filtered down to 428849
Merging base file with filtered extra file
...finished in 0:03:39.842832

--- Parsing Routeviews pfx2as ---

Unzipping /Users/FJ/projects/python/kartograf/data/1751385600/collectors/routeviews_pfx2asn_ip4.txt.gz
Formatting /Users/FJ/projects/python/kartograf/out/1751385600/collectors/routeviews_pfx2asn_ip4.txt
Unzipping /Users/FJ/projects/python/kartograf/data/1751385600/collectors/routeviews_pfx2asn_ip6.txt.gz
Formatting /Users/FJ/projects/python/kartograf/out/1751385600/collectors/routeviews_pfx2asn_ip6.txt
Cleaning /Users/FJ/projects/python/kartograf/out/1751385600/collectors/pfx2asn.txt
Entries after cleanup: 1273630
...finished in 0:00:18.486839

--- Merging Routeviews and base data ---

Parse base file to dictionary
Parse extra file to Pandas DataFrame
Merging extra prefixes that were not included in the base file.
Finished merging extra prefixes.
Finished filtering! Originally 1273630 entries filtered down to 335368
Merging base file with filtered extra file
...finished in 0:02:58.543762

--- Sorting results ---

...finished in 0:00:07.170504

--- Finishing Kartograf ---

The SHA-256 hash of the result file is: 824e5336642ddd5007511acec61c3b2ceaccda0eb6490c46e0953cac506dafe0
Total runtime: 0:14:24.095579

[Sjors]

The SHA-256 hash of the result file is: 620203796293f1e4f84336cc23b723ebaeefe198f6e4bf27f535370f0184d185

logs

./run map -w=1751385600  -irr -rv

--- Start Kartograf ---

Kartograf version: 0.4.10
Using rpki-client version 9.5 (recommended).
Coordinated launch mode: Waiting until 1751385600 (2025-07-01 18:00:00 CEST) to launch mapping process.
Countdown: 8 minute(s), 39 second(s)
Countdown: 0 second(s)
Starting...
The epoch for this run is: 1751385600 (2025-07-01 16:00:00 UTC, local: 2025-07-01 18:00:00 CEST)

--- Fetching RPKI ---

Downloaded TAL for AFRINIC to /home/sjors/dev/kartograf/data/1751385600/rpki/tals/afrinic.tal, file hash: 2838ef30ea27ce5705abf5f5adb131d8c35b1f50858338a2f3c84bb207c2fa35
Downloaded TAL for APNIC to /home/sjors/dev/kartograf/data/1751385600/rpki/tals/apnic.tal, file hash: 472e551f7c551c2e999e582b7c9437d3bee4900fe53afff62aeb28d4940ade94
Downloaded TAL for ARIN to /home/sjors/dev/kartograf/data/1751385600/rpki/tals/arin.tal, file hash: 1f8bdb03bcc30a3b8e11fd9a87102fba250c22137a3c8baa9c81b139cb412639
Downloaded TAL for LACNIC to /home/sjors/dev/kartograf/data/1751385600/rpki/tals/lacnic.tal, file hash: d44bb9394ab009c8b53e5efebf2a1c9450bab61a27efe00de5a3e4587a3a2f6a
Downloaded TAL for RIPE to /home/sjors/dev/kartograf/data/1751385600/rpki/tals/ripe.tal, file hash: 59ca27ef93f23682749fcefe7c6d70fbc723343549ff9e4d3996acaff79817fb
Downloading RPKI Data, this may take a while.
Downloaded RPKI Data, hash sum: 39df623996e3f662d2dcb76bc337a78183c94e148bf6b7ac17d865ca54406b40
...finished in 0:02:21.448351

--- Fetching IRR ---

Downloading afrinic.db.gz
Downloaded afrinic.db.gz, file hash: 65881e9aee7b3f26a56a7f67d4cbd875968d6b5682939e3df1b4f84729fff3e4
Downloading apnic.db.route.gz
Downloaded apnic.db.route.gz, file hash: 794d6fb922548c44817c54e693b521a71932d1bb287ce896abe5ec3f43cc30af
Downloading apnic.db.route6.gz
Downloaded apnic.db.route6.gz, file hash: 40ff09ff1d32c0619b58908f3f028c4dfa478b753170deed7108cd8ec7bf1fe6
Downloading arin.db.gz
Downloaded arin.db.gz, file hash: 38b793fd1eb1fc959bdcbdc8396d7c7bd5f358007f5af2d6bf1828bd190dd191
Downloading lacnic.db.gz
Downloaded lacnic.db.gz, file hash: a9ae17cd7e5dec5a6d09f6b61d3cc6e063df46f9f6590e7db30ba59b5cfc1563
Downloading ripe.db.route.gz
Downloaded ripe.db.route.gz, file hash: 76c9db4e7a3bb6b547db0a0717631524126d6556166ccd5fe0f4ad05eabded14
Downloading ripe.db.route6.gz
Downloaded ripe.db.route6.gz, file hash: 9bb28f7409e83a343c5d4a003b2c971866b47a55ebb61826edcdb763524675bb
...finished in 0:00:22.192501

--- Fetching Routeviews pfx2as ---

The page at https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2025/07/ couldn't be fetched. Trying the previous month.
Downloading from https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2025/06/routeviews-rv2-20250629-1200.pfx2as.gz
Downloaded /home/sjors/dev/kartograf/data/1751385600/collectors/routeviews_pfx2asn_ip4.txt.gz, file hash: c4204182bab19ac1eadaf41269f270e48a95f63c31ba70690833014d67ffe9f6
The page at https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2025/07/ couldn't be fetched. Trying the previous month.
Downloading from https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2025/06/routeviews-rv6-20250630-2000.pfx2as.gz
Downloaded /home/sjors/dev/kartograf/data/1751385600/collectors/routeviews_pfx2asn_ip6.txt.gz, file hash: f7d0c19711cf54bf76f144ad40b3441990308062da0a4717a346dbd5c5509db0
...finished in 0:00:08.141338

--- Validating RPKI ---

Validating RPKI ROAs
317712 raw RKPI ROA files found.
100%|████████████████████████████████████████████████████████████████████████████████████████████████| 1271/1271 [00:19<00:00, 65.97it/s]
317712 RKPI ROAs validated and saved to /home/sjors/dev/kartograf/out/1751385600/rpki/rpki_raw.json, file hash: 5cb95e40a8f55a4dc31fcd470178edc02713f8f0ac05021350cd5a3c2f9edbf3
...finished in 0:00:30.917340

--- Parsing RPKI ---

Parsing 317712 ROAs
Result entries written: 634310
Duplicates found: 92307
Invalids found: 2295
Incompletes: 0
Non-ROA files: 0
...finished in 0:00:21.181694

--- Parsing IRR ---

Extracting afrinic.db.gz
Extracting apnic.db.route.gz
Extracting apnic.db.route6.gz
Extracting arin.db.gz
Extracting lacnic.db.gz
Extracting ripe.db.route.gz
Extracting ripe.db.route6.gz
Parsing /home/sjors/dev/kartograf/out/1751385600/irr/arin.db
Found in this file: 107461
Parsing /home/sjors/dev/kartograf/out/1751385600/irr/afrinic.db
Found in this file: 116768
Parsing /home/sjors/dev/kartograf/out/1751385600/irr/lacnic.db
Found in this file: 15280
Parsing /home/sjors/dev/kartograf/out/1751385600/irr/ripe.db.route
Found in this file: 386802
Parsing /home/sjors/dev/kartograf/out/1751385600/irr/apnic.db.route6
Found in this file: 632639
Parsing /home/sjors/dev/kartograf/out/1751385600/irr/ripe.db.route6
Found in this file: 149461
Parsing /home/sjors/dev/kartograf/out/1751385600/irr/apnic.db.route
Found in this file: 611242
Found valid, unique entries: 2019653
...finished in 0:01:22.783888

--- Merging RPKI and IRR data ---

Parse base file to dictionary
Parse extra file to Pandas DataFrame
Merging extra prefixes that were not included in the base file.
Finished merging extra prefixes.
Finished filtering! Originally 2019653 entries filtered down to 428825
Merging base file with filtered extra file
...finished in 0:05:15.479608

--- Parsing Routeviews pfx2as ---

Unzipping /home/sjors/dev/kartograf/data/1751385600/collectors/routeviews_pfx2asn_ip4.txt.gz
Formatting /home/sjors/dev/kartograf/out/1751385600/collectors/routeviews_pfx2asn_ip4.txt
Unzipping /home/sjors/dev/kartograf/data/1751385600/collectors/routeviews_pfx2asn_ip6.txt.gz
Formatting /home/sjors/dev/kartograf/out/1751385600/collectors/routeviews_pfx2asn_ip6.txt
Cleaning /home/sjors/dev/kartograf/out/1751385600/collectors/pfx2asn.txt
Entries after cleanup: 1273630
...finished in 0:00:29.873702

--- Merging Routeviews and base data ---

Parse base file to dictionary
Parse extra file to Pandas DataFrame
Merging extra prefixes that were not included in the base file.
Finished merging extra prefixes.
Finished filtering! Originally 1273630 entries filtered down to 335362
Merging base file with filtered extra file
...finished in 0:04:24.285850

--- Sorting results ---

...finished in 0:00:13.652969

--- Finishing Kartograf ---

The SHA-256 hash of the result file is: 620203796293f1e4f84336cc23b723ebaeefe198f6e4bf27f535370f0184d185
Total runtime: 0:15:31.299108

First mismatch with @fjahr is the "Downloaded RPKI Data".

[dunxen]

I got 95ff01a5f594db1934b2532ec76a05e695ba6c9c988a4c2c465a06ff70f13319

logs

./run map -w=1751385600 -irr -rv

--- Start Kartograf ---

Kartograf version: 0.4.10
Using rpki-client version 9.5 (recommended).
Coordinated launch mode: Waiting until 1751385600 (2025-07-01 18:00:00 SAST) to launch mapping process.
Countdown: 0 second(s)
Starting...
The epoch for this run is: 1751385600 (2025-07-01 16:00:00 UTC, local: 2025-07-01 18:00:00 SAST)

--- Fetching RPKI ---

Downloaded TAL for AFRINIC to /home/dunxen/repos/github.com/asmap/kartograf/data/1751385600/rpki/tals/afrinic.tal, file hash: 2838ef30ea27ce5705abf5f5adb131d8c35b1f50858338a2f3c84bb207c2fa35
Downloaded TAL for APNIC to /home/dunxen/repos/github.com/asmap/kartograf/data/1751385600/rpki/tals/apnic.tal, file hash: 472e551f7c551c2e999e582b7c9437d3bee4900fe53afff62aeb28d4940ade94
Downloaded TAL for ARIN to /home/dunxen/repos/github.com/asmap/kartograf/data/1751385600/rpki/tals/arin.tal, file hash: 1f8bdb03bcc30a3b8e11fd9a87102fba250c22137a3c8baa9c81b139cb412639
Downloaded TAL for LACNIC to /home/dunxen/repos/github.com/asmap/kartograf/data/1751385600/rpki/tals/lacnic.tal, file hash: d44bb9394ab009c8b53e5efebf2a1c9450bab61a27efe00de5a3e4587a3a2f6a
Downloaded TAL for RIPE to /home/dunxen/repos/github.com/asmap/kartograf/data/1751385600/rpki/tals/ripe.tal, file hash: 59ca27ef93f23682749fcefe7c6d70fbc723343549ff9e4d3996acaff79817fb
Downloading RPKI Data, this may take a while.
Downloaded RPKI Data, hash sum: 6b04e8457ef582060ae6422803bc4290dc2db0f46f9937b8195744f18d5f2566
...finished in 0:12:22.177602

--- Fetching IRR ---

Downloading afrinic.db.gz
Downloaded afrinic.db.gz, file hash: 65881e9aee7b3f26a56a7f67d4cbd875968d6b5682939e3df1b4f84729fff3e4
Downloading apnic.db.route.gz
Downloaded apnic.db.route.gz, file hash: 794d6fb922548c44817c54e693b521a71932d1bb287ce896abe5ec3f43cc30af
Downloading apnic.db.route6.gz
Downloaded apnic.db.route6.gz, file hash: 40ff09ff1d32c0619b58908f3f028c4dfa478b753170deed7108cd8ec7bf1fe6
Downloading arin.db.gz
Downloaded arin.db.gz, file hash: 38b793fd1eb1fc959bdcbdc8396d7c7bd5f358007f5af2d6bf1828bd190dd191
Downloading lacnic.db.gz
Downloaded lacnic.db.gz, file hash: a9ae17cd7e5dec5a6d09f6b61d3cc6e063df46f9f6590e7db30ba59b5cfc1563
Downloading ripe.db.route.gz
Downloaded ripe.db.route.gz, file hash: 76c9db4e7a3bb6b547db0a0717631524126d6556166ccd5fe0f4ad05eabded14
Downloading ripe.db.route6.gz
Connection issue while downloading ripe.db.route6.gz: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). Retrying... (Attempt 1/5)
Connection issue while downloading ripe.db.route6.gz: ('Connection broken: IncompleteRead(32483 bytes read, 2647155 more expected)', IncompleteRead(32483 bytes read, 2647155 more expected)). Retrying... (Attempt 2/5)
Downloaded ripe.db.route6.gz, file hash: 9bb28f7409e83a343c5d4a003b2c971866b47a55ebb61826edcdb763524675bb
...finished in 0:01:07.146135

--- Fetching Routeviews pfx2as ---

The page at https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2025/07/ couldn't be fetched. Trying the previous month.
Downloading from https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2025/06/routeviews-rv2-20250629-1200.pfx2as.gz
Downloaded /home/dunxen/repos/github.com/asmap/kartograf/data/1751385600/collectors/routeviews_pfx2asn_ip4.txt.gz, file hash: c4204182bab19ac1eadaf41269f270e48a95f63c31ba70690833014d67ffe9f6
The page at https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2025/07/ couldn't be fetched. Trying the previous month.
Downloading from https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2025/06/routeviews-rv6-20250630-2000.pfx2as.gz
Downloaded /home/dunxen/repos/github.com/asmap/kartograf/data/1751385600/collectors/routeviews_pfx2asn_ip6.txt.gz, file hash: f7d0c19711cf54bf76f144ad40b3441990308062da0a4717a346dbd5c5509db0
...finished in 0:00:16.013588

--- Validating RPKI ---

Validating RPKI ROAs
317570 raw RKPI ROA files found.
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1271/1271 [00:43<00:00, 29.40it/s]
317570 RKPI ROAs validated and saved to /home/dunxen/repos/github.com/asmap/kartograf/out/1751385600/rpki/rpki_raw.json, file hash: 53c17a700f6881650fef4e38214a4326f8a158ab0eb235aeb3eede23c40fa93f
...finished in 0:01:01.776811

--- Parsing RPKI ---

Parsing 317570 ROAs
Result entries written: 624726
Duplicates found: 91177
Invalids found: 11008
Incompletes: 0
Non-ROA files: 0
...finished in 0:00:24.271627

--- Parsing IRR ---

Extracting afrinic.db.gz
Extracting apnic.db.route.gz
Extracting apnic.db.route6.gz
Extracting arin.db.gz
Extracting lacnic.db.gz
Extracting ripe.db.route.gz
Extracting ripe.db.route6.gz
Parsing /home/dunxen/repos/github.com/asmap/kartograf/out/1751385600/irr/afrinic.db
Found in this file: 116768
Parsing /home/dunxen/repos/github.com/asmap/kartograf/out/1751385600/irr/apnic.db.route
Found in this file: 611242
Parsing /home/dunxen/repos/github.com/asmap/kartograf/out/1751385600/irr/apnic.db.route6
Found in this file: 632639
Parsing /home/dunxen/repos/github.com/asmap/kartograf/out/1751385600/irr/arin.db
Found in this file: 107461
Parsing /home/dunxen/repos/github.com/asmap/kartograf/out/1751385600/irr/lacnic.db
Found in this file: 15280
Parsing /home/dunxen/repos/github.com/asmap/kartograf/out/1751385600/irr/ripe.db.route
Found in this file: 386802
Parsing /home/dunxen/repos/github.com/asmap/kartograf/out/1751385600/irr/ripe.db.route6
Found in this file: 149461
Found valid, unique entries: 2019653
...finished in 0:01:49.415681

--- Merging RPKI and IRR data ---

Parse base file to dictionary
Parse extra file to Pandas DataFrame
Merging extra prefixes that were not included in the base file.
Finished merging extra prefixes.
Finished filtering! Originally 2019653 entries filtered down to 436016
Merging base file with filtered extra file
...finished in 0:06:20.528712

--- Parsing Routeviews pfx2as ---

Unzipping /home/dunxen/repos/github.com/asmap/kartograf/data/1751385600/collectors/routeviews_pfx2asn_ip4.txt.gz
Formatting /home/dunxen/repos/github.com/asmap/kartograf/out/1751385600/collectors/routeviews_pfx2asn_ip4.txt
Unzipping /home/dunxen/repos/github.com/asmap/kartograf/data/1751385600/collectors/routeviews_pfx2asn_ip6.txt.gz
Formatting /home/dunxen/repos/github.com/asmap/kartograf/out/1751385600/collectors/routeviews_pfx2asn_ip6.txt
Cleaning /home/dunxen/repos/github.com/asmap/kartograf/out/1751385600/collectors/pfx2asn.txt
Entries after cleanup: 1273630
...finished in 0:00:33.820904

--- Merging Routeviews and base data ---

Parse base file to dictionary
Parse extra file to Pandas DataFrame
Merging extra prefixes that were not included in the base file.
Finished merging extra prefixes.
Finished filtering! Originally 1273630 entries filtered down to 338459
Merging base file with filtered extra file
...finished in 0:05:12.869793

--- Sorting results ---

...finished in 0:00:14.049262

--- Finishing Kartograf ---

The SHA-256 hash of the result file is: 95ff01a5f594db1934b2532ec76a05e695ba6c9c988a4c2c465a06ff70f13319
Total runtime: 0:29:24.147487

[jurraca]

I got: 2e9219125d1fd7bf90b403a025e66f85fb90f1c4a8783a7bbe134f643116672c

logs

--- Start Kartograf ---

Kartograf version: 0.4.10
Using rpki-client version 9.5 (recommended).
Coordinated launch mode: Waiting until 1751385600 (2025-07-01 18:00:00 CEST) to launch mapping process.
Countdown: 0 second(s)
Starting...
The epoch for this run is: 1751385600 (2025-07-01 16:00:00 UTC, local: 2025-07-01 18:00:00 CEST)

--- Fetching RPKI ---

Downloaded TAL for AFRINIC to /home/base/code/asmap/kartograf/data/1751385600/rpki/tals/afrinic.tal, file hash: 2838ef30ea27ce
5705abf5f5adb131d8c35b1f50858338a2f3c84bb207c2fa35
Downloaded TAL for APNIC to /home/base/code/asmap/kartograf/data/1751385600/rpki/tals/apnic.tal, file hash: 472e551f7c551c2e99
9e582b7c9437d3bee4900fe53afff62aeb28d4940ade94
Downloaded TAL for ARIN to /home/base/code/asmap/kartograf/data/1751385600/rpki/tals/arin.tal, file hash: 1f8bdb03bcc30a3b8e11
fd9a87102fba250c22137a3c8baa9c81b139cb412639
Downloaded TAL for LACNIC to /home/base/code/asmap/kartograf/data/1751385600/rpki/tals/lacnic.tal, file hash: d44bb9394ab009c8
b53e5efebf2a1c9450bab61a27efe00de5a3e4587a3a2f6a
Downloaded TAL for RIPE to /home/base/code/asmap/kartograf/data/1751385600/rpki/tals/ripe.tal, file hash: 59ca27ef93f23682749f
cefe7c6d70fbc723343549ff9e4d3996acaff79817fb
Downloading RPKI Data, this may take a while.
Downloaded RPKI Data, hash sum: 717d67d6f2aa7c75b843dfd0b2439d3d99fb55721410d889cfe4631c20401a31
...finished in 0:05:07.274786

--- Fetching IRR ---

Downloading afrinic.db.gz
Downloaded afrinic.db.gz, file hash: 65881e9aee7b3f26a56a7f67d4cbd875968d6b5682939e3df1b4f84729fff3e4
Downloading apnic.db.route.gz
Downloaded apnic.db.route.gz, file hash: 794d6fb922548c44817c54e693b521a71932d1bb287ce896abe5ec3f43cc30af
Downloading apnic.db.route6.gz
Downloaded apnic.db.route6.gz, file hash: 40ff09ff1d32c0619b58908f3f028c4dfa478b753170deed7108cd8ec7bf1fe6
Downloading arin.db.gz
Downloaded arin.db.gz, file hash: 38b793fd1eb1fc959bdcbdc8396d7c7bd5f358007f5af2d6bf1828bd190dd191
Downloading lacnic.db.gz
Downloaded lacnic.db.gz, file hash: a9ae17cd7e5dec5a6d09f6b61d3cc6e063df46f9f6590e7db30ba59b5cfc1563
Downloading ripe.db.route.gz
Downloaded ripe.db.route.gz, file hash: 76c9db4e7a3bb6b547db0a0717631524126d6556166ccd5fe0f4ad05eabded14
Downloading ripe.db.route6.gz
Downloaded ripe.db.route6.gz, file hash: 9bb28f7409e83a343c5d4a003b2c971866b47a55ebb61826edcdb763524675bb
...finished in 0:01:16.148176

--- Fetching Routeviews pfx2as ---

The page at https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2025/07/ couldn't be fetched. Trying the previo
us month.
Downloading from https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2025/06/routeviews-rv2-20250629-1200.pfx2a
s.gz
Downloaded /home/base/code/asmap/kartograf/data/1751385600/collectors/routeviews_pfx2asn_ip4.txt.gz, file hash: c4204182bab19a
c1eadaf41269f270e48a95f63c31ba70690833014d67ffe9f6
The page at https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2025/07/ couldn't be fetched. Trying the previ
ous month.
Downloading from https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2025/06/routeviews-rv6-20250630-2000.pfx2
as.gz
Downloaded /home/base/code/asmap/kartograf/data/1751385600/collectors/routeviews_pfx2asn_ip6.txt.gz, file hash: f7d0c19711cf54
bf76f144ad40b3441990308062da0a4717a346dbd5c5509db0
...finished in 0:00:19.960371

--- Validating RPKI ---

Validating RPKI ROAs
317641 raw RKPI ROA files found.
100%|█████████████████████████████████████████████████████████████████████████████████████| 1271/1271 [02:29<00:00,  8.50it/s]
317641 RKPI ROAs validated and saved to /home/base/code/asmap/kartograf/out/1751385600/rpki/rpki_raw.json, file hash: 2962fb6d
5c98001f01d663c7cc3b6c232bcf942f1f4b1d6c678dfd637d34f0e5
...finished in 0:02:52.011286

--- Parsing RPKI ---

Parsing 317641 ROAs
Result entries written: 634260
Duplicates found: 92305
Invalids found: 2275
Incompletes: 0
Non-ROA files: 0
...finished in 0:00:35.319185

--- Parsing IRR ---

Extracting afrinic.db.gz
Extracting apnic.db.route.gz
Extracting apnic.db.route6.gz
Extracting arin.db.gz
Extracting lacnic.db.gz
Extracting ripe.db.route.gz
Extracting ripe.db.route6.gz
Parsing /home/base/code/asmap/kartograf/out/1751385600/irr/afrinic.db
Found in this file: 116768
Parsing /home/base/code/asmap/kartograf/out/1751385600/irr/arin.db
Found in this file: 107461
Parsing /home/base/code/asmap/kartograf/out/1751385600/irr/ripe.db.route6
Found in this file: 149461
Parsing /home/base/code/asmap/kartograf/out/1751385600/irr/apnic.db.route6
Found in this file: 632639
Parsing /home/base/code/asmap/kartograf/out/1751385600/irr/lacnic.db
Found in this file: 15280
Parsing /home/base/code/asmap/kartograf/out/1751385600/irr/ripe.db.route
Found in this file: 386802
Parsing /home/base/code/asmap/kartograf/out/1751385600/irr/apnic.db.route
Found in this file: 611242
Found valid, unique entries: 2019653
...finished in 0:02:33.746077

--- Merging RPKI and IRR data ---

Parse base file to dictionary
Parse extra file to Pandas DataFrame
Merging extra prefixes that were not included in the base file.
Finished merging extra prefixes.
Finished filtering! Originally 2019653 entries filtered down to 428862
Merging base file with filtered extra file
...finished in 0:12:56.317018

--- Parsing Routeviews pfx2as ---

Unzipping /home/base/code/asmap/kartograf/data/1751385600/collectors/routeviews_pfx2asn_ip4.txt.gz
Formatting /home/base/code/asmap/kartograf/out/1751385600/collectors/routeviews_pfx2asn_ip4.txt
Unzipping /home/base/code/asmap/kartograf/data/1751385600/collectors/routeviews_pfx2asn_ip6.txt.gz
Formatting /home/base/code/asmap/kartograf/out/1751385600/collectors/routeviews_pfx2asn_ip6.txt
Cleaning /home/base/code/asmap/kartograf/out/1751385600/collectors/pfx2asn.txt
Entries after cleanup: 1273630
...finished in 0:00:51.142852

--- Merging Routeviews and base data ---

Parse base file to dictionary
Parse extra file to Pandas DataFrame
Merging extra prefixes that were not included in the base file.
Finished merging extra prefixes.
Finished filtering! Originally 1273630 entries filtered down to 335369
Merging base file with filtered extra file
...finished in 0:10:59.042783

--- Sorting results ---

...finished in 0:00:20.194678

--- Finishing Kartograf ---

The SHA-256 hash of the result file is: 2e9219125d1fd7bf90b403a025e66f85fb90f1c4a8783a7bbe134f643116672c
Total runtime: 0:37:53.236523

I have to step away for a while, but I'll collate everybody's results and see where we differ. It looks like IRR is the same for everyone, and RPKI is not.

[Emzy]

I got: 620203796293f1e4f84336cc23b723ebaeefe198f6e4bf27f535370f0184d185

logs

--- Start Kartograf ---

Kartograf version: 0.4.10
Using rpki-client version 9.5 (recommended).
Coordinated launch mode: Waiting until 1751385600 (2025-07-01 18:00:00 CEST) to launch mapping process.
Countdown: 0 second(s)
Starting...
The epoch for this run is: 1751385600 (2025-07-01 16:00:00 UTC, local: 2025-07-01 18:00:00 CEST)

--- Fetching RPKI ---

Downloaded TAL for AFRINIC to /home/emzy/kartograf/data/1751385600/rpki/tals/afrinic.tal, file hash: 2838ef30ea27ce5705abf5f5adb131d8c35b1f50858338
a2f3c84bb207c2fa35
Downloaded TAL for APNIC to /home/emzy/kartograf/data/1751385600/rpki/tals/apnic.tal, file hash: 472e551f7c551c2e999e582b7c9437d3bee4900fe53afff62a
eb28d4940ade94
Downloaded TAL for ARIN to /home/emzy/kartograf/data/1751385600/rpki/tals/arin.tal, file hash: 1f8bdb03bcc30a3b8e11fd9a87102fba250c22137a3c8baa9c81
b139cb412639
Downloaded TAL for LACNIC to /home/emzy/kartograf/data/1751385600/rpki/tals/lacnic.tal, file hash: d44bb9394ab009c8b53e5efebf2a1c9450bab61a27efe00d
e5a3e4587a3a2f6a
Downloaded TAL for RIPE to /home/emzy/kartograf/data/1751385600/rpki/tals/ripe.tal, file hash: 59ca27ef93f23682749fcefe7c6d70fbc723343549ff9e4d3996
acaff79817fb
Downloading RPKI Data, this may take a while.
Downloaded RPKI Data, hash sum: fbab3e3478b9d0b19d6e53ed2f2e4d9204c15939df9001bd43bb13eeec1eb2b0
...finished in 0:06:53.032739

--- Fetching IRR ---

Downloading afrinic.db.gz
Downloaded afrinic.db.gz, file hash: 65881e9aee7b3f26a56a7f67d4cbd875968d6b5682939e3df1b4f84729fff3e4
Downloading apnic.db.route.gz
Downloaded apnic.db.route.gz, file hash: 794d6fb922548c44817c54e693b521a71932d1bb287ce896abe5ec3f43cc30af
Downloading apnic.db.route6.gz
Downloaded apnic.db.route6.gz, file hash: 40ff09ff1d32c0619b58908f3f028c4dfa478b753170deed7108cd8ec7bf1fe6
Downloading arin.db.gz
Downloaded arin.db.gz, file hash: 38b793fd1eb1fc959bdcbdc8396d7c7bd5f358007f5af2d6bf1828bd190dd191
Downloading lacnic.db.gz
Downloaded lacnic.db.gz, file hash: a9ae17cd7e5dec5a6d09f6b61d3cc6e063df46f9f6590e7db30ba59b5cfc1563
Downloading ripe.db.route.gz
Downloaded ripe.db.route.gz, file hash: 76c9db4e7a3bb6b547db0a0717631524126d6556166ccd5fe0f4ad05eabded14
Downloading ripe.db.route6.gz
Downloaded ripe.db.route6.gz, file hash: 9bb28f7409e83a343c5d4a003b2c971866b47a55ebb61826edcdb763524675bb
...finished in 0:00:18.219140

--- Fetching Routeviews pfx2as ---

The page at https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2025/07/ couldn't be fetched. Trying the previous month.
Downloading from https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2025/06/routeviews-rv2-20250629-1200.pfx2as.gz
Downloaded /home/emzy/kartograf/data/1751385600/collectors/routeviews_pfx2asn_ip4.txt.gz, file hash: c4204182bab19ac1eadaf41269f270e48a95f63c31ba70
690833014d67ffe9f6
The page at https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2025/07/ couldn't be fetched. Trying the previous month.
Downloading from https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2025/06/routeviews-rv6-20250630-2000.pfx2as.gz
Downloaded /home/emzy/kartograf/data/1751385600/collectors/routeviews_pfx2asn_ip6.txt.gz, file hash: f7d0c19711cf54bf76f144ad40b3441990308062da0a47
17a346dbd5c5509db0
...finished in 0:00:07.610400

--- Validating RPKI ---

Validating RPKI ROAs
317712 raw RKPI ROA files found.
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1271/1271 [07:06<00:00, 2.98it/
s]
317712 RKPI ROAs validated and saved to /home/emzy/kartograf/out/1751385600/rpki/rpki_raw.json, file hash: 0111c49ac866c1109017fd46f7cc621fbcc6d7f4
85967c269da77aa062cdbc1b
...finished in 0:07:31.579735

--- Parsing RPKI ---

Parsing 317712 ROAs
Result entries written: 634310
Duplicates found: 92307
Invalids found: 2295
Incompletes: 0
Non-ROA files: 0
...finished in 0:00:36.564541

--- Parsing IRR ---

Extracting afrinic.db.gz
Extracting apnic.db.route.gz
Extracting apnic.db.route6.gz
Extracting arin.db.gz
Extracting lacnic.db.gz
Extracting ripe.db.route.gz
Extracting ripe.db.route6.gz
Parsing /home/emzy/kartograf/out/1751385600/irr/lacnic.db
Found in this file: 15280
Parsing /home/emzy/kartograf/out/1751385600/irr/ripe.db.route
Found in this file: 386802
Parsing /home/emzy/kartograf/out/1751385600/irr/apnic.db.route
Found in this file: 611242
Parsing /home/emzy/kartograf/out/1751385600/irr/afrinic.db
Found in this file: 116768
Parsing /home/emzy/kartograf/out/1751385600/irr/arin.db
Found in this file: 107461
Parsing /home/emzy/kartograf/out/1751385600/irr/apnic.db.route6
Found in this file: 632639
Parsing /home/emzy/kartograf/out/1751385600/irr/ripe.db.route6
Found in this file: 149461
Found valid, unique entries: 2019653
...finished in 0:02:50.426917

--- Merging RPKI and IRR data ---

Parse base file to dictionary
Parse extra file to Pandas DataFrame
Merging extra prefixes that were not included in the base file.
Finished merging extra prefixes.
Finished filtering! Originally 2019653 entries filtered down to 428825
Merging base file with filtered extra file
...finished in 0:11:44.322757

--- Parsing Routeviews pfx2as ---

Unzipping /home/emzy/kartograf/data/1751385600/collectors/routeviews_pfx2asn_ip4.txt.gz
Formatting /home/emzy/kartograf/out/1751385600/collectors/routeviews_pfx2asn_ip4.txt
Unzipping /home/emzy/kartograf/data/1751385600/collectors/routeviews_pfx2asn_ip6.txt.gz
Formatting /home/emzy/kartograf/out/1751385600/collectors/routeviews_pfx2asn_ip6.txt
Cleaning /home/emzy/kartograf/out/1751385600/collectors/pfx2asn.txt
Entries after cleanup: 1273630
...finished in 0:00:51.870375

--- Merging Routeviews and base data ---

Parse base file to dictionary
Parse extra file to Pandas DataFrame
Merging extra prefixes that were not included in the base file.
Finished merging extra prefixes.
Finished filtering! Originally 1273630 entries filtered down to 335362
Merging base file with filtered extra file
...finished in 0:09:58.907538

--- Sorting results ---

...finished in 0:00:18.968162

--- Finishing Kartograf ---

The SHA-256 hash of the result file is: 620203796293f1e4f84336cc23b723ebaeefe198f6e4bf27f535370f0184d185
Total runtime: 0:41:15.144628

Same as @Sjors

[fjahr]

Phew, a bit rough so far (pending @0xB10C and maybe other surprise participants?) the good news is that @jurraca and me have been working on a few features that should make the process more reliable. Some of these didn't have the desired effect in test runs but just recently asmap/kartograf#82 looks very promising, so even if this one is a failure I am very hopeful that we will have a better outcome with the next try after we have released a new version :)

[jurraca]

@fjahr and @Emzy could you send me your out/1751385600/rpki/rpki_final.txt out/1751385600/rpki/rpki_raw.json file? here or email or whatever. thanks.

[Emzy]

@fjahr and @Emzy could you send me your out/1751385600/rpki/rpki_final.txt file? here or email or whatever. thanks.

out-1751385600-rpki-rpki_final.txt

[luisschwab]

Sorry, missed this one.

[jurraca]

@fjahr and @Emzy could you send me your out/1751385600/rpki/rpki_final.txt file? here or email or whatever. thanks.

out-1751385600-rpki-rpki_final.txt

sorry @Emzy i need your rpki-raw.json actually, so i can see the sources. 🙏

[Emzy]

@fjahr and @Emzy could you send me your out/1751385600/rpki/rpki_final.txt file? here or email or whatever. thanks.

out-1751385600-rpki-rpki_final.txt

sorry @Emzy i need your rpki-raw.json actually, so i can see the sources. 🙏

here the file:
out-1751385600-rpki-rpki_raw.json.bz2
(I will delete it later)

[jurraca]

Between Sjors/Emzy 's RPKI data and mine, they have a superset of my data, i.e. 69 more networks that aren't present in mine. Of these, 33 were issued by sia rsync.paas.rpki.ripe.net, 31 by sakuya.nat.moe, and another 5 by rpki.ripe.net.

I was 403'd on rpki-rsync.warpnet.xyz via rrdp:

114:rpki-client: Error retrieving https://rpki-rrdp.warpnet.xyz/notification.xml (135.125.201.197): 403
115:rpki-client: https://rpki-rrdp.warpnet.xyz/notification.xml: load from network failed, fallback to rsync
138:rpki-client: rsync rsync://rpki-rsync.warpnet.xyz/repo failed
139:rpki-client: .rsync/rpki-rsync.warpnet.xyz/repo: load from network failed, fallback to cache

and missing a bunch of networks from cloudie-repo.rpki.app and entirely missing rpki.apernet.io. I suspect others have similar small differences.

I think we're going to have to approach syncing with RPKI repos differently.