Add ability to include arbitrary BGP dump file

[jurraca]

It would be interesting to let users bring their own BGP dump, particularly if they run their own AS or have easy access to such data. This would allow a perhaps closer level of precision or insight compared to third-party databases such as IRR or Routeviews. See discussion in bitcoin-core/asmap-data#17

This would entail a flag to pass the data file on the CLI.
When merging with other data sources, perhaps it should supersede IRR and Routeviews but not RPKI ?

The format would be identical to existing network-ASN pairs, i.e. one pair by line, space-separated:

1.0.0.0/24 13335
1.0.4.0/22 38803
...

[jorisstrakeljahn]

I've been reading through the merge pipeline and the discussion in asmap-data#17, and here's a proposal for how this could work.

Where custom data fits in the pipeline

The merge chain in kartograf.py determines priority implicitly by merge order. general_merge() (merge.py:171) keeps all base file entries and only adds extra entries whose prefix isn't already covered by the base (via BaseNetworkIndex.contains_row() at merge.py:58):

RPKI (rpki_final.txt)
  → merge_irr()      [merge.py:81]  — base=RPKI, extra=IRR
  → merge_pfx2as()   [merge.py:98]  — base=RPKI(+IRR), extra=Routeviews
  → sort_result_by_pfx()

Since the merge is additive (extra entries can only fill gaps, never override the base), inserting custom data right after RPKI gives us exactly the priority from the issue. RPKI wins over custom, custom wins over IRR/Routeviews:

RPKI (rpki_final.txt)
  → merge_custom()   — base=RPKI, extra=Custom  <- NEW
  → merge_irr()      — base=RPKI+Custom, extra=IRR
  → merge_pfx2as()   — base=RPKI+Custom(+IRR), extra=Routeviews
  → sort_result_by_pfx()

Without --bgp-dump, the pipeline is identical. Purely additive feature.

A sort.py concern

sort_result_by_pfx() at sort.py:8-16 determines its input file via an if/elif chain over args.irr and args.routeviews (4 combinations). A third boolean would make this 8 combinations.

Each merge step already copies its output to context.final_result_file (merge.py:94, merge.py:118). If sort.py read from final_result_file instead of reconstructing filenames (plus writing rpki_final.txt there in the RPKI-only case), the if/elif chain becomes unnecessary and future source additions don't require touching sort.py.

Implementation outline

File	Change
cli.py	Add --bgp-dump argument to parser_map, similar to the existing -irr / -rv flags
New: custom/parse.py	Read the custom file line by line, validate prefix ASN format, filter via is_bogon_pfx() / is_bogon_asn() / is_out_of_encoding_range() from bogon.py, write cleaned output. Similar to how collectors/parse.py handles Routeviews data
merge.py	Add merge_custom() analogous to merge_irr(): call general_merge() with rpki_final.txt as base and the cleaned custom file as extra
kartograf.py	New section between RPKI parsing and IRR that calls the parse and merge functions above
merge.py	Both merge_irr() and merge_pfx2as() currently hardcode their base file paths. Both need to pick the custom-merged file when --bgp-dump is set
sort.py	Read from context.final_result_file instead of the if/elif filename reconstruction
context.py	Store the custom file path and copy it into data/<epoch>/custom/ for reproducibility

general_merge() and BaseNetworkIndex stay untouched.

Open questions

Reproducibility: Should the custom file be copied into data/<epoch>/ so --reproduce works? Currently Context.__init__() reconstructs source flags from the subdirectories present in the data path. A custom/ folder would need the same treatment.

Collaborative launches: --bgp-dump with --wait would produce a unique hash by design. Should these be mutually exclusive, or is a warning sufficient?

Does this approach make sense, or would you go in a different direction?

[fjahr]

Thanks for looking into this @jorisstrakeljahn ! I am giving my thoughts below, let me know if I missed something were you would still like to have input.

Where custom data fits in the pipeline

Yeah, that roughly matches what I had in mind. I think for the start it is reasonable to assume that the person using the BGP dump is still interested in at least RPKI data since it is signed. If they only wanted to use their dump they wouldn't need Kartograf and could just encode a map from their bump directly. I am also assuming that the user has a long running BGP session and a complete routing table, if the user would like to rather include some limited view into the map, I think that particular idea would need further discussion, especially if it should take preference over RPKI. The most likely case I am assuming is RPKI + dump as the only inputs but backfilling with the other source should be possible.

A sort.py concern

Yeah, these hardcoded sources in different places should go away as we go more flexible and allow for more sources. Your plan sounds good, feel free to suggest additional refactoring to further generalize this. The intermediate verbose/duplicate file names were mainly making it easy to debug in the early days but we haven't needed them in a while.

Add --bgp-dump argument to parser_map

Naming nit: Maybe use something less generic since routeview basically is also based on bgp dumps. Maybe custom_source or so, I would like the naming of the module internally to be similar to the option name, if possible.

Reproducibility: Should the custom file be copied into data// so --reproduce works?

That's a good question that I haven't thought about. For convenience I would say yes.

Collaborative launches: --bgp-dump with --wait would produce a unique hash by design. Should these be mutually exclusive, or is a warning sufficient?

You mean just because everyone would use their own custom dump, right? I would say rather that this just needs to be documented reasonably well but we can assume our users understand that they can't participate in a collaborative run if they bring their own data. We make it clear in the run instructions what flags they need to set each time. But I do think it could be possible that there may be some collaborative runs in the future that take some custom data as additional input which is distributed to all of the participants before the start. At least some people have found this idea appealing. Because of that I don't think it makes sense to make them mutually exclusive or show a warning.

[jorisstrakeljahn]

Thanks @fjahr !

Agreed on --custom-source, will use that for the CLI flag and internal module.

if the user would like to rather include some limited view into the map, I think that particular idea would need further discussion, especially if it should take preference over RPKI

I think the additive merge actually covers this well already: even with partial data, slotting below RPKI means custom only fills gaps that RPKI doesn't cover, so the user gets their local view exactly where it matters, without touching anything RPKI already attests. The only case where the relationship would differ is if custom contradicts a ROA for the same prefix, but since ROAs are cryptographically signed by the prefix holder, that almost always means the custom data is wrong. If overriding RPKI ever becomes a real need, that feels like a different trust model worth its own discussion.

I'll start on a PR with this approach + the sort.py cleanup

[jorisstrakeljahn]

Opened a PR for this: #133

While implementing this I also refactored the merge pipeline so all merge steps read from and write back to final_result_file. The alternative would have been extending the if/elif chains in merge_pfx2as and sort.py for each new source, which doesn't scale well. The refactor is kept in its own commit so it can easily be split out.