Kademlia provider record spillover

[gmelodie]

Summary

Adds kad-dht/provider-record-spillover.md, a working-draft specification for an opt-in extension to the Kademlia DHT ADD_PROVIDER flow that addresses provider record hotspots on popular keys.

Problem

Under the base kad-dht protocol, the k closest peers to a key unconditionally accumulate every ADD_PROVIDER record for it.

For popular keys, this creates an unbounded load concentration with no mechanism for a peer to decline new provider records or for advertisers to spill over to other peers. This has been a known scalability issue, including:

• libp2p/go-libp2p-kad-dht#316
• ipfs/kubo#5613

This proposal implements the “rejection + sloppy hashing / spillover” approach originally proposed in libp2p/specs#163, drawing from the Coral DHT paper:

Freedman & Mazières, 2003

What This Spec Defines

Server-side — provider record limits

A node MAY enforce maxProvidersPerKey.

Once at capacity, it MUST reject ADD_PROVIDER requests from new providers for that key, while always accepting re-advertisements from existing providers so records can continue to be refreshed.

Wire protocol

A new optional providerStatus field (field 11) is added to the ADD_PROVIDER response message with two possible values:

• accepted (0)
• rejected (1)

For backward compatibility:

• An absent providerStatus field MUST be treated as accepted.
• providerStatus is response-only.
• Advertisers MUST NOT set it in requests.
• Receivers MUST ignore it if present in a request.

Client-side — spillover algorithm

The advertiser performs the normal iterative FIND_NODE lookup, then processes the sorted candidate list in chunks of size α, from closest to farthest.

After each chunk, it counts peers that:

• accepted the provider record, or
• did not respond

toward the replication target k.

If k has not yet been reached, the advertiser continues to the next chunk (a spillover round).

The process stops when either:

• k successful placements are reached, or
• all candidates are exhausted.

Non-responding peers count as accepted to preserve compatibility with nodes that predate this extension.

Compatibility

This extension is fully backward compatible.

Nodes that do not implement the extension:

• never write providerStatus
• silently ignore providerStatus if received

No changes are made to:

• GET_PROVIDERS
• FIND_NODE
• any other message type

Related

• DHT provider records: adopt sloppiness, rejection and backtracking from Coral #163
• Limit the size of provider sets go-libp2p-kad-dht#316

[guillaumemichel]

Hi @gmelodie, thank you for submitting this PR! I like the idea of provider record spillover and think it could indeed be very useful for the ecosystem.

However, since this spec is about Kad-DHT provider records and provider records are mostly used in IPFS-land, I'd recommend opening the PR against ipfs/specs instead. The canonical IPFS DHT spec has moved there and is now published at specs.ipfs.tech.

[gmelodie]

Closing in favor of ipfs/specs#537