py-libp2p Multihash Integration Analysis

[acul71]

py-libp2p Multihash Integration Analysis

Overview

This document analyzes where py-multihash's new features (version 3.0.0+) can be leveraged to improve py-libp2p's codebase. The analysis covers current multihash usage patterns and identifies specific opportunities for enhancement.

Current py-multihash Features (v3.0.0+)

Key New Features

1. Streaming Hash Computation (sum_stream()) - Memory-efficient hashing for large files
2. Hash Truncation Support - Truncate digests to specific lengths
3. SHAKE Variable-Length Hashes - SHAKE-128 and SHAKE-256 support
4. JSON Serialization - to_json() and from_json() methods
5. MultihashSet Collection - Specialized set for managing multihash collections
6. Verify Method - Built-in verify() method on Multihash objects
7. Stream Read/Write - read() and write() methods for binary streams
8. Modern Hash Functions - BLAKE3, MurmurHash3, BLAKE2 variants (125+ hash functions)
9. Go-Compatible API - sum() function for Go migration compatibility

Current py-libp2p Multihash Usage

1. Peer ID Generation (libp2p/peer/id.py)

Current Implementation:

• Uses multihash.digest() for peer ID creation from public keys
• Uses multihash.decode() for extracting public keys from identity multihashes
• Manually handles identity hash registration via FuncReg.register()

Code Location: Lines 85-118

Status: ✅ Already using modern API appropriately

Details:

# Line 88-91: Peer ID creation
algo = multihash.Func.sha2_256
if ENABLE_INLINING and len(serialized_key) <= MAX_INLINE_KEY_LENGTH:
    algo = IDENTITY_MULTIHASH_CODE
mh_digest = multihash.digest(serialized_key, algo)
return cls(mh_digest.encode())

# Line 108: Public key extraction
mh_decoded = multihash.decode(self._bytes)

2. Kademlia DHT (libp2p/kad_dht/)

2.1 Routing Table (libp2p/kad_dht/routing_table.py)

Current Implementation:

• Uses multihash.digest() for peer ID to key conversion (line 49)
• Converts peer IDs to 256-bit keys for routing table operations

Code Location: Line 41-49

Current Code:

def peer_id_to_key(peer_id: ID) -> bytes:
    """
    Convert a peer ID to a 256-bit key for routing table operations.
    This normalizes all peer IDs to exactly 256 bits by hashing them with SHA-256.
    """
    return multihash.digest(peer_id.to_bytes(), "sha2-256").digest

Improvement Opportunities:

• Could use hash truncation for shorter keys in specific scenarios (though 256-bit is standard)
• Could use verify() method if peer ID verification is needed
• Status: ✅ Current implementation is appropriate

2.2 DHT Utils (libp2p/kad_dht/utils.py)

Current Implementation:

• Uses multihash.digest() for key generation from binary data (line 104)
• Uses multihash.digest() for peer ID hashing in distance calculations (line 160)

Code Locations: Lines 93-104, 145-163

Current Code:

def create_key_from_binary(binary_data: bytes) -> bytes:
    """Creates a key for the DHT by hashing binary data with SHA-256."""
    return multihash.digest(binary_data, "sha2-256").digest

def sort_peer_ids_by_distance(target_key: bytes, peer_ids: list[ID]) -> list[ID]:
    def get_distance(peer_id: ID) -> int:
        peer_hash = multihash.digest(peer_id.to_bytes(), "sha2-256").digest
        return xor_distance(target_key, peer_hash)
    return sorted(peer_ids, key=get_distance)

Improvement Opportunities:

• Could use sum() function (Go-compatible API) for consistency
• Status: ✅ Current implementation is appropriate

3. Bitswap CID Module (libp2p/bitswap/cid.py)

Current Implementation:

• Manually constructs multihash format: bytes([HASH_SHA256, len(digest)]) + digest
• Uses hashlib.sha256() directly instead of multihash API
• Manual CID verification logic with byte manipulation

Code Locations: Lines 24-45, 48-69, 127-182

Current Issues:

• Manual multihash construction violates DRY principle
• Error-prone byte manipulation
• Doesn't leverage multihash library features
• Manual verification logic is complex and error-prone

Current Code Examples:

# Line 38-43: Manual multihash construction
digest = hashlib.sha256(data).digest()
multihash = bytes([HASH_SHA256, len(digest)]) + digest

# Line 152-160: Manual verification
if len(cid) >= 2 and cid[0] == HASH_SHA256:
    hash_length = cid[1]
    if len(cid) >= 2 + hash_length:
        cid_digest = cid[2 : 2 + hash_length]
        match = digest == cid_digest

Improvement Opportunities:

HIGH PRIORITY: Replace manual multihash construction with multihash.encode() or multihash.digest().encode()

HIGH PRIORITY: Use multihash.decode() and verify() method for CID verification

Proposed Changes:

import multihash

# Instead of manual construction:
# digest = hashlib.sha256(data).digest()
# multihash = bytes([HASH_SHA256, len(digest)]) + digest

# Use:
mh = multihash.digest(data, multihash.Func.sha2_256)
multihash_bytes = mh.encode()

# For verification, instead of manual parsing:
# if len(cid) >= 2 and cid[0] == HASH_SHA256:
#     hash_length = cid[1]
#     cid_digest = cid[2 : 2 + hash_length]
#     match = digest == cid_digest

# Use:
try:
    mh = multihash.decode(cid)
    match = mh.verify(data)
except ValueError:
    match = False

Benefits:

• Eliminates error-prone manual byte manipulation
• Uses tested library code
• Cleaner, more maintainable code
• Better error handling

4. Bitswap DAG (libp2p/bitswap/dag.py)

Current Implementation:

• Processes files in chunks for memory efficiency (line 195)
• Computes CID for each chunk using compute_cid_v1() (line 197)
• Uses manual hashing approach through compute_cid_v1()

Code Location: Lines 104-244

Current Code:

# Process file in chunks (memory efficient)
for i, chunk_data in enumerate(chunk_file(file_path, chunk_size)):
    # Compute CID for chunk
    chunk_cid = compute_cid_v1(chunk_data, codec=CODEC_RAW)
    # Store chunk
    await self.bitswap.add_block(chunk_cid, chunk_data)

Improvement Opportunities:

HIGH PRIORITY: Use sum_stream() for streaming hash computation of large files

Proposed Changes:

from multihash import sum_stream
from io import open

# For computing hash of entire file or large chunks:
with open(file_path, "rb") as f:
    mh = sum_stream(f, multihash.Func.sha2_256)
    # Convert multihash to CID format
    root_cid = compute_cid_v1_from_multihash(mh)

Benefits:

• Memory efficient for large files (processes in chunks automatically)
• Cleaner code
• Better performance for large file operations
• Leverages optimized streaming implementation

Note: The current chunking approach is good, but the hash computation per chunk could also benefit from streaming if chunks are large.

5. SECIO Transport (libp2p/security/secio/transport.py)

Current Implementation:

• Uses multihash.digest() for creating SHA-256 multihashes (line 211)
• Wrapper function _mk_multihash_sha256() creates multihash (line 210-212)

Code Location: Lines 210-212

Current Code:

def _mk_multihash_sha256(data: bytes) -> bytes:
    mh = multihash.digest(data, "sha2-256")
    return mh.encode()

Status: ✅ Already using multihash API appropriately

Note: Could potentially use sum() function for Go-compatibility, but current approach is fine.

6. Records Validation (libp2p/records/pubkey.py)

Current Implementation:

• Uses multihash.decode() for validating multihash format (line 39)
• Manual validation logic with exception handling

Code Location: Lines 18-54

Current Code:

keyhash = bytes.fromhex(key)
try:
    _ = multihash.decode(keyhash)
except Exception:
    raise InvalidRecordType("key did not contain valid multihash")

Improvement Opportunities:

• Could use multihash.is_valid() for validation checks (more efficient, no exception overhead)
• Could use verify() method if data verification is needed

Proposed Changes:

# Instead of:
try:
    _ = multihash.decode(keyhash)
except Exception:
    raise InvalidRecordType("key did not contain valid multihash")

# Use:
if not multihash.is_valid(keyhash):
    raise InvalidRecordType("key did not contain valid multihash")

Benefits:

• More efficient (no exception overhead for invalid input)
• Cleaner code
• Better performance for validation-heavy code paths

Specific Improvement Recommendations

Priority 1: Bitswap CID Module Refactoring

File: libp2p/bitswap/cid.py

Impact: High - Affects core Bitswap functionality

Current Issues:

• Manual multihash construction violates DRY principle
• Error-prone byte manipulation
• Doesn't leverage multihash library features
• Complex manual verification logic

Proposed Refactoring:

1. Replace compute_cid_v0() function:

def compute_cid_v0(data: bytes) -> bytes:
    """Compute a CIDv0 for data using multihash API."""
    mh = multihash.digest(data, multihash.Func.sha2_256)
    return mh.encode()

2. Replace compute_cid_v1() function:

def compute_cid_v1(data: bytes, codec: int = CODEC_RAW) -> bytes:
    """Compute a CIDv1 for data using multihash API."""
    mh = multihash.digest(data, multihash.Func.sha2_256)
    multihash_bytes = mh.encode()
    
    # CIDv1 format: <version><codec><multihash>
    return bytes([CID_V1, codec]) + multihash_bytes

3. Refactor verify_cid() function:

def verify_cid(cid: bytes, data: bytes) -> bool:
    """Verify that data matches the given CID using multihash API."""
    try:
        # Extract multihash from CID
        if len(cid) >= 4 and cid[0] == CID_V1:
            # CIDv1: <version><codec><multihash>
            multihash_bytes = cid[2:]
        elif len(cid) >= 2 and cid[0] == HASH_SHA256:
            # CIDv0: just multihash
            multihash_bytes = cid
        else:
            return False
        
        # Decode and verify
        mh = multihash.decode(multihash_bytes)
        return mh.verify(data)
    except (ValueError, IndexError):
        return False

Testing Considerations:

• Ensure backward compatibility with existing CIDs
• Test with various CID formats (v0 and v1)
• Verify performance is not degraded

Priority 2: Streaming Hash for Large Files

File: libp2p/bitswap/dag.py

Impact: High - Improves memory efficiency for large file operations

Current Implementation:

• Processes files in chunks for memory efficiency
• Computes hash per chunk manually
• Could benefit from streaming API for root hash computation

Proposed Changes:

1. For root hash computation of entire file:

from multihash import sum_stream

async def compute_file_hash(file_path: str) -> multihash.Multihash:
    """Compute multihash of entire file using streaming."""
    with open(file_path, "rb") as f:
        return sum_stream(f, multihash.Func.sha2_256)

2. For large chunk hashing:

from io import BytesIO
from multihash import sum_stream

# If chunks are large, use streaming
chunk_stream = BytesIO(chunk_data)
mh = sum_stream(chunk_stream, multihash.Func.sha2_256)
chunk_cid = compute_cid_v1_from_multihash(mh)

Benefits:

• Memory efficient for large files
• Cleaner code
• Better performance
• Automatic chunking optimization

Priority 3: MultihashSet for Collections

Potential Use Cases:

1. DHT Provider Store - Track multiple content CIDs
2. Bitswap Block Tracking - Manage collections of block CIDs
3. Peer ID Collections - Track groups of peer IDs

Example Implementation:

from multihash import MultihashSet, sum, Func

class ContentTracker:
    """Track multiple content hashes efficiently."""
    
    def __init__(self):
        self.content_hashes = MultihashSet()
    
    def add_content(self, data: bytes) -> multihash.Multihash:
        """Add content and return its multihash."""
        mh = sum(data, Func.sha2_256)
        self.content_hashes.add(mh)
        return mh
    
    def has_content(self, mh: multihash.Multihash) -> bool:
        """Check if content exists."""
        return mh in self.content_hashes
    
    def get_all_hashes(self) -> list[multihash.Multihash]:
        """Get all tracked content hashes."""
        return self.content_hashes.All()

Benefits:

• Type-safe multihash collections
• Efficient membership testing
• Set operations (union, intersection, difference)
• Both Go-style and Python-style APIs

Priority 4: JSON Serialization

Potential Use Cases:

1. API Responses - Return CID/metadata as JSON
2. Configuration Storage - Store CIDs in config files
3. Debugging and Logging - Serialize multihashes for logs
4. REST APIs - Expose multihash data via HTTP APIs

Example Implementation:

from multihash import digest, from_json, Func

# Serialize CID for API
def get_cid_json(data: bytes) -> str:
    """Get CID as JSON for API response."""
    mh = digest(data, Func.sha2_256)
    return mh.to_json(verbose=True)  # Includes name field

# Deserialize from API
def parse_cid_json(json_str: str) -> multihash.Multihash:
    """Parse CID from JSON API input."""
    return from_json(json_str)

# Usage in API endpoint
@app.route("/api/content/<cid_json>")
def get_content(cid_json: str):
    mh = from_json(cid_json)
    # Use mh to retrieve content
    pass

Benefits:

• Easy integration with web APIs
• Human-readable format for debugging
• Standard JSON format
• Supports both compact and verbose modes

Priority 5: Modern Hash Functions

Potential Use Cases:

1. Performance-Critical Hashing - Use BLAKE3 for faster hashing
2. Non-Cryptographic Hashing - Use MurmurHash3 for DHT keys (if compatible)
3. Configurable Algorithms - Allow users to choose hash functions

Important Considerations:

• Compatibility: Must ensure other libp2p implementations support chosen algorithms
• Security: Cryptographic hashes required for peer IDs and content addressing
• Performance: BLAKE3 is faster but must verify compatibility

Example (with compatibility checks):

from multihash import digest, Func

# For performance-critical content hashing (if compatible)
def fast_content_hash(data: bytes) -> multihash.Multihash:
    """
    Use BLAKE3 for faster hashing.
    WARNING: Only use if other libp2p implementations support it.
    """
    # Check compatibility first
    if not is_blake3_supported():
        return digest(data, Func.sha2_256)  # Fallback to SHA-256
    return digest(data, Func.blake3)

# For non-cryptographic use cases (DHT internal operations)
def dht_internal_hash(data: bytes) -> multihash.Multihash:
    """
    Use MurmurHash3 for fast, non-cryptographic hashing.
    Only for internal DHT operations, not for peer IDs or content.
    """
    return digest(data, Func.murmur3_128)

Note: Any new hash algorithms must be carefully evaluated for:

• Compatibility with other libp2p implementations
• Security requirements of the use case
• Performance benefits vs. compatibility trade-offs

Priority 6: Stream Read/Write

Potential Use Cases:

1. Protocol Serialization - Write multihashes to network streams
2. Binary Protocol Buffers - Efficient multihash I/O
3. File Storage - Store multihashes in binary format

Example Implementation:

from multihash import Multihash
from io import BytesIO

# Writing multihash to stream
def serialize_multihash(mh: multihash.Multihash, stream: BinaryIO) -> int:
    """Write multihash to binary stream."""
    return mh.write(stream)

# Reading multihash from stream
def deserialize_multihash(stream: BinaryIO) -> multihash.Multihash:
    """Read multihash from binary stream."""
    return Multihash.read(stream)

# Usage in protocol handler
async def send_cid(stream: INetStream, cid: bytes) -> None:
    """Send CID as multihash over network stream."""
    mh = multihash.decode(cid)
    mh.write(stream)  # Efficient binary serialization

async def receive_cid(stream: INetStream) -> bytes:
    """Receive CID as multihash from network stream."""
    mh = Multihash.read(stream)
    return mh.encode()

Benefits:

• Efficient binary I/O
• Standardized format
• Easy integration with protocol buffers
• Type-safe serialization

Implementation Checklist

High Priority

• Refactor libp2p/bitswap/cid.py to use multihash API

  • Replace compute_cid_v0() with multihash API
  • Replace compute_cid_v1() with multihash API
  • Refactor verify_cid() to use multihash.decode() and verify()
  • Remove manual multihash construction
  • Add comprehensive tests

• Implement streaming hash in libp2p/bitswap/dag.py for large files

  • Use sum_stream() for root hash computation
  • Consider streaming for large chunks
  • Add performance tests

• Update libp2p/records/pubkey.py to use multihash.is_valid()

  • Replace exception-based validation with is_valid()
  • Improve error messages
  • Add tests

Medium Priority

• Consider MultihashSet for hash collections

  • Evaluate use cases in DHT provider store
  • Evaluate use cases in Bitswap block tracking
  • Implement if beneficial

• Add JSON serialization support where appropriate

  • API endpoints for CID metadata
  • Configuration storage
  • Debugging/logging utilities

• Standardize on sum() function for Go-compatibility

  • Replace digest() calls with sum() where appropriate
  • Ensure consistency across codebase

Low Priority

• Evaluate modern hash functions (BLAKE3, MurmurHash3) for specific use cases

  • Research compatibility with other libp2p implementations
  • Benchmark performance improvements
  • Implement only if compatible and beneficial

• Use stream read/write methods for protocol serialization

  • Evaluate protocol buffer integration
  • Implement if performance benefits are significant

• Add hash truncation support where beneficial

  • Evaluate use cases for shorter keys
  • Implement if compatible with libp2p specifications

Compatibility Considerations

1. Hash Algorithm Compatibility

Critical Requirement: Any hash algorithms used must be supported by other libp2p implementations (Go, Rust, JavaScript, etc.).

Current Standard Algorithms:

• SHA-256 (0x12) - ✅ Universally supported
• SHA-512 (0x13) - ✅ Universally supported
• Identity (0x00) - ✅ Universally supported

New Algorithms to Evaluate:

• BLAKE3 (0x1E) - ⚠️ Check compatibility
• MurmurHash3 (0x22, 0x23) - ⚠️ Check compatibility
• BLAKE2 variants - ⚠️ Check compatibility

Recommendation: Stick with SHA-256 for peer IDs and content addressing unless compatibility is verified.

2. Backward Compatibility

Critical Requirement: All changes must maintain compatibility with:

• Existing peer IDs
• Existing CIDs
• Existing protocol messages
• Other libp2p implementations

Testing Strategy:

• Test with existing peer IDs and CIDs
• Verify interoperability with other libp2p implementations
• Ensure no breaking changes to public APIs

3. Performance

Considerations:

• New implementations should not degrade performance
• Streaming APIs should improve performance for large files
• Validation improvements should reduce overhead

Benchmarking:

• Compare performance before/after changes
• Profile memory usage for large files
• Measure validation performance improvements

4. Testing

Requirements:

• All changes must include comprehensive tests
• Test edge cases (empty data, invalid formats, etc.)
• Test compatibility with existing data
• Test performance characteristics

Code Examples

Example 1: Refactored CID Module

"""
Refactored CID module using py-multihash API.
"""
import multihash

# CID version constants
CID_V0 = 0
CID_V1 = 1

# Multicodec constants
CODEC_DAG_PB = 0x70
CODEC_RAW = 0x55


def compute_cid_v0(data: bytes) -> bytes:
    """
    Compute a CIDv0 for data using multihash API.
    CIDv0 is just a base58-encoded multihash (SHA-256).
    """
    mh = multihash.digest(data, multihash.Func.sha2_256)
    return mh.encode()


def compute_cid_v1(data: bytes, codec: int = CODEC_RAW) -> bytes:
    """
    Compute a CIDv1 for data using multihash API.
    CIDv1 format: <version><codec><multihash>
    """
    mh = multihash.digest(data, multihash.Func.sha2_256)
    multihash_bytes = mh.encode()
    
    # CIDv1 format: <version><codec><multihash>
    return bytes([CID_V1, codec]) + multihash_bytes


def verify_cid(cid: bytes, data: bytes) -> bool:
    """
    Verify that data matches the given CID using multihash API.
    """
    try:
        # Extract multihash from CID
        if len(cid) >= 4 and cid[0] == CID_V1:
            # CIDv1: <version><codec><multihash>
            multihash_bytes = cid[2:]
        elif len(cid) >= 2:
            # CIDv0: just multihash (starts with hash code)
            multihash_bytes = cid
        else:
            return False
        
        # Decode and verify using multihash API
        mh = multihash.decode(multihash_bytes)
        return mh.verify(data)
    except (ValueError, IndexError):
        return False

Example 2: Streaming Hash for Large Files

"""
Streaming hash computation for large files.
"""
from multihash import sum_stream, Func
from libp2p.bitswap.cid import compute_cid_v1


async def compute_file_cid(file_path: str) -> bytes:
    """
    Compute CID for entire file using streaming hash.
    Memory efficient for large files.
    """
    with open(file_path, "rb") as f:
        mh = sum_stream(f, Func.sha2_256)
        # Convert multihash to CIDv1 format
        multihash_bytes = mh.encode()
        return bytes([CID_V1, CODEC_RAW]) + multihash_bytes

Example 3: Improved Records Validation

"""
Improved records validation using multihash.is_valid().
"""
import multihash
from libp2p.records.utils import InvalidRecordType, split_key


class PublicKeyValidator(Validator):
    """Validator for public key records using multihash API."""
    
    def validate(self, key: str, value: bytes) -> None:
        """Validate a public key record."""
        ns, key = split_key(key)
        if ns != "pk":
            raise InvalidRecordType("namespace not 'pk'")
        
        keyhash = bytes.fromhex(key)
        
        # Use is_valid() instead of exception-based validation
        if not multihash.is_valid(keyhash):
            raise InvalidRecordType("key did not contain valid multihash")
        
        # Rest of validation logic...

Conclusion

py-multihash v3.0.0+ provides significant opportunities to improve py-libp2p's code quality, maintainability, and performance. The highest impact improvements are:

1. Bitswap CID Module Refactoring - Eliminates error-prone manual byte manipulation
2. Streaming Hash for Large Files - Improves memory efficiency and performance
3. Improved Validation - More efficient and cleaner code

Careful implementation with proper testing will ensure compatibility and reliability. The recommended approach is to:

1. Start with high-priority improvements (CID module, streaming)
2. Test thoroughly for compatibility
3. Gradually adopt medium and low-priority features
4. Always verify compatibility with other libp2p implementations

By leveraging py-multihash's modern features, py-libp2p can achieve better code quality, improved performance, and easier maintenance while maintaining full compatibility with the libp2p ecosystem.

[yashksaini-coder]

Compatibility Considerations

1. Hash Algorithm Compatibility

Critical Requirement: Any hash algorithms used must be supported by other libp2p implementations (Go, Rust, JavaScript, etc.).

Current Standard Algorithms:

• SHA-256 (0x12) - ✅ Universally supported
• SHA-512 (0x13) - ✅ Universally supported
• Identity (0x00) - ✅ Universally supported

New Algorithms to Evaluate:

• BLAKE3 (0x1E) - ⚠️ Check compatibility
• MurmurHash3 (0x22, 0x23) - ⚠️ Check compatibility
• BLAKE2 variants - ⚠️ Check compatibility

Recommendation: Stick with SHA-256 for peer IDs and content addressing unless compatibility is verified.

@gerceboss Okh I see it now here, then proceed with SHA-256 hash algorithm,

[acul71]

Summary of current status for anyone picking up work from this discussion:

• Priority 3 (records): Done and in main — libp2p/records/pubkey.py uses multihash.is_valid() with fallback (from PR Addresses #1180 Integrate py-multihash v3 API in Bitswap CID module  #1186).
• Priority 1 (Bitswap CID) and Priority 2 (streaming): Were implemented in PR Addresses #1180 Integrate py-multihash v3 API in Bitswap CID module  #1186, but the CID module was later refactored in PR add multicodec #1194 (multicodec integration). That refactor replaced the py-multihash-based CID code with multicodec + hashlib, so the Priority 1/2 changes in libp2p/bitswap/cid.py are no longer present.
• Going forward: The project has standardised Bitswap CID on py-cid — PR feat: Bitswap py-cid Migration Phase 1 + Phase 2 #1235 (Phase 1+2) is merged, issue Bitswap py-cid Migration - Phase 1 + Phase 2 #1234 is closed. We are not planning to re-apply the py-multihash refactor in cid.py; the py-cid migration is the intended path.

If you want to contribute around multihash, please avoid opening a new PR that rewrites libp2p/bitswap/cid.py. Using py-multihash in other areas (e.g. DHT, SECIO) and continuing to rely on is_valid() in records is still aligned with this discussion.