Building Embedded IPFS for Python Services

[asabya]

The Vision

Python is everywhere — APIs, data pipelines, ML services, IoT — but when these services need content-addressed data exchange, they typically shell out to a Go IPFS daemon or call an HTTP API. What if any Python process could natively speak IPFS?

py-ipfs-lite is a lightweight, embeddable IPFS peer built entirely on py-libp2p. It's a Python port of ipfs-lite (Go) that gives any Python service the ability to add files, resolve CIDs, and exchange blocks with Go IPFS peers — all in-process, no daemon required.

This post covers what was built, how Go interop was achieved, and the py-libp2p improvements that were made along the way. It demonstrates that py-libp2p is ready to be a foundation for real protocol implementations — and collaboration on getting it there would be welcome.

What works today:

• CIDv1/base32/sha2-256 block creation and storage
• Bitswap 1.1.0/1.2.0 protocol with two-phase want-have → want-block negotiation
• In-memory and filesystem blockstores
• UnixFS file add/get with chunking
• Block announcement — peers that want a block receive it automatically when it's added
• DHT content routing — provide() on block add, find_providers() fallback in get_block
• Full interop with Go ipfs-lite peers

The Road to Go Interop

Getting Python and Go ipfs-lite peers to exchange blocks was the hardest and most rewarding part of this project. The challenge wasn't the protobuf messages — it was understanding how Go's bitswap engine manages streams.

The key insight: Go's bitswap does not respond on the same stream where it receives a wantlist. Instead, it closes the read end immediately and opens a new outbound stream to push blocks back. The initial Python implementation assumed same-stream request/response, which worked fine between two Python peers but silently failed with Go.

Two-phase block fetch: To match Go's behavior, a two-phase negotiation was implemented:

1. Discovery — WANT_HAVE is sent to all connected peers; the first HAVE response is awaited (or a direct block if Go decides the block is small enough to skip the HAVE phase)
2. Retrieval — WANT_BLOCK is sent only to peers that responded with HAVE, then block delivery is awaited on a new inbound stream

The handle_stream handler also had to be switched to a looping design (reading multiple messages per stream, like Go's handleNewStream) since Go reuses streams for sequential want messages.

The result: a Python peer can now add_file() locally and a Go peer can get_block() for that CID over the network — and vice versa.

What Was Fixed in py-libp2p's Yamux

Building a real protocol on py-libp2p exposed two significant issues in the yamux stream muxer. Both have been fixed in a fork and are being contributed upstream.

1. Frame type correctness (#1271)

SYN, ACK, FIN, and RST control frames were being sent as TYPE_DATA instead of TYPE_WINDOW_UPDATE. This is a spec violation — go-yamux sends all control frames as TYPE_WINDOW_UPDATE, and the length field carries the window delta, not a data payload size. Additionally, concurrent writes could interleave frames on the wire, corrupting messages.

Fix: All control frames were changed to TYPE_WINDOW_UPDATE and a _write_lock was added to serialize writes. (asabya/py-libp2p#1)

2. Receive window auto-tuning (#1270)

py-libp2p uses a fixed 256KB receive window. Go-yamux dynamically grows it up to 16MB based on RTT, which means a 1GB transfer needs ~4,000 round-trips with the fixed window vs ~64 with auto-tuning.

Fix: go-yamux's auto-tuning algorithm was ported — the window doubles each RTT epoch with 50% hysteresis, plus background RTT measurement via ping/pong. (asabya/py-libp2p#2, upstream PR #1269)

What's Next

py-ipfs-lite is functional but early (v0.1.0). Community input and collaboration would be greatly appreciated:

• Upstreaming the yamux fixes — The frame type bug is a correctness issue that affects any Go ↔ Python connection. Both fixes are ready to be merged into py-libp2p.
• Testing with more Go implementations — So far testing has been done against Go ipfs-lite; testing against Kubo and other implementations would increase confidence.
• Reprovider loop — Periodically re-announcing all stored blocks via DHT (like Go's 12-hour reprovide interval) is planned.
• Performance profiling — Throughput and latency measurements for large file transfers

Feedback, questions, and contributions are all welcome. The project is at github.com/asabya/py-ipfs-lite.