Skip to content

feat(iroh): add persistence to a local iroh-blobs store#21

Merged
cameronfyfe merged 10 commits into
eqtylab:mainfrom
rklaehn:persistence
Dec 18, 2025
Merged

feat(iroh): add persistence to a local iroh-blobs store#21
cameronfyfe merged 10 commits into
eqtylab:mainfrom
rklaehn:persistence

Conversation

@rklaehn

@rklaehn rklaehn commented Nov 12, 2025

Copy link
Copy Markdown
Contributor

This is very WIP and mostly serves as a place to have a discussion about the needed feature set.

I added a new trait for CRPs to persist data and implemented it for the IrohCrp. So far persistence is only concerned with small to medium sized blobs, so no streaming. But of course the iroh-blobs protocol does support streaming if we need it later.

The implementation contains some utilities that are copied from iroh-blobs and should really be in iroh-blobs in the future.

Persistence is exposed as a new REST endpoint:

/v1/data POST

Cid type will be inferred from http content type, falling back to Raw if no content type is given, and erroring out if no supported content type is given.

In principle this supports multiple storage places, but this opens up the question if a request should be considered successful if a) at least 1 storage backed was successful or b) if all storage backends were successful.

You could also implement an iroh inline Crp that lives in process, but this is TBD. I am not quite sure where the cid router lives in relation to the storage backend.

To try this out, you need a blob store that does support PUSH requests, which are disabled by default for security reasons. Such a store can be found in n0-computer/iroh-blobs#195 .

You have to explicitly enable the source endpoint id of the pusher. To give a stable endpoint id I have just used the identity of the core also for the iroh endpoint.

Cid-router needs to be brought up to date with the latest iroh and blobs, and this will require some changes - most notably node_id is now called endpoint_id. But this can happen in a different PR.

Additional changes:

  • removed size requests
  • removed remote iroh CRP
  • updated swagger API with the new routes

@rklaehn

rklaehn commented Nov 12, 2025

Copy link
Copy Markdown
Contributor Author

Implements #20

@rklaehn

rklaehn commented Nov 18, 2025

Copy link
Copy Markdown
Contributor Author

Todo: inline?

@rklaehn

rklaehn commented Dec 9, 2025

Copy link
Copy Markdown
Contributor Author

Test with curl:

Adding data:

cid-router on  persistence [!?] via 🦀 v1.89.0 
❯ curl -X POST --data-binary @test-file.bin http://127.0.0.1:3080/v1/data | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1024k  100   166  100 1024k   7357  44.3M --:--:-- --:--:-- --:--:-- 45.4M
{
  "cid": "bafkr4igjxmpnibvuydqgfatacfimog6dmfrlxpihe2jlbdczyunhd5ande",
  "location": "/v1/data/bafkr4igjxmpnibvuydqgfatacfimog6dmfrlxpihe2jlbdczyunhd5ande",
  "size": 1048576
}

Getting info:

❯ curl http://127.0.0.1:3080/v1/routes/bafkr4igjxmpnibvuydqgfatacfimog6dmfrlxpihe2jlbdczyunhd5ande | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   144  100   144    0     0   121k      0 --:--:-- --:--:-- --:--:--  140k
{
  "routes": [
    {
      "provider_id": "iroh",
      "type": "iroh",
      "size": 1048576,
      "url": "foo",
      "cid": "bafkr4igjxmpnibvuydqgfatacfimog6dmfrlxpihe2jlbdczyunhd5ande"
    }
  ]
}

Getting data:

cid-router on  persistence [!?] via 🦀 v1.89.0 
❯ curl -v -v http://127.0.0.1:3080/v1/data/bafkr4igjxmpnibvuydqgfatacfimog6dmfrlxpihe2jlbdczyunhd5ande | b3sum
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 127.0.0.1:3080...
* Connected to 127.0.0.1 (127.0.0.1) port 3080
> GET /v1/data/bafkr4igjxmpnibvuydqgfatacfimog6dmfrlxpihe2jlbdczyunhd5ande HTTP/1.1
> Host: 127.0.0.1:3080
> User-Agent: curl/8.7.1
> Accept: */*
> 
* Request completely sent off
< HTTP/1.1 200 OK
< content-type: application/octet-stream
< transfer-encoding: chunked
< date: Tue, 09 Dec 2025 09:49:37 GMT
< 
{ [102276 bytes data]
100 1024k    0 1024k    0     0  42.2M      0 --:--:-- --:--:-- --:--:-- 43.4M
* Connection #0 to host 127.0.0.1 left intact
c9bb1ed406b4c0e06282601150c71bc36162bbbd072692b08c59c51a71f40d19  -

cid-router on  persistence [!?] via 🦀 v1.89.0 
❯ b3sum test-file.bin 
c9bb1ed406b4c0e06282601150c71bc36162bbbd072692b08c59c51a71f40d19  test-file.bin

@rklaehn rklaehn changed the title WIP add persistence to a remote iroh-blobs store via iroh-blobs PUSH requests feat(iroh): add persistence to a local iroh-blobs store Dec 9, 2025
@rklaehn
rklaehn marked this pull request as ready for review December 9, 2025 11:14
Add schema to swagger
Fix imports
@rklaehn

rklaehn commented Dec 9, 2025

Copy link
Copy Markdown
Contributor Author

Updated the swagger types:

image

@cameronfyfe

Copy link
Copy Markdown
Contributor

This looks good and is working for me locally. I think we can merge after resolving merge conflicts

@rklaehn

rklaehn commented Dec 18, 2025

Copy link
Copy Markdown
Contributor Author

@cameronfyfe fixed the merge conflicts

@cameronfyfe cameronfyfe left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 I'll fix CI failure in another PR

@cameronfyfe
cameronfyfe merged commit 3a6b798 into eqtylab:main Dec 18, 2025
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants