feat(synapse-core): add SP-to-SP piece pull functionality #544

rvagg · 2026-01-12T11:24:25Z

Ref: filecoin-project/curio#864
Part of #494

cloudflare-workers-and-pages · 2026-01-12T11:27:01Z

Deploying with Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	synapse-dev	`59e576b`	Commit Preview URL Branch Preview URL	Feb 06 2026, 02:02 PM

rvagg · 2026-01-17T11:46:56Z

Added a temporary utils/example-sp-fetch-e2e.js testing file to test the sp-sp fetch flow. It stores pieces with one SP, confirms they are parked, then asks the second SP to fetch from the first and confirms they are parked. It's working currently on devnet using FilOzone/foc-devnet#34 👌.

rvagg · 2026-01-28T10:30:40Z

Rebased on top of the latest @hugomrdias stack #571

utils/example-pull-e2e.js

rvagg · 2026-02-05T10:32:39Z

latest fixup in here adds the ability to supply a pre-built extraData since we use it for auth but then use it again to addPieces and we don't want the user to have to sign twice for the same thing if we can help it

rvagg · 2026-02-05T12:15:45Z

And another fix, I realised I still had a ton of "fetch" in here. Now all "pull".

Add low-level Pull module for initiating piece pulls from external storage providers to the local SP via Curio's POST /pdp/piece/pull endpoint. Features: - fetchPieces: Initiate pull requests with EIP-712 signed authorization - waitForFetchStatus: Poll until completion with status callbacks - Support for both existing data sets (signAddPieces) and new data sets (signCreateDataSetAndAddPieces) - Configurable timeout and abort signal support The pull endpoint is idempotent - repeated calls with the same extraData return status updates rather than creating duplicates.

hugomrdias

im not the biggest fan of this idempotent POST endpoint, left some comment inline.

hugomrdias · 2026-02-10T10:45:07Z

packages/synapse-core/src/warm-storage/pieces.ts

if you are pre signing (still need to figure out why), why dont you use PDP.addPieces directly ?

hugomrdias · 2026-02-10T10:50:00Z

packages/synapse-core/src/warm-storage/pull.ts

i think this should be inside the sp folder

hugomrdias · 2026-02-10T10:55:48Z

packages/synapse-core/src/warm-storage/pull.ts

+    /** Minimum time between poll attempts in milliseconds (default: 4000). */
+    minTimeout?: number


Suggested change

/** Minimum time between poll attempts in milliseconds (default: 4000). */

minTimeout?: number

/** The timeout in milliseconds. Defaults to 5 minutes. */

timeout?: number

/** The polling interval in milliseconds. Defaults to 4 seconds. */

pollInterval?: number

changed to this pattern in the other waitFor* helps in the tests

hugomrdias · 2026-02-10T11:02:15Z

packages/synapse-core/src/sp.ts

+  const fetchWithFreshRequest: typeof globalThis.fetch = (input, init) => {
+    // iso-web passes the Request object as input, extract signal from it
+    const signal = input instanceof Request ? input.signal : init?.signal
+    return globalThis.fetch(url, { method: 'POST', body, headers, signal })
+  }


how does this, fixes body being consumed ?

hugomrdias · 2026-02-10T11:07:12Z

packages/synapse-core/test/pull.test.ts

+
+  beforeEach(() => {
+    server.resetHandlers()
+    SP.setTimeout(1000) // Short timeout for tests


right, needs rebasing into the other branch, there no setTimeout method anymore you can pass the max timeout as an option

hugomrdias · 2026-02-10T11:12:39Z

packages/synapse-core/test/pull.test.ts

+      const result = await SP.waitForPullStatus({
+        ...baseOptions(),
+        minTimeout: 10,
+        onStatus: (response) => statusUpdates.push(response.status),
+      })


yeah this is why you need to presign because you need it for the waitFor function, and you poll from a http POST endpoint.

this breaks the pattern of, do mutation, get status url to http poll from.

github-project-automation bot added this to FOC Jan 12, 2026

github-project-automation bot moved this to 📌 Triage in FOC Jan 12, 2026

BigLep assigned rvagg Jan 15, 2026

BigLep moved this from 📌 Triage to ⌨️ In Progress in FOC Jan 15, 2026

BigLep linked an issue Jan 16, 2026 that may be closed by this pull request

GA DURABILITY: Multi-copy upload via SP-to-SP pull #494

Open

5 tasks

rvagg force-pushed the rvagg/sp-sp-fetch branch from c2011f6 to 7af205b Compare January 17, 2026 06:01

rvagg mentioned this pull request Jan 17, 2026

feat: support SP-to-SP fetch via host.docker.internal resolution FilOzone/foc-devnet#34

Merged

rvagg force-pushed the rvagg/sp-sp-fetch branch from ac3552a to 3c5cdb9 Compare January 20, 2026 09:08

rvagg changed the base branch from master to hugomrdias/move-tests January 20, 2026 09:08

hugomrdias force-pushed the hugomrdias/move-tests branch 2 times, most recently from 0a20df0 to a231d32 Compare January 26, 2026 12:20

rvagg force-pushed the rvagg/sp-sp-fetch branch from 3c5cdb9 to 429c2e3 Compare January 28, 2026 10:29

rvagg changed the base branch from hugomrdias/move-tests to hugomrdias/session-key January 28, 2026 10:30

rjan90 added this to the M4.0: mainnet staged milestone Jan 30, 2026

Base automatically changed from hugomrdias/session-key to hugomrdias/payments January 30, 2026 12:03

Base automatically changed from hugomrdias/payments to hugomrdias/move-tests February 2, 2026 19:19

rvagg force-pushed the rvagg/sp-sp-fetch branch from 429c2e3 to 8ec5942 Compare February 3, 2026 03:23

hugomrdias force-pushed the hugomrdias/move-tests branch from a5f8971 to d75fb75 Compare February 3, 2026 11:52

rvagg force-pushed the rvagg/sp-sp-fetch branch from 8ec5942 to b6d3479 Compare February 4, 2026 10:44

rvagg changed the base branch from hugomrdias/move-tests to master February 4, 2026 10:45

rvagg changed the title ~~(WIP) feat(synapse-core): add sp-fetch module for SP-to-SP piece transfers~~ feat(synapse-core): add sp-fetch module for SP-to-SP piece transfers Feb 4, 2026

rvagg marked this pull request as ready for review February 4, 2026 10:45

rvagg requested a review from hugomrdias as a code owner February 4, 2026 10:45

rvagg commented Feb 4, 2026

View reviewed changes

utils/example-pull-e2e.js Outdated Show resolved Hide resolved

rvagg force-pushed the rvagg/sp-sp-fetch branch 2 times, most recently from b221f97 to ff14a1d Compare February 4, 2026 11:31

rjan90 moved this from ⌨️ In Progress to 🔎 Awaiting review in FOC Feb 4, 2026

rvagg mentioned this pull request Feb 4, 2026

GA DURABILITY: Multi-copy upload via SP-to-SP pull #494

Open

5 tasks

rvagg force-pushed the rvagg/sp-sp-fetch branch from ff14a1d to 13ed974 Compare February 5, 2026 10:30

rvagg changed the title ~~feat(synapse-core): add sp-fetch module for SP-to-SP piece transfers~~ feat(synapse-core): add SP-to-SP piece pull functionality Feb 5, 2026

rvagg force-pushed the rvagg/sp-sp-fetch branch from 13ed974 to 2d1a65e Compare February 5, 2026 10:32

rvagg force-pushed the rvagg/sp-sp-fetch branch from 2d1a65e to caebe09 Compare February 5, 2026 12:15

rvagg force-pushed the rvagg/sp-sp-fetch branch from caebe09 to 2f8c0ca Compare February 5, 2026 12:16

rvagg force-pushed the rvagg/sp-sp-fetch branch from 2f8c0ca to 59e576b Compare February 6, 2026 13:56

rjan90 mentioned this pull request Feb 6, 2026

Cut Synapse-SDK v0.38.0 #591

Open

2 tasks

rvagg mentioned this pull request Feb 6, 2026

feat(storage): multi-copy upload with store->pull->commit flow #593

Open

hugomrdias requested changes Feb 10, 2026

View reviewed changes

github-project-automation bot moved this from 🔎 Awaiting review to ⌨️ In Progress in FOC Feb 10, 2026

		/** Minimum time between poll attempts in milliseconds (default: 4000). */
		minTimeout?: number

-    /** Minimum time between poll attempts in milliseconds (default: 4000). */
-    minTimeout?: number
+    /** The timeout in milliseconds. Defaults to 5 minutes. */
+    timeout?: number
+    /** The polling interval in milliseconds. Defaults to 4 seconds. */
+    pollInterval?: number

feat(synapse-core): add SP-to-SP piece pull functionality #544

Are you sure you want to change the base?

feat(synapse-core): add SP-to-SP piece pull functionality #544

Uh oh!

Conversation

rvagg commented Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cloudflare-workers-and-pages bot commented Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying with Cloudflare Workers

Uh oh!

rvagg commented Jan 17, 2026

Uh oh!

rvagg commented Jan 28, 2026

Uh oh!

Uh oh!

rvagg commented Feb 5, 2026

Uh oh!

rvagg commented Feb 5, 2026

Uh oh!

hugomrdias left a comment

Choose a reason for hiding this comment

Uh oh!

hugomrdias Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

hugomrdias Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

hugomrdias Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

hugomrdias Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

hugomrdias Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

hugomrdias Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rvagg commented Jan 12, 2026 •

edited

Loading

cloudflare-workers-and-pages bot commented Jan 12, 2026 •

edited

Loading