Skip to content

rfc: forge s3 sharding strategy#2

Open
alanshaw wants to merge 2 commits intomainfrom
ash/docs/forge-s3-flat-file-sharding-strategy
Open

rfc: forge s3 sharding strategy#2
alanshaw wants to merge 2 commits intomainfrom
ash/docs/forge-s3-flat-file-sharding-strategy

Conversation

@alanshaw
Copy link
Copy Markdown
Member

@alanshaw alanshaw commented Apr 29, 2026


## Design

The Sharded DAG Index will gain a `blocks` property - a list of blocks "inlined" in the index. See [#66](https://github.com/storacha/RFC/pull/66).
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: Do we actually need a blocks property at all? We'll already have a root CID that we're looking up, and we'll get the index back from the indexer. That's already a CAR. Can we just include that block in the CAR, and not link to it from the index block?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could, but that block might not round trip serialization/deserialization since there's nothing linking to it.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, right: indexes are rooted at a block that's given as a root in their CAR. The CAR is the transport, it's not the index itself. I'm thinking of shards, which don't have roots and where the CAR itself is the shard.

Yeah, they should be linked into the DAG and reachable from the root. But maybe we should call them nodes:? That's the more UnixFS-specific terminology. We don't expect arbitrary blocks here, right? And if we needed them, we might link to them under a different key?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fine by me.

@alanshaw alanshaw requested review from Peeja, frrist and hannahhoward May 5, 2026 12:55
@alanshaw
Copy link
Copy Markdown
Member Author

alanshaw commented May 5, 2026

@frrist is this ok by you?

Copy link
Copy Markdown
Member

@frrist frrist left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm.

I have some concerns wrt passthrough from the upload service/s3 facade to the storage nodes, but that's probably more of an implementation detail.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants