Update everything to m2.5 contracts

[juliangruber]

Tests haven't been looked at yet. Next: Run tests, get everything green, add comments where necessary

[bajtos]

Great work!

It's a lot of changes to review. I don't think it's possible to catch all issues in the review process. I expect there will be bugs we discover only after we run this against the real contract & Goldsky pipeline. With that in mind, I did not dive deep into all the details; I think speed is more important than perfection in this case.

I would also like @pyropy to review the changes; he may be more familiar with the new smart contracts and their APIs.

[bajtos]

+1 to use a more descriptive name.

However, the suffix -handlers does not match the content of this file. There are no handlers AFAIK, only functions querying the database.

Let's find a better suffix please.

Suggestions:

• indexer/lib/pdp-verifier-db.js
• indexer/lib/pdp-verifier-repository.js
• indexer/lib/pdp-verifier-queries.js
• indexer/lib/pdp-verifier-store.js

[bajtos]

If that's the case, can we delegate the process of removing the provider from our DB to handleProviderRemoved?

[bajtos]

A question: why the data-access functions insertDataSetPieces and removeDataSetPieces live in their own file, but the data-access code for manipulating the list of providers does not live in functions outside of this file?

[juliangruber]

I don't understand the question. The functions you are referring to are handlers for PDP routes, and live in the PDP handler file. The functions you are quoting on are handlers for service provider registry routes, and live in the service provider registry handler file.

[Copilot]

Pull Request Overview

This PR updates the codebase to use m2.5 contracts, refactoring the database schema and API from proof-set-based to data-set-based terminology. The changes affect the core data model, moving from owner/proof-set relationships to storage-provider/data-set relationships.

• Database schema migration from proof sets to data sets with new table structures
• API endpoint updates from proof-set-related paths to data-set-related paths
• Terminology changes throughout: owner → storageProvider, rootCid → pieceCid, proofSetId → dataSetId

Reviewed Changes

Copilot reviewed 21 out of 24 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
retriever/test/test-data.js	Updates test data structure from owner/rootCid to storageProvider/pieceCid
retriever/test/test-data-builders.js	Refactors test builders from proof set to data set terminology and database structure
retriever/test/store.test.js	Updates store tests to use new data set terminology and database schema
retriever/test/retriever.test.js	Updates retriever tests to use piece CIDs instead of root CIDs
retriever/test/retrieval.test.js	Updates retrieval tests to use pieceCid parameter
retriever/test/request.test.js	Updates request parsing tests for pieceCid
retriever/lib/store.js	Major refactoring from proof set to data set operations and database queries
retriever/lib/retrieval.js	Updates URL construction to use serviceUrl instead of pieceRetrievalUrl
retriever/lib/request.js	Updates request parsing to handle pieceCid instead of rootCid
indexer/wrangler.toml	Removes deprecated RPC_URL and PDP_VERIFIER_ADDRESS environment variables
indexer/test/test-data.js	Updates test data from proof set to data set structure
indexer/test/pdp-verifier.test.js	Removes entire test file for deprecated PDP verifier functionality
indexer/test/indexer.test.js	Major refactoring of indexer tests for new API endpoints and data structures
indexer/package.json	Removes multiformats dependency no longer needed
indexer/lib/store.js	Removes deprecated proof set operations
indexer/lib/service-provider-registry-handlers.js	New handlers for service provider registry events
indexer/lib/provider-events-handler.js	Removes deprecated provider event handlers
indexer/lib/pdp-verifier.js	Removes deprecated PDP verifier client
indexer/lib/pdp-verifier-handlers.js	New handlers for PDP verifier data set operations
indexer/lib/filecoin-warm-storage-service-handlers.js	Updates handlers for new data set structure
db/migrations/0013_m2_5_upgrades.sql	Database migration script dropping old tables and creating new m2.5 schema

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[pyropy]

Great work so far!

[pyropy]

This might cause a edge-case that @bajtos has mentioned above.

If provider deletes the product before it gets deleted from the registry we'll get response with status code 404.

[juliangruber]

could you please describe this edge case more?

[pyropy]

I would suggest adding a active column to the providers table as providers can have isActive field inside the struct.

[juliangruber]

I don't see any events for this state change, do you know when we would update this field?

[Copilot]

Pull Request Overview

This PR updates the codebase from m2.4 to m2.5 contracts, reflecting a migration from the PDP verifier system to a new architecture with data sets, pieces, and service providers.

Key changes include:

• Renaming core entities: proof sets → data sets, roots → pieces, owners → storage providers
• Updating database schema with new table structures for providers, data_sets, and pieces
• Replacing PDP verifier contract calls with service provider registry integration

Reviewed Changes

Copilot reviewed 21 out of 24 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
retriever/test/test-data.js	Updates test data structure to use new field names for storage providers and data sets
retriever/test/test-data-builders.js	Refactors helper functions to work with new data set and piece model
retriever/test/store.test.js	Updates test cases for storage provider validation and data set statistics
retriever/test/retriever.test.js	Modifies retrieval tests to use piece CIDs and storage provider addresses
retriever/test/retrieval.test.js	Updates retrieval function tests with new parameter names
retriever/test/request.test.js	Changes URL parsing tests to use piece CID terminology
retriever/lib/store.js	Major refactor of database queries and validation logic for new schema
retriever/lib/retrieval.js	Updates URL construction to use service URLs instead of piece retrieval URLs
retriever/lib/request.js	Modifies request parsing to handle piece CIDs instead of root CIDs
indexer/wrangler.toml	Removes PDP verifier configuration variables
indexer/test/test-data.js	Updates test data structure for new piece-based model
indexer/test/pdp-verifier.test.js	Removes PDP verifier client tests
indexer/test/indexer.test.js	Comprehensive test updates for new service provider registry endpoints
indexer/package.json	Removes multiformats dependency
indexer/lib/store.js	Removes proof set root management functions
indexer/lib/service-provider-registry-handlers.js	New module for handling service provider registry events
indexer/lib/provider-events-handler.js	Removes legacy provider event handling
indexer/lib/pdp-verifier.js	Removes PDP verifier client implementation
indexer/lib/pdp-verifier-handlers.js	New module for PDP verifier data set and piece operations
indexer/lib/filecoin-warm-storage-service-handlers.js	Updates to handle new data set creation model
db/migrations/0013_m2_5_upgrades.sql	Database migration script for new schema

Comments suppressed due to low confidence (1)

retriever/lib/store.js:1

• [nitpick] The variable 'storageProviderAddress2' on line 293 should be renamed to distinguish it from 'storageProviderAddress1'. Currently both have similar values ending in '...e7' and '...e9' which could cause confusion.

import { httpAssert } from './http-assert.js'

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[juliangruber]

I'm going to go ahead and merge, so that I can follow up with more updates and not be blocked. Let's follow up with any other open points afterwards please.