Add import dataset

[HeDo88TH]

As title says

[Copilot]

Pull request overview

Adds the “Import Dataset” capability to the Registry backend, wiring a new org-scoped API + orchestration layer to a worker-executed heavy tool that can populate a newly created dataset from supported remote sources.

Changes:

• Registers Import Dataset services on both web host and processing node, including shared Data Protection for credential encryption/decryption.
• Introduces import orchestration (IImportManager/ImportManager), a new ImportController, and source infrastructure (IImportSource*, SSRF guard, remote registry client, archive URL source).
• Adds the import-dataset heavy tool to fetch, index, enqueue build jobs, and invalidate caches; improves HTTP download retry behavior to honor Retry-After.

Reviewed changes

Copilot reviewed 27 out of 27 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
Registry.Web/Utilities/ProcessingNodeServiceCollectionExtensions.cs	Registers import sources on processing nodes so the worker can execute import tasks.
Registry.Web/Utilities/ImportServiceCollectionExtensions.cs	Adds DI wiring for import sources, credential protection, and the import-dataset heavy tool.
Registry.Web/Utilities/DataProtectionExtensions.cs	Adds shared Data Protection key-ring configuration for cross-host credential decrypt.
Registry.Web/Startup.cs	Registers import sources + IImportManager on the web host.
Registry.Web/Services/Ports/IImportManager.cs	Defines the orchestration interface for verify/create import flows.
Registry.Web/Services/Managers/ImportManager.cs	Implements verify + create flow, quota gating, task submission, and rollback on submit failure.
Registry.Web/Services/Import/SsrfGuard.cs	Adds SSRF protection for outbound import connections.
Registry.Web/Services/Import/RemoteRegistryClient.cs	Implements remote registry auth/list/download with parallel downloads and progress reporting.
Registry.Web/Services/Import/RegistryImportSource.cs	Implements the “registry” import source using SSRF guard + remote client.
Registry.Web/Services/Import/IRemoteRegistryClient.cs	Defines the remote-registry client abstraction used by the registry import source.
Registry.Web/Services/Import/ImportSourceFactory.cs	Resolves registered sources by type (case-insensitive).
Registry.Web/Services/Import/ImportCredentialProtector.cs	Implements credential protection with ASP.NET Core Data Protection purpose isolation.
Registry.Web/Services/Import/ArchiveUrlImportSource.cs	Implements the “archive-url” import source (download to scratch + extract with zip-slip protection).
Registry.Web/Services/HeavyTasks/Tools/ImportDatasetTool.cs	Worker-side heavy tool that fetches into dataset folder, enforces budgets, indexes, and schedules builds.
Registry.Web/Registry.Web.csproj	Adds DataProtection StackExchangeRedis package for shared key persistence.
Registry.Web/Models/DTO/ImportSourceDto.cs	Adds DTOs for verify/create import API requests and responses.
Registry.Web/Models/Configuration/ImportSettings.cs	Adds configuration model for import feature behavior and SSRF policy.
Registry.Web/Models/Configuration/AppSettings.cs	Adds Import and DataProtectionKeysPath settings to AppSettings.
Registry.Web/Controllers/ImportController.cs	Adds REST endpoints for listing sources, verifying, and creating imports under /orgs/{org}/import.
Registry.Web.Test/SsrfGuardTests.cs	Adds coverage for SSRF guard blocking/allowing address scenarios.
Registry.Web.Test/ImportSourceFactoryTests.cs	Adds tests for factory resolution behavior and available types.
Registry.Ports/Import/ImportModels.cs	Adds shared import port models (ImportSourceProbe, ImportProgress).
Registry.Ports/Import/IImportSourceFactory.cs	Adds port interface for source resolution + available types.
Registry.Ports/Import/IImportSource.cs	Adds port interface for import sources (probe/fetch).
Registry.Ports/Import/IImportCredentialProtector.cs	Adds port interface for credential encrypt/decrypt bridging web and worker.
Registry.Common/HttpHelper.cs	Enhances retry behavior to honor Retry-After for retryable HTTP statuses.

[Copilot]

Pull request overview

Copilot reviewed 38 out of 38 changed files in this pull request and generated 4 comments.