Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
d5a1b11
feat: add Nexus-compatible Terraform repositories
klboke Jul 14, 2026
84ad2a8
fix: parse Terraform upload filenames in linear time
klboke Jul 14, 2026
b956f15
test: cover Terraform repository service workflows
klboke Jul 14, 2026
2871e48
test: expand Terraform migration and validation coverage
klboke Jul 15, 2026
e96bba7
fix: address Terraform review findings
klboke Jul 15, 2026
18dd871
fix: tighten Terraform authorization and group metadata
klboke Jul 15, 2026
df60e7f
fix: enforce Terraform module coordinates and metadata ttl
klboke Jul 15, 2026
9c74c5e
fix: preserve Terraform go-getter module sources
klboke Jul 15, 2026
0c7ee3a
fix: bound Terraform group keys and parse URL tokens
klboke Jul 15, 2026
579303d
fix: coordinate Terraform publication and metadata history
klboke Jul 15, 2026
8f754e3
fix: isolate Terraform migration and discovery state
klboke Jul 15, 2026
7ee1402
fix: filter Terraform component search
klboke Jul 15, 2026
1400088
fix: recover Terraform provider publication state
klboke Jul 15, 2026
5b07cda
fix: accept Terraform JSON module syntax
klboke Jul 15, 2026
a7fe034
fix: inspect provider uploads before leasing
klboke Jul 15, 2026
f2e556e
fix: discard invalid Terraform proxy uploads
klboke Jul 15, 2026
2e01c19
fix: propagate Terraform upstream metadata failures
klboke Jul 15, 2026
ff6097e
fix: complete Terraform browse uploads
klboke Jul 15, 2026
4cce21a
fix: resolve Terraform discovery service URLs
klboke Jul 15, 2026
9e6a395
fix: pin Terraform proxy metadata routes
klboke Jul 15, 2026
53a89c4
fix: harden Terraform proxy compatibility
klboke Jul 15, 2026
b156007
fix: preserve Terraform private download auth
klboke Jul 15, 2026
958986d
fix(terraform): preserve Nexus signing key IDs
klboke Jul 15, 2026
a079a98
fix(terraform): preserve Nexus archive links
klboke Jul 15, 2026
6cfc292
test(terraform): import Nexus alias URL parser
klboke Jul 15, 2026
02b9439
fix(terraform): bound XZ decoder memory
klboke Jul 15, 2026
62e74fb
fix(terraform): return Nexus provider archive URLs
klboke Jul 15, 2026
e181bb8
fix(terraform): resolve relative module sources
klboke Jul 15, 2026
ff70c8a
fix(terraform): reject unsafe unknown paths
klboke Jul 15, 2026
c7577de
fix(terraform): harden token fallback and redaction
klboke Jul 15, 2026
02d6ed7
fix(terraform): allow provider platform redeploys
klboke Jul 15, 2026
52a2fca
fix(terraform): require edit for provider redeploys
klboke Jul 15, 2026
f13e211
fix(terraform): hide internal cache assets
klboke Jul 15, 2026
956001e
test(terraform): pin checksum metadata routes
klboke Jul 15, 2026
c349efb
fix(terraform): keep provider metadata URLs safe
klboke Jul 15, 2026
d710432
ci: isolate migration matrix artifacts
klboke Jul 15, 2026
b7d9467
fix(terraform): redact e2e tokens and close migration streams
klboke Jul 15, 2026
f89ec18
fix(terraform): partition proxy provider routes
klboke Jul 15, 2026
c3a004a
docs(terraform): complete project documentation
klboke Jul 15, 2026
cdd2136
fix(terraform): reject unsafe proxy cache migration
klboke Jul 15, 2026
6e7e42b
fix(terraform): migrate Nexus proxy cache archives
klboke Jul 15, 2026
293cd2c
fix(terraform): propagate group merge failures
klboke Jul 15, 2026
ce4bfb2
fix(terraform): surface empty group upstream failures
klboke Jul 15, 2026
4b3b27a
fix(terraform): preserve proxy cache snapshots
klboke Jul 15, 2026
06d190b
fix(terraform): return not found for empty packages
klboke Jul 15, 2026
96f94c5
fix(terraform): persist provider protocols
klboke Jul 15, 2026
a5f4372
test(terraform): cover provider protocol header
klboke Jul 15, 2026
99aaea0
fix(terraform): align logical components and browse behavior
klboke Jul 16, 2026
644bd82
fix(terraform): renew publication leases
klboke Jul 16, 2026
1166205
fix(terraform): verify migrated provider cache
klboke Jul 16, 2026
92b1519
perf(terraform): optimize metadata and asset hot paths
klboke Jul 16, 2026
ca13380
test(terraform): cover non-token path normalization
klboke Jul 16, 2026
41c264f
fix(terraform): resolve browse aliases before delete
klboke Jul 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions .github/workflows/live-compat.yml
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,22 @@ jobs:
if: env.LIVE_COMPAT_SUITE == 'client-e2e'
uses: oras-project/setup-oras@v2

- name: Set up Terraform 0.13 and current stable
if: env.LIVE_COMPAT_SUITE == 'client-e2e'
shell: bash
run: |
set -euo pipefail
mkdir -p "$RUNNER_TEMP/terraform/0.13.7" "$RUNNER_TEMP/terraform/1.15.8"
for version in 0.13.7 1.15.8; do
archive="terraform_${version}_linux_amd64.zip"
curl -fsSLO "https://releases.hashicorp.com/terraform/${version}/${archive}"
curl -fsSLO "https://releases.hashicorp.com/terraform/${version}/terraform_${version}_SHA256SUMS"
grep " ${archive}$" "terraform_${version}_SHA256SUMS" | sha256sum -c -
unzip -q "$archive" -d "$RUNNER_TEMP/terraform/$version"
done
echo "TERRAFORM_013_BIN=$RUNNER_TEMP/terraform/0.13.7/terraform" >> "$GITHUB_ENV"
echo "TERRAFORM_CURRENT_BIN=$RUNNER_TEMP/terraform/1.15.8/terraform" >> "$GITHUB_ENV"

- name: Install Python client build tools
if: env.LIVE_COMPAT_SUITE == 'client-e2e'
run: python -m pip install --upgrade build twine
Expand Down
29 changes: 25 additions & 4 deletions .github/workflows/migration-e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ concurrency:

jobs:
nexus-migration-e2e:
name: Nexus ${{ matrix.nexus-version }} migration E2E
name: Nexus ${{ matrix.nexus-version }} (${{ matrix.metadata-engine }}) migration E2E
runs-on: ubuntu-latest
timeout-minutes: 120
if: github.event_name == 'workflow_dispatch' || (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'run-migration-e2e'))
Expand Down Expand Up @@ -133,21 +133,42 @@ jobs:
- name: Prepare migration fixtures
run: scripts/ci/live-compat-setup.sh

- name: Set up Terraform current stable
if: matrix.nexus-version == '3.92.0'
shell: bash
run: |
set -euo pipefail
version=1.15.8
archive="terraform_${version}_linux_amd64.zip"
mkdir -p "$RUNNER_TEMP/terraform/$version"
curl -fsSLO "https://releases.hashicorp.com/terraform/${version}/${archive}"
curl -fsSLO "https://releases.hashicorp.com/terraform/${version}/terraform_${version}_SHA256SUMS"
grep " ${archive}$" "terraform_${version}_SHA256SUMS" | sha256sum -c -
unzip -q "$archive" -d "$RUNNER_TEMP/terraform/$version"
echo "TERRAFORM_CURRENT_BIN=$RUNNER_TEMP/terraform/$version/terraform" >> "$GITHUB_ENV"

- name: Prepare Nexus Terraform migration fixture
if: matrix.nexus-version == '3.92.0'
run: >-
mvn -B -ntp -pl compat-test -am
-Dtest=TerraformRepositoryBlackBoxCompatibilityTest#preparesNexusHostedMigrationFixtureWhenConfigured
-Dsurefire.failIfNoSpecifiedTests=false test

- name: Run Nexus metadata and repository-data migration E2E
run: scripts/docker-compat/migration-e2e.sh

- name: Collect migration logs
if: always()
run: |
mkdir -p artifacts
docker compose -f "$COMPOSE_FILE" ps > "artifacts/migration-e2e-${{ matrix.nexus-version }}-ps.txt" || true
docker compose -f "$COMPOSE_FILE" logs --no-color --timestamps > "artifacts/migration-e2e-${{ matrix.nexus-version }}-compose.log" || true
docker compose -f "$COMPOSE_FILE" ps > "artifacts/migration-e2e-${{ matrix.nexus-version }}-${{ matrix.metadata-engine }}-ps.txt" || true
docker compose -f "$COMPOSE_FILE" logs --no-color --timestamps > "artifacts/migration-e2e-${{ matrix.nexus-version }}-${{ matrix.metadata-engine }}-compose.log" || true

- name: Upload migration logs
if: always()
uses: actions/upload-artifact@v7
with:
name: migration-e2e-${{ matrix.nexus-version }}-logs
name: migration-e2e-${{ matrix.nexus-version }}-${{ matrix.metadata-engine }}-logs
path: artifacts/
if-no-files-found: ignore

Expand Down
3 changes: 2 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

构建一个兼容 Nexus 的自托管制品仓库,包含:

- Maven、npm、PyPI、Go、Helm、Cargo/Rust、Dart/Pub、Composer/PHP、Docker/OCI、NuGet、RubyGems、Yum 和 Raw 制品支持;具体 hosted/proxy/group 边界以兼容性矩阵为准。
- Maven、npm、PyPI、Go、Helm、Cargo/Rust、Dart/Pub、Composer/PHP、Terraform、Docker/OCI、NuGet、RubyGems、Yum 和 Raw 制品支持;具体 hosted/proxy/group 边界以兼容性矩阵为准。
- 面向用户、用户组、角色、仓库权限和 CI token 的完整权限管理。
- 兼容 Nexus 的客户端协议行为和 `/repository/<repo>/...` URL 布局。
- 使用 MySQL 存储元数据、用户数据、权限数据、token 数据、审计数据和迁移状态。
Expand Down Expand Up @@ -51,6 +51,7 @@
- `protocol-cargo`:Cargo sparse registry、publish、index 和 crate 行为。
- `protocol-pub`:Dart / Flutter Pub hosted repository 行为。
- `protocol-composer`:Composer 2 repository metadata、dist、hosted、proxy 和 group 行为。
- `protocol-terraform`:Terraform Module/Provider Registry path、metadata、版本、平台和 checksum 行为。
- `protocol-docker`:Docker Registry HTTP API V2 和 OCI 行为。
- `protocol-nuget`:NuGet 仓库行为。
- `protocol-rubygems`:RubyGems 仓库行为。
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ When a protocol change affects behavior that real package clients exercise direc
scripts/ci/run-live-compat.sh client-e2e
```

It covers Maven, npm, PyPI, Go resolve, Helm, Cargo/Rust, Dart/Pub, Flutter Pub, Composer/PHP, NuGet, RubyGems, Yum, and Docker/OCI client flows. Logs and downloaded metadata are written under `artifacts/client-e2e/`.
It covers Maven, npm, PyPI, Go resolve, Helm, Cargo/Rust, Dart/Pub, Flutter Pub, Composer/PHP, Terraform 0.13/current, NuGet, RubyGems, Yum, and Docker/OCI client flows. Logs and downloaded metadata are written under `artifacts/client-e2e/`; credential-bearing Terraform metadata is redacted before artifacts are uploaded.

See [compat-test/README.md](compat-test/README.md) for suite options, environment variables, and Maven properties.

Expand Down
21 changes: 11 additions & 10 deletions README.cn.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,11 @@

[English](README.md) | **中文**

kkRepo 是一款社区驱动、完全开源的自托管制品仓库,旨在解决 Sonatype Nexus 社区版的各类限制与痛点,为社区提供开放、可靠且可持续演进的制品管理方案。目前已支持 Maven、npm、PyPI、Go、Helm、Cargo/Rust、Dart/Pub、Composer/PHP、Docker/OCI、NuGet、RubyGems、Yum 和 Raw 等制品格式。
kkRepo 是一款社区驱动、完全开源的自托管制品仓库,旨在解决 Sonatype Nexus 社区版的各类限制与痛点,为社区提供开放、可靠且可持续演进的制品管理方案。目前已支持 Maven、npm、PyPI、Go、Helm、Cargo/Rust、Dart/Pub、Composer/PHP、Terraform、Docker/OCI、NuGet、RubyGems、Yum 和 Raw 等制品格式。

## 功能特性

- 支持 13+ 种主流仓库格式,覆盖 hosted、proxy 和 group 仓库管理。
- 支持 14+ 种主流仓库格式,覆盖 hosted、proxy 和 group 仓库管理。
- 兼容 Sonatype Nexus API 协议、用户权限模型、和 `/repository/<repo>/...` URL 布局。
- 可使用 kkRepo 平替 Sonatype Nexus,支持存量数据一键迁移并沿用原仓库域名和 URL,原有客户端配置与 CI 工作流无需改动。
- 支持完整的身份与访问控制,覆盖 Local、LDAP、OIDC 认证、匿名访问策略和细粒度权限管理。
Expand Down Expand Up @@ -74,13 +74,14 @@ curl -fsSL https://raw.githubusercontent.com/klboke/kkrepo/main/scripts/quicksta
| Cargo / Rust | hosted / proxy / group | 支持 `cargo publish`、yank/unyank、`CargoToken` 认证和 UI/API `.crate` 上传 | 支持 sparse index 和 `cargo search` | 支持 Cargo 仓库迁移 |
| Dart / Pub | hosted / proxy / group | 支持 `dart pub publish`、`dart pub get`、`flutter pub get`、`PubToken` 认证和 UI/API `.tar.gz` 上传 | 支持 package/version metadata、archive 属性和 Pub 搜索 | 支持 Nexus 3.92.0 Pub hosted 迁移,以及显式选择的 proxy cache 迁移 |
| Composer / PHP | hosted / proxy / group | Composer 没有标准 publish 命令;支持 Components API 和 UI 上传 zip/tar archive,并通过 Composer 2 安装 | 支持 package/version、dist、HTML View、Browse/Search 和 Usage | Nexus 原生 Composer proxy 配置可迁移;cache 仅在管理员显式选择且 source profile 证明后迁移 |
| Terraform Provider / Module Registry | hosted / proxy / group | 支持 Nexus 兼容 PUT 和 UI/API archive 上传;`terraform init` 可通过 group 解析 hosted 与 proxy module/provider | 支持 module/provider coordinate、version、platform、Browse/Search 和 Usage | 支持 Nexus Terraform hosted 数据和显式选择的 proxy archive cache 迁移;同时迁移 proxy/group 配置 |
| Docker / OCI | hosted / proxy / group | 支持 Registry V2 login、hosted push/pull、proxy pull、group pull、OCI referrers、cleanup 和 connector port 访问 | 支持 manifest/tag/blob metadata | Docker hosted 仓库数据迁移走 Nexus Repository Data 流程 |
| NuGet | hosted / proxy / group | 支持 package push 和管理台上传 | 支持 v3 service index / search | 默认迁移 hosted;proxy 可作为可选仓库迁移 |
| RubyGems | hosted / proxy / group | 支持 gem push/yank 和管理台上传 | 支持 | 默认迁移 hosted;proxy 可作为可选仓库迁移 |
| Yum | hosted / proxy / group | 支持 RPM 上传和管理台上传 | 支持 repodata | 默认迁移 hosted;proxy 可作为可选仓库迁移 |
| Raw | hosted / proxy / group | 支持 PUT 上传和管理台上传 | 支持 | 默认迁移 hosted;proxy 可作为可选仓库迁移 |

仓库数据迁移默认扫描 hosted 仓库;如需迁移源 Sonatype Nexus Repository 部署中作为历史备份或回源缓存使用的 proxy 仓库,可以在迁移页面的 `Optional proxy repositories` 中显式指定仓库名。Cargo / Rust、Dart / PubNexus 原生 Composer proxy cache 迁移已支持;Composer proxy 必须由管理员显式选择
仓库数据迁移默认扫描 hosted 仓库;如需迁移源 Sonatype Nexus Repository 部署中作为历史备份或回源缓存使用的 proxy 仓库,可以在迁移页面的 `Optional proxy repositories` 中显式指定仓库名。Cargo / Rust、Dart / PubNexus 原生 Composer 和 Terraform proxy cache 在显式选择且 source profile 接受时可迁移。Terraform 使用独立 proxy-cache writer 恢复 module/provider archive、保留 Nexus 公开 asset path,且不会把 cache asset 当作 hosted publication 重放。已恢复的 module archive 可不回源直接完成 module download discovery;Provider route 从已配置上游重建并校验 checksum/signature snapshot,其缓存 blob 在 metadata 有效期内保持固定

## 从 Sonatype Nexus Repository 迁移

Expand All @@ -99,7 +100,7 @@ curl -fsSL https://raw.githubusercontent.com/klboke/kkrepo/main/scripts/quicksta
| 维度 | Sonatype Nexus Repository OSS / Community Edition | kkRepo |
| --- | --- |-------------------------------------------------------------------------------------------------------------------|
| 产品定位 | 通用制品仓库管理平台,功能完整,覆盖大量官方格式和管理能力 | 提供面向迁移的客户端行为、权限模型和 `/repository/<repo>/...` URL 布局兼容,同时采用共享关系数据库、OSS/S3-first、适合多副本部署的架构 |
| 支持格式 | 官方支持格式更多,具体能力随版本和发行形态变化 | 聚焦常用制品格式,当前支持 Maven、npm、PyPI、Go、Helm、Cargo/Rust、Dart/Pub、Composer/PHP、Docker/OCI、NuGet、RubyGems、Yum 和 Raw;每个格式以独立 protocol 模块实现,便于按优先级扩展和验证 |
| 支持格式 | 官方支持格式更多,具体能力随版本和发行形态变化 | 聚焦常用制品格式,当前支持 Maven、npm、PyPI、Go、Helm、Cargo/Rust、Dart/Pub、Composer/PHP、Terraform、Docker/OCI、NuGet、RubyGems、Yum 和 Raw;每个格式以独立 protocol 模块实现,便于按优先级扩展和验证 |
| 使用限制 | Community Edition 面向个人和小团队,官方限制为最多 40,000 components、100,000 requests/day;超过阈值后会暂停新增 component,直到用量回到限制以下 | 不内置 Community Edition 这类版本授权用量限制;容量边界由所选关系数据库、OSS/S3、运行副本数和部署规格决定,适合按实际业务规模扩容 |
| 高可用部署 | 开源版适合单实例或基础 Kubernetes 部署;官方 HA deployment 属于 Pro 能力 | 从设计上默认支持多副本:session、认证 ticket、catalog 水位、锁、迁移进度和短生命周期协同状态都落 MySQL 或 PostgreSQL,进程内缓存只作为可重建热缓存 |
| 稳定性和升级 | 版本边界复杂:3.70.x 是最后支持 OrientDB 的版本;3.71.0 起新安装默认 H2,但 H2 仍是内嵌数据库;Community Edition 到 3.77.0+ 才支持免费使用外部 PostgreSQL;3.88.0 起搜索才完全改为 SQL、替代 Elasticsearch。旧版 OrientDB/Elasticsearch/本地数据目录组合升级窗口重,文件损坏后恢复高度依赖备份、修复任务和人工介入 | 运行时支持 MySQL/PostgreSQL,不依赖 OrientDB 和内嵌 Elasticsearch;核心状态在共享关系数据库,blob 在 OSS/S3/File blob store,缓存和索引均可重建,更适合滚动升级、故障切换和数据恢复 |
Expand All @@ -124,7 +125,7 @@ curl -fsSL https://raw.githubusercontent.com/klboke/kkrepo/main/scripts/quicksta

![前台仓库列表](docs/img/img_7.png)

按格式搜索组件,支持 Maven、npm、PyPI、Go、Helm、Cargo/Rust、Dart/Pub、Composer/PHP、Docker/OCI、NuGet、RubyGems、Yum 和 Raw 等仓库类型的制品检索。
按格式搜索组件,支持 Maven、npm、PyPI、Go、Helm、Cargo/Rust、Dart/Pub、Composer/PHP、Terraform、Docker/OCI、NuGet、RubyGems、Yum 和 Raw 等仓库类型的制品检索。

![前台制品搜索](docs/img/img.png)

Expand Down Expand Up @@ -170,14 +171,14 @@ AI agent 和贡献者的开发说明见 [AGENTS.md](AGENTS.md)。
2. ✅ Cargo / Rust - 仓库能力已完成,包含搜索、UI/API 上传和迁移能力([设计说明](docs/zh/dev/cargo-rust-repository-design.md))
3. ✅ Dart / Pub - 仓库能力已完成,包含 hosted/proxy/group、真实客户端 E2E、UI/API 上传、搜索和 Nexus 迁移能力([设计说明](docs/zh/dev/dart-pub-repository-design.md))
4. ✅ Composer / PHP - hosted、proxy、group、UI/API 上传、搜索、真实客户端 E2E、强制 Nexus live 对比和显式选择的 Nexus proxy cache 迁移 E2E 已实现([设计说明](docs/zh/dev/composer-php-repository-design.md))
5. ohpm / HarmonyOS - 规划中,覆盖 hosted、proxy、group、导入和管理端能力([设计说明](docs/zh/dev/ohpm-repository-design.md))
6. Swift Package Registry
7. APT / Debian
8. Terraform Provider / Module Registry - 规划中,覆盖 hosted、proxy、group、Provider GPG 签名、Nexus 路径兼容、真实 Terraform CLI E2E 和 Nexus 迁移([设计说明](docs/zh/dev/terraform-repository-design.md))
5. ✅ Terraform Provider / Module Registry - hosted、proxy、group、Provider GPG 签名、Nexus 路径兼容、UI/API 上传、搜索、真实 Terraform CLI E2E、Nexus hosted 数据迁移和显式选择的 proxy cache 迁移已实现([设计说明](docs/zh/dev/terraform-repository-design.md))
6. ohpm / HarmonyOS - 规划中,覆盖 hosted、proxy、group、导入和管理端能力([设计说明](docs/zh/dev/ohpm-repository-design.md))
7. Swift Package Registry
8. APT / Debian
9. Conan
10. Conda

用户和管理端 UI 已暴露的 token 类型包括协议专用 token(`NpmToken`、`CargoToken`、`PubToken`、`NuGetApiKey`、`RubyGemsApiKey`),以及面向 CI、脚本和自定义 HTTP 客户端的 `GenericToken`;`GenericToken` 适用于能够发送已配置 API-key header 或 bearer token 的调用方。
用户和管理端 UI 已暴露的 token 类型包括协议专用 token(`NpmToken`、`CargoToken`、`PubToken`、`NuGetApiKey`、`RubyGemsApiKey`),以及面向 Terraform 服务 URL、CI、脚本和自定义 HTTP 客户端的 `GenericToken`;`GenericToken` 适用于能够发送已配置 API-key header 或 bearer token 的调用方。

## 参与贡献

Expand Down
Loading
Loading