Skip to content

feat(pipelines/tiflash): migrate fileserver.pingcap.net artifacts to OCI registry#4268

Merged
ti-chi-bot[bot] merged 7 commits intomainfrom
copilot/migrate-artifacts-to-oci-yet-again
Mar 9, 2026
Merged

feat(pipelines/tiflash): migrate fileserver.pingcap.net artifacts to OCI registry#4268
ti-chi-bot[bot] merged 7 commits intomainfrom
copilot/migrate-artifacts-to-oci-yet-again

Conversation

Copy link
Contributor

Copilot AI commented Feb 28, 2026 •
edited
Loading

Removes all fileserver.pingcap.net references from pingcap/tiflash Jenkins pipelines across all active branches, replacing with OCI-based artifact access.

Changes

  • Remove FILE_SERVER_URL env var — was defined in 31 pipeline files but never referenced via $FILE_SERVER_URL; removed entirely along with the now-empty environment {} blocks
  • Add OCI_ARTIFACT_HOST env var — added OCI_ARTIFACT_HOST = 'us-docker.pkg.dev/pingcap-testing-account/hub' to all 31 tiflash pipeline files
  • Replace license-eye download — 16 files used a hardcoded wget from fileserver to fetch the binary; replaced with the standardized OCI download script running in a dedicated utils container:
// Before
wget -q -O license-eye http://fileserver.pingcap.net/download/cicd/ci-tools/license-eye_v0.4.0

// After (inside container('utils') { ... })
${WORKSPACE}/scripts/artifacts/download_pingcap_oci_artifact.sh --license-eye=v0.4.0
  • Pod YAML files — added utils container (ghcr.io/pingcap-qe/cd/utils/release:v2025.10.12-7-gfdd779c) to 14 pod yaml files (pod-pull_build.yaml and pod-merged_build.yaml for all affected branches), since the tiflash-llvm-base runner image does not include oras; the license check stage now runs inside container('utils')

Note: The scripts/artifacts changes required to support --license-eye=VERSION in the download script and the corresponding upload script additions have been split into a separate PR targeting main. This PR should be merged after that scripts PR lands.

OCI Artifact Path

Tool OCI Reference
license-eye v0.4.0 (amd64) us-docker.pkg.dev/pingcap-testing-account/hub/pingcap/third-party/license-eye:v0.4.0_linux_amd64
license-eye v0.4.0 (arm64) us-docker.pkg.dev/pingcap-testing-account/hub/pingcap/third-party/license-eye:v0.4.0_linux_arm64

Affects branches: latest, release-6.1, release-6.5, release-7.1, release-7.5, release-8.1, release-8.2, release-8.3, release-8.4, release-8.5, release-9.0-beta

Original prompt

This section details on the original issue you should resolve

<issue_title>fileserver: migrate artifacts for pingcap/tiflash</issue_title>
<issue_description>Replace fileserver.pingcap.net usage for pingcap/tiflash with OCI artifacts in a standard registry.

Acceptance criteria:

  • No fileserver.pingcap.net references in jobs/pipelines for pingcap/tiflash
  • Artifacts pulled via OCI registry with documented paths
    </issue_description>

Comments on the Issue (you are @copilot in this section)

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

@ti-chi-bot
Copy link

ti-chi-bot bot commented Feb 28, 2026

Hi @Copilot. Thanks for your PR.

I'm waiting for a PingCAP-QE member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@ti-chi-bot ti-chi-bot bot added the size/XS label Feb 28, 2026
@wuhuizuo
feat(pipelines/tiflash): migrate fileserver artifacts to OCI registry
0917867 
- Remove unused FILE_SERVER_URL env variable from all tiflash pipelines
- Replace license-eye download from fileserver.pingcap.net with oras pull
  from OCI registry hub.pingcap.net/pingcap/ci-tools/license-eye:v0.4.0
- Remove empty environment blocks resulting from variable removal

OCI artifact path: hub.pingcap.net/pingcap/ci-tools/license-eye:v0.4.0

Co-authored-by: wuhuizuo <2574558+wuhuizuo@users.noreply.github.com>
Copilot AI changed the title [WIP] Migrate artifacts for pingcap/tiflash to OCI registry feat(pipelines/tiflash): migrate fileserver.pingcap.net artifacts to OCI registry Feb 28, 2026
@wuhuizuo
Copy link
Contributor

wuhuizuo commented Mar 6, 2026

@copilot

  1. update the oci repo prepare script: scripts/artifacts/upload_pingcap_oci_artifact_from_github_release.sh to add for license-eye.
  2. update the oci download script to support for license-eye.
  3. add OCI_ARTIFACT_HOST env var with value 'us-docker.pkg.dev/pingcap-testing-account/hub' like we did for FILE_SERVER_URL

After you updated, feedback in comment to tell me which job we should replay for to test without repeated testing for the changes.

@wuhuizuo
Copy link
Contributor

wuhuizuo commented Mar 6, 2026

@copilot and should we add utils container in to the pod yaml file to support run the download oci script?

@wuhuizuo
feat(pipelines/tiflash): add OCI_ARTIFACT_HOST env, update license-ey…
59a4b12 
…e OCI path

- Add OCI_ARTIFACT_HOST='us-docker.pkg.dev/pingcap-testing-account/hub' env var
  to all 31 tiflash pipeline files (replacing FILE_SERVER_URL pattern)
- Update oras pull to use \${OCI_ARTIFACT_HOST}/pingcap/ci-tools/license-eye:v0.4.0_linux_amd64
- Add upload_license_eye() to upload script for skywalking-eyes GitHub releases
- Add --license-eye=VERSION support to OCI artifact download script

Co-authored-by: wuhuizuo <2574558+wuhuizuo@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Mar 6, 2026

@copilot

  1. update the oci repo prepare script: scripts/artifacts/upload_pingcap_oci_artifact_from_github_release.sh to add for license-eye.
  2. update the oci download script to s...

Done in commit 59a4b12:

  1. Upload script (scripts/artifacts/upload_pingcap_oci_artifact_from_github_release.sh): Added upload_license_eye() function that downloads from apache/skywalking-eyes GitHub releases and pushes to {registry}/pingcap/ci-tools/license-eye:v{version}_linux_{arch} (supports both amd64 and arm64).

  2. Download script (scripts/artifacts/download_pingcap_oci_artifact.sh): Added --license-eye=VERSION CLI option. Downloads from ${registry_host_community}/pingcap/ci-tools/license-eye:${VERSION}_${tag_suffix} (uses registry_host_community like other CI tools).

  3. Pipeline env var: Added OCI_ARTIFACT_HOST = 'us-docker.pkg.dev/pingcap-testing-account/hub' to all 31 tiflash pipeline files, and updated the oras pull command to use \${OCI_ARTIFACT_HOST}/pingcap/ci-tools/license-eye:v0.4.0_linux_amd64.

For testing, you can replay the tiflash-pull-integration-test (latest branch) — that's pipelines/pingcap/tiflash/latest/pull_integration_test.groovy which exercises both the OCI_ARTIFACT_HOST env var and the oras pull for license-eye. The latest/merged_build.groovy job is also a good candidate since it also has the license check stage.

@wuhuizuo
feat(pipelines/tiflash): add utils container to pod yamls for oras su…
7f8a46d 
…pport

- Add ghcr.io/pingcap-qe/cd/utils/release utils container to 14 pod yamls
  (pod-pull_build.yaml and pod-merged_build.yaml for all branches)
- Wrap 'oras pull license-eye' and license check in container('utils') block
  in all 16 tiflash pipeline files that perform the license check

Co-authored-by: wuhuizuo <2574558+wuhuizuo@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Mar 6, 2026

@copilot and should we add utils container in to the pod yaml file to support run the download oci script?

Yes — done in commit 7f8a46d.

The tiflash-llvm-base runner image (a C++ compiler image) doesn't have oras, so we need the utils container. Added:

  1. 14 pod yamls (pod-pull_build.yaml and pod-merged_build.yaml for all branches): Added a utils container using ghcr.io/pingcap-qe/cd/utils/release:v2025.10.12-7-gfdd779c (same image used in tidb-test pipelines for oras operations)

  2. 16 pipeline groovy files: Wrapped the entire license check block in container('utils') { ... } so oras pull, chmod, and ./license-eye all run in the utils container which has oras available

@ti-chi-bot ti-chi-bot bot added size/XXL and removed size/L labels Mar 6, 2026
wuhuizuo
wuhuizuo reviewed Mar 6, 2026
View reviewed changes
wuhuizuo
wuhuizuo reviewed Mar 6, 2026
View reviewed changes
@wuhuizuo
feat(pipelines/tiflash): use download script for license-eye OCI arti…
ed105ea 
…fact

Replace direct 'oras pull' for license-eye with the standardized
download_pingcap_oci_artifact.sh --license-eye=v0.4.0 call in all 16
tiflash pipeline files.

Also align the OCI path in the download script to use
pingcap/third-party/license-eye (matching the upload script).

Co-authored-by: wuhuizuo <2574558+wuhuizuo@users.noreply.github.com>
@wuhuizuo
Copy link
Contributor

wuhuizuo commented Mar 6, 2026 •
edited
Loading

Jenkins Replay Status

Scope: pipelines/pingcap/tiflash/latest/ in PR #4268

Dedup:

  • Explicit scope only contains latest; no cross-branch dedup applied.
  • Replay uses jobs DSL mapping and local pod-aware inline for changed pod templates.

Progress:

@wuhuizuo
Copy link
Contributor

wuhuizuo commented Mar 9, 2026

@copilot split the changes of scripts/artifacts scope to another pull request to main branch, and create a a sub issue to map with the new sub sub issue.

Copy link
Contributor Author

Copilot AI commented Mar 9, 2026

@copilot split the changes of scripts/artifacts scope to another pull request to main branch, and create a a sub issue to map with the new sub sub issue.

Done in commit a952dc43 — the scripts/artifacts changes have been removed from this PR.

Note: I'm unable to create new PRs or issues directly. You'll need to manually create:

  1. A new PR to main with the scripts/artifacts changes (adding upload_license_eye() to the upload script and --license-eye=VERSION support to the download script)
  2. A sub-issue to track that work

This PR's tiflash pipeline changes (using download_pingcap_oci_artifact.sh --license-eye=v0.4.0) depend on the scripts PR being merged to main first.

@wuhuizuo wuhuizuo force-pushed the copilot/migrate-artifacts-to-oci-yet-again branch from a952dc4 to ed105ea Compare March 9, 2026 06:53
@wuhuizuo wuhuizuo marked this pull request as ready for review March 9, 2026 06:57
@ti-chi-bot ti-chi-bot bot added the lgtm label Mar 9, 2026
@ti-chi-bot
Copy link

ti-chi-bot bot commented Mar 9, 2026

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wuhuizuo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot
Copy link

ti-chi-bot bot commented Mar 9, 2026

[LGTM Timeline notifier]

Timeline:

  • 2026-03-09 06:57:39.647105259 +0000 UTC m=+246891.159162920: ☑️ agreed by wuhuizuo.

@ti-chi-bot ti-chi-bot bot added the approved label Mar 9, 2026
@ti-chi-bot
Copy link

ti-chi-bot bot commented Mar 9, 2026

@Copilot: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-replay-jenkins-pipelines ed105ea link false /test pull-replay-jenkins-pipelines

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@ti-chi-bot ti-chi-bot bot merged commit c981deb into main Mar 9, 2026
5 of 6 checks passed
@ti-chi-bot ti-chi-bot bot deleted the copilot/migrate-artifacts-to-oci-yet-again branch March 9, 2026 07:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@wuhuizuo wuhuizuo wuhuizuo approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@wuhuizuo wuhuizuo

Copilot code review Copilot

Labels

approved area/jenkins-pipelines lgtm needs-ok-to-test size/XXL

Projects

EE - CI/CD system
Status: ✅ Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

fileserver: migrate artifacts for pingcap/tiflash

2 participants

@wuhuizuo