Skip to content

ci: compile shim-rs in gitlab ci and gha#53

Open
TheRayquaza wants to merge 6 commits intodatadogfrom
mateo.lelong/datadog-ci-rust-shim
Open

ci: compile shim-rs in gitlab ci and gha#53
TheRayquaza wants to merge 6 commits intodatadogfrom
mateo.lelong/datadog-ci-rust-shim

Conversation

@TheRayquaza
Copy link
Copy Markdown

@TheRayquaza TheRayquaza commented Apr 8, 2026

Summary

Adds CI support for compiling and distributing the Rust containerd shim (`containerd-shim-kata-v2-rs`) as a static musl binary alongside the existing Go shim, across all CI systems.

GitLab CI (.gitlab-ci.yml):

  • New build-shim-rust-amd64 and build-shim-rust-arm64 jobs using registry.ddbuild.io/images/mirror/rust:1.91-alpine with musl cross-compilation toolchains
  • publish-artifacts updated to upload the Rust shim to S3 at s3://kata-containers-ci-artifacts/$TAG/{amd64,arm64}/containerd-shim-kata-v2

GHA static tarball workflows (build-kata-static-tarball-{amd64,arm64}.yaml):

  • New build-asset-shim-rs job compiles and uploads the Rust shim as a GHA artifact
  • create-kata-tarball bundles inside the release tarball

GHA release workflow (build-kata-os.yml):

  • New "Build containerd-shim-kata-v2-rs" step compiles and bundles the Rust shim into artifacts-{amd64,arm64}.zip

Testing

The build was validated on run #24140664525 (both amd64 and arm64).

To verify the Rust shim is present in the release zip:

gh run download 24140664525 --repo DataDog/kata-containers -n artifacts-amd64 -D /tmp/kata-amd64
python3 -c "import zipfile; [print(n) for n in zipfile.ZipFile('/tmp/kata-amd64/artifacts-amd64.zip').namelist()]"
# Expected output includes: containerd-shim-kata-v2-rs
gh run download 24140664525 --repo DataDog/kata-containers -n artifacts-arm64 -D /tmp/kata-arm64
python3 -c "import zipfile; [print(n) for n in zipfile.ZipFile('/tmp/kata-arm64/artifacts-arm64.zip').namelist()]"
# Expected output includes: containerd-shim-kata-v2-rs

@TheRayquaza TheRayquaza marked this pull request as ready for review April 8, 2026 12:59
@TheRayquaza TheRayquaza marked this pull request as draft April 8, 2026 13:40
@TheRayquaza TheRayquaza marked this pull request as ready for review April 8, 2026 14:51
Add CI support for compiling and distributing the Rust containerd shim
(containerd-shim-kata-v2-rs) as a static musl binary across all CI systems.

GitLab CI (.gitlab-ci.yml):
- Add build-shim-rust-amd64/arm64 jobs using the internal mirror registry
  (registry.ddbuild.io/images/mirror/rust:1.91-alpine) with musl cross-
  compilation toolchains
- Update publish-artifacts to upload the Rust shim to S3 alongside the Go shim

GHA static tarball workflows (build-kata-static-tarball-{amd64,arm64}.yaml):
- Add build-asset-shim-rs job that builds the Rust shim and uploads it as a
  GHA artifact
- Update create-kata-tarball to include the Rust shim at
  opt/kata/runtime-rs/bin/containerd-shim-kata-v2 inside the release tarball
- Install cmake, required by libz-sys for musl cross-compilation

GHA release workflow (build-kata-os.yml):
- Add "Build containerd-shim-kata-v2-rs" step and bundle it into
  artifacts-{amd64,arm64}.zip alongside the kernel, rootfs, and Go shim
- Reclaim workspace ownership with `sudo chown -R ... .` before building:
  the self-hosted arm64 runner reuses its workspace between runs and previous
  Docker steps leave root-owned files that block protobuf codegen
- Install cmake and create the musl C++ symlink for cross-compilation

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@TheRayquaza TheRayquaza force-pushed the mateo.lelong/datadog-ci-rust-shim branch from b8be637 to 62d84ff Compare April 8, 2026 14:56
Comment thread .gitlab-ci.yml Outdated
Comment thread .github/workflows/build-kata-os.yml Outdated
Comment on lines +29 to +32
- name: Install Rust toolchain
run: |
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain 1.91
echo "$HOME/.cargo/bin" >> $GITHUB_PATH
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if there exists github actions for that.

Comment thread .github/workflows/build-kata-os.yml Outdated
Comment thread .github/workflows/build-kata-os.yml Outdated
Comment thread .github/workflows/build-kata-static-tarball-arm64.yaml Outdated
Comment thread .github/workflows/build-kata-static-tarball-amd64.yaml Outdated
@TheRayquaza TheRayquaza force-pushed the mateo.lelong/datadog-ci-rust-shim branch from d7bb3ff to 3299c43 Compare April 10, 2026 09:04
Comment thread .github/workflows/build-kata-os.yml Outdated
- uses: actions/setup-go@v5
with:
go-version: ">=1.24.0"
- uses: dtolnay/rust-toolchain@1.91
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any idea why they don't need it in the kata upstream github action?

Comment thread .github/workflows/build-kata-os.yml Outdated
- uses: dtolnay/rust-toolchain@1.91
- name: Install dependencies
run: sudo apt-get update && sudo apt-get install -y libelf-dev flex bison libssl-dev pahole
run: sudo apt-get update && sudo apt-get install -y libelf-dev flex bison libssl-dev pahole musl-tools g++ cmake
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you still need g++ and cmake after you changes?

Comment on lines +44 to +48
rustup target add ${{ matrix.musl_arch }}-unknown-linux-musl
cd src/runtime-rs
make clean-generated-files
make PREFIX=/opt/kata
cp ../../target/${{ matrix.musl_arch }}-unknown-linux-musl/release/containerd-shim-kata-v2 /tmp/containerd-shim-kata-v2-rs
Copy link
Copy Markdown

@zaymat zaymat Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is it done upstream. I'm under the impression they generate both shims in one make command.

@TheRayquaza TheRayquaza force-pushed the mateo.lelong/datadog-ci-rust-shim branch 2 times, most recently from 2d85b94 to 3299c43 Compare April 17, 2026 15:37
@TheRayquaza
Copy link
Copy Markdown
Author

TheRayquaza commented Apr 17, 2026

The action dtolnay/rust-toolchain@1.91 is not allowed in DataDog/kata-containers because all actions must be from a repository owned by your enterprise or created by GitHub.

actually that s an issue, going back to manual download of rust using upstream script

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants