-
Notifications
You must be signed in to change notification settings - Fork 155
Add macOS Apple Silicon (aarch64-darwin) Habitat build & promote pipelines #475
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| [chef-vault] | ||
| build_targets = [ | ||
| "x86_64-linux", | ||
| "aarch64-linux", | ||
| "x86_64-windows" | ||
| ] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| --- | ||
| # Expeditor's built-in habitat/build does not support aarch64-darwin targets. | ||
| # This pipeline builds the aarch64-darwin (macOS Apple Silicon) habitat package | ||
| # and uploads it to the habitat builder (unstable channel). | ||
| # | ||
| # macOS builds run natively on Anka VMs (not Docker) because Habitat must | ||
| # produce a native aarch64-darwin .hart package. | ||
|
|
||
| expeditor: | ||
| defaults: | ||
| buildkite: | ||
| timeout_in_minutes: 60 | ||
| retry: | ||
| automatic: | ||
| limit: 3 | ||
| secrets: | ||
| HAB_AUTH_TOKEN: | ||
| path: account/static/habitat/chef-ci | ||
| field: auth_token | ||
|
|
||
| steps: | ||
|
|
||
| - label: ":mac: Build aarch64-darwin habitat package" | ||
| agents: | ||
| queue: 'default-macos-arm64-privileged' | ||
| plugins: | ||
| - chef/anka#v0.7.2: | ||
| vm-name: buildkite-core-14-arm64 | ||
| always-pull: true | ||
| wait-network: true | ||
| inherit-environment-vars: true | ||
| env: | ||
| BUILD_PKG_TARGET: "aarch64-darwin" | ||
| HOMEBREW_NO_INSTALL_CLEANUP: 1 | ||
| HOMEBREW_NO_AUTO_UPDATE: 1 | ||
| commands: | ||
| - ". vault-util-init" | ||
| - "sudo -E ./.expeditor/buildkite/build_hab_aarch64_darwin.sh" | ||
| - "./.expeditor/buildkite/upload_hab_aarch64_darwin.sh" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| # Builds the macOS Apple Silicon (aarch64-darwin) chef-vault habitat package. | ||
| # Expeditor's built-in habitat/build pipeline does not support aarch64-darwin | ||
| # targets, so this script handles the build on a native macOS ARM Anka VM. | ||
| # | ||
| # Prerequisites (provided by the Anka VM image): | ||
| # - macOS on Apple Silicon (ARM64) | ||
| # - Xcode Command Line Tools (provides git) | ||
| # - Homebrew (for any additional dependencies) | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| export HAB_ORIGIN='chef' | ||
| export PLAN='chef-vault' | ||
| export CHEF_LICENSE="accept-no-persist" | ||
| export HAB_LICENSE="accept-no-persist" | ||
| export HAB_NONINTERACTIVE="true" | ||
| export HAB_BLDR_CHANNEL="base-2025" | ||
| export HAB_REFRESH_CHANNEL="base-2025" | ||
|
|
||
| echo "--- :git: Checking for git" | ||
| if ! command -v git &> /dev/null; then | ||
| echo "Git is not installed. Installing via Xcode Command Line Tools..." | ||
| xcode-select --install 2>/dev/null || true | ||
| # Wait for installation to complete | ||
| until command -v git &> /dev/null; do sleep 5; done | ||
| else | ||
| echo "Git is already installed." | ||
| git --version | ||
| fi | ||
|
|
||
| echo "--- :mac: Installing Habitat" | ||
| curl https://raw.githubusercontent.com/habitat-sh/habitat/main/components/hab/install.sh | sudo bash -s -- -t aarch64-darwin | ||
|
|
||
| hab_binary="/usr/local/bin/hab" | ||
| echo "--- :habicat: Hab binary: ${hab_binary} (version: $(${hab_binary} --version))" | ||
|
|
||
| echo "--- :key: Downloading origin keys" | ||
| ${hab_binary} origin key download "$HAB_ORIGIN" | ||
| ${hab_binary} origin key download "$HAB_ORIGIN" --secret | ||
|
|
||
| # Copy keys to system hab cache so they're available for root builds | ||
| sudo mkdir -p /hab/cache/keys | ||
| sudo cp -r ~/.hab/cache/keys/* /hab/cache/keys/ 2>/dev/null || true | ||
|
|
||
| echo "--- :construction: Building $PLAN aarch64-darwin package" | ||
| ${hab_binary} pkg build . --refresh-channel base-2025 | ||
|
|
||
| project_root="$(pwd)" | ||
| source "${project_root}/results/last_build.env" || { echo "ERROR: unable to determine build details"; exit 1; } | ||
|
|
||
| echo "--- :package: Uploading artifact to Buildkite" | ||
| cd "${project_root}/results" | ||
| buildkite-agent artifact upload "$pkg_artifact" || { echo "ERROR: unable to upload artifact"; exit 1; } | ||
|
|
||
| echo "--- Setting CHEF_VAULT_HAB_ARTIFACT_DARWIN_AARCH64 metadata for buildkite agent" | ||
| buildkite-agent meta-data set "CHEF_VAULT_HAB_ARTIFACT_DARWIN_AARCH64" "$pkg_artifact" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| #!/bin/bash | ||
|
|
||
| # Promotes the aarch64-darwin (macOS Apple Silicon) chef-vault package between | ||
| # habitat channels. Expeditor's built-in promote_habitat_packages does not | ||
| # support aarch64-darwin targets, so this script handles it manually at each | ||
| # promotion stage. | ||
| # | ||
| # Context is auto-detected from EXPEDITOR_ environment variables: | ||
| # - project_promoted: uses EXPEDITOR_SOURCE_CHANNEL → EXPEDITOR_TARGET_CHANNEL | ||
| # - buildkite_hab_build_group_published: defaults to unstable → current | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| PKG_ORIGIN="chef" | ||
| PKG_NAME="chef-vault" | ||
| PKG_TARGET="aarch64-darwin" | ||
|
|
||
| export HAB_LICENSE="accept-no-persist" | ||
| export HAB_NONINTERACTIVE="true" | ||
|
|
||
| # Determine the package version from Expeditor environment variables. | ||
| # For buildkite_hab_build_group_published, the aarch64 build is a separate | ||
| # pipeline (hab_aarch64/build) that runs in parallel with habitat/build. | ||
| # Both build from the same git commit so they produce the same version. | ||
| # The aarch64 target is NOT in .bldr.toml so it's absent from pkg_idents; | ||
| # we extract the version from the x86_64-linux ident instead. | ||
| # Expeditor flattens Hash metadata keys by appending with "_" and stripping | ||
| # non-word chars (\W), then uppercases the key, so: | ||
| # pkg_idents["chef-vault-x86_64-linux"] | ||
| # -> EXPEDITOR_PKG_IDENTS_CHEFVAULTX86_64LINUX | ||
| PKG_VERSION="${EXPEDITOR_PKG_VERSION:-${EXPEDITOR_PROMOTABLE:-}}" | ||
| if [[ -z "$PKG_VERSION" && -n "${EXPEDITOR_PKG_IDENTS_CHEFVAULTX86_64LINUX:-}" ]]; then | ||
| PKG_VERSION=$(echo "${EXPEDITOR_PKG_IDENTS_CHEFVAULTX86_64LINUX}" | cut -d'/' -f3) | ||
| fi | ||
|
|
||
| # Determine source and target channels based on Expeditor workload context | ||
| if [[ -n "${EXPEDITOR_TARGET_CHANNEL:-}" ]]; then | ||
| # project_promoted workload | ||
| SOURCE_CHANNEL="${EXPEDITOR_SOURCE_CHANNEL}" | ||
| TARGET_CHANNEL="${EXPEDITOR_TARGET_CHANNEL}" | ||
| else | ||
| # buildkite_hab_build_group_published workload | ||
| SOURCE_CHANNEL="unstable" | ||
| TARGET_CHANNEL="current" | ||
| fi | ||
|
|
||
| echo "--- Promoting ${PKG_ORIGIN}/${PKG_NAME} (${PKG_TARGET}) from ${SOURCE_CHANNEL} to ${TARGET_CHANNEL}" | ||
|
|
||
| # Use HAB_AUTH_TOKEN from the pipeline secret if available, otherwise fetch from vault | ||
| if [[ -z "${HAB_AUTH_TOKEN:-}" ]]; then | ||
| HAB_AUTH_TOKEN=$(vault kv get -field auth_token account/static/habitat/chef-ci) | ||
| export HAB_AUTH_TOKEN | ||
| fi | ||
|
|
||
| # Find the exact aarch64 package ident for this version | ||
| if [[ -n "$PKG_VERSION" ]]; then | ||
| echo "--- Looking up ${PKG_TARGET} package for version ${PKG_VERSION}" | ||
| PKG_IDENT=$(curl -sf "https://bldr.habitat.sh/v1/depot/pkgs/${PKG_ORIGIN}/${PKG_NAME}/${PKG_VERSION}/latest?target=${PKG_TARGET}" | jq -r '.ident_array | join("/")') | ||
| else | ||
| echo "WARNING: No version info available. Skipping aarch64 promotion." | ||
| exit 0 | ||
| fi | ||
|
|
||
| if [[ -z "$PKG_IDENT" || "$PKG_IDENT" == "null" ]]; then | ||
| echo "WARNING: No ${PKG_TARGET} package found for version ${PKG_VERSION}. Skipping promotion." | ||
| exit 0 | ||
| fi | ||
|
|
||
| echo "--- Found package: ${PKG_IDENT}" | ||
| echo "--- Promoting ${PKG_IDENT} to ${TARGET_CHANNEL} channel" | ||
|
|
||
| hab pkg promote "${PKG_IDENT}" "${TARGET_CHANNEL}" "${PKG_TARGET}" | ||
|
|
||
| echo "--- Successfully promoted ${PKG_IDENT} (${PKG_TARGET}) to ${TARGET_CHANNEL}" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| # Uploads the aarch64-darwin (macOS Apple Silicon) habitat package to the | ||
| # habitat builder. This runs on the same macOS Anka VM after the build step. | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| export HAB_ORIGIN='chef' | ||
| export CHEF_LICENSE="accept-no-persist" | ||
| export HAB_LICENSE="accept-no-persist" | ||
| export HAB_NONINTERACTIVE="true" | ||
|
|
||
| error () { | ||
| local message="$1" | ||
| echo -e "\nERROR: ${message}\n" >&2 | ||
| exit 1 | ||
| } | ||
|
|
||
| echo "--- Downloading aarch64-darwin package artifact" | ||
| PKG_ARTIFACT=$(buildkite-agent meta-data get "CHEF_VAULT_HAB_ARTIFACT_DARWIN_AARCH64") | ||
| buildkite-agent artifact download "$PKG_ARTIFACT" . || error 'unable to download aarch64-darwin artifact' | ||
|
|
||
| echo "--- :habicat: Uploading aarch64-darwin package to habitat builder (unstable channel)" | ||
| hab pkg upload "$PKG_ARTIFACT" --auth "$HAB_AUTH_TOKEN" --channel unstable || error 'unable to upload aarch64-darwin package to habitat builder' |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| --- | ||
| # Pipeline to promote the aarch64-darwin (macOS Apple Silicon) chef-vault | ||
| # habitat package between channels. Expeditor's built-in | ||
| # promote_habitat_packages does not support aarch64-darwin targets, so this | ||
| # pipeline handles it manually. | ||
| # | ||
| # The promote_hab_aarch64_darwin.sh script auto-detects source/target channels | ||
| # from EXPEDITOR_ environment variables set by the triggering workload. | ||
|
|
||
| expeditor: | ||
| defaults: | ||
| buildkite: | ||
| timeout_in_minutes: 10 | ||
| retry: | ||
| automatic: | ||
| limit: 1 | ||
|
|
||
| steps: | ||
|
|
||
| - label: ":mac: Promote aarch64-darwin habitat package" | ||
| command: | ||
| - ./.expeditor/buildkite/promote_hab_aarch64_darwin.sh | ||
| expeditor: | ||
| secrets: | ||
| HAB_AUTH_TOKEN: | ||
| path: account/static/habitat/chef-ci | ||
| field: auth_token |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe pipeline definition is missing in config.yml. See https://expeditor.chef.io/docs/pipelines/habitat/#create-pipeline