Skip to content

Optimize integration test runtime #689

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
davidgamero merged 1 commit into from
Feb 11, 2026
Merged

Optimize integration test runtime #689

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
143 changes: 107 additions & 36 deletions .github/workflows/integration-per-language.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,15 +103,21 @@ jobs:
helm-version: "v3.19.2"
releaseName: "test-release"
id: bake
- name: Set up buildx
run: |
docker buildx create --use --name draft-builder || docker buildx use draft-builder
docker buildx inspect --bootstrap
- name: Build and Push image
run: |
export SHELL=/bin/bash
eval $(minikube -p minikube docker-env)
Comment on lines +106 to 113
Copy link

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docker buildx create --use is executed before eval $(minikube ... docker-env), so the builder may be created against the runner’s default Docker daemon instead of the Minikube daemon. That can cause the subsequent docker buildx build (after switching DOCKER_HOST) to fail to find the builder or to build/push from the wrong daemon. Consider moving buildx setup into the same step after the Minikube docker-env call (or explicitly creating/using the builder after switching to Minikube’s Docker endpoint).

Suggested change
- name: Set up buildx
run: |
docker buildx create --use --name draft-builder || docker buildx use draft-builder
docker buildx inspect --bootstrap
- name: Build and Push image
run: |
export SHELL=/bin/bash
eval $(minikube -p minikube docker-env)
- name: Build and Push image
run: |
export SHELL=/bin/bash
eval $(minikube -p minikube docker-env)
docker buildx create --use --name draft-builder || docker buildx use draft-builder
docker buildx inspect --bootstrap

Copilot uses AI. Check for mistakes.
docker build -f ./langtest/Dockerfile -t testapp ./langtest/
docker tag testapp host.minikube.internal:5001/testapp
echo -n "verifying images:"
docker images
docker push host.minikube.internal:5001/testapp
docker buildx build \
--cache-from=type=gha,scope=${{ inputs.language }}-helm \
--cache-to=type=gha,scope=${{ inputs.language }}-helm,mode=max \
-f ./langtest/Dockerfile \
-t host.minikube.internal:5001/testapp \
--push \
./langtest/
echo 'Curling host.minikube.internal test app images from minikube'
minikube ssh "curl http://host.minikube.internal:5001/v2/testapp/tags/list"
# Deploys application based on manifest files from previous step
Expand Down Expand Up @@ -146,14 +152,29 @@ jobs:
kubectl get svc
echo 'Starting minikube tunnel'
minikube tunnel > /dev/null 2>&1 & tunnelPID=$!
sleep 120
trap 'kill $tunnelPID' EXIT
Copy link

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

trap 'kill $tunnelPID' EXIT can make the step fail if minikube tunnel exits early or the PID is already gone, because kill will return non-zero under the default bash -e runner shell. Make the cleanup non-fatal (e.g., ignore errors / redirect stderr) so a successful curl doesn’t get marked as failed during teardown.

Suggested change
trap 'kill $tunnelPID' EXIT
trap 'kill "$tunnelPID" >/dev/null 2>&1 || true' EXIT

Copilot uses AI. Check for mistakes.
echo 'Waiting for service IP'
for i in {1..30}; do
SERVICEIP=$(kubectl get svc -o jsonpath={'.items[1].status.loadBalancer.ingress[0].ip'})
Copy link

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Polling the service IP via kubectl get svc ... .items[1]... is brittle because it depends on list ordering and the presence of at least two services. Prefer selecting the specific Service by name (e.g., the release service) or by label selector, then JSONPath into that object.

Suggested change
SERVICEIP=$(kubectl get svc -o jsonpath={'.items[1].status.loadBalancer.ingress[0].ip'})
SERVICEIP=$(kubectl get svc test-release-testapp -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

Copilot uses AI. Check for mistakes.
if [ -n "$SERVICEIP" ]; then
break
fi
sleep 4
done
kubectl get svc
SERVICEIP=$(kubectl get svc -o jsonpath={'.items[1].status.loadBalancer.ingress[0].ip'})
echo "SERVICEIP: $SERVICEIP"
echo 'Curling service IP'
curl -m 3 $SERVICEIP:${{env.serviceport}}
sleep 5
kill $tunnelPID
if [ -z "$SERVICEIP" ]; then
echo 'Service IP not ready'
exit 1
fi
echo 'Curling service IP with retries'
for i in {1..20}; do
if curl -m 3 "http://$SERVICEIP:${{env.serviceport}}"; then
exit 0
fi
sleep 3
done
exit 1
- run: |
./draft -v generate-workflow \
-d ./langtest/ \
Expand Down Expand Up @@ -282,16 +303,22 @@ jobs:
renderEngine: "kustomize"
kustomizationPath: ./langtest/base
kubectl-version: "latest"
- name: Set up buildx
run: |
docker buildx create --use --name draft-builder || docker buildx use draft-builder
docker buildx inspect --bootstrap
- name: Build and Push Image
continue-on-error: true
run: |
export SHELL=/bin/bash
eval $(minikube -p minikube docker-env)
Comment on lines +306 to 314
Copy link

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as earlier: buildx is set up before eval $(minikube ... docker-env), which risks creating/using the builder against the wrong Docker daemon. Move buildx setup to after switching to Minikube’s Docker endpoint (or re-create/use the builder after switching).

Suggested change
- name: Set up buildx
run: |
docker buildx create --use --name draft-builder || docker buildx use draft-builder
docker buildx inspect --bootstrap
- name: Build and Push Image
continue-on-error: true
run: |
export SHELL=/bin/bash
eval $(minikube -p minikube docker-env)
- name: Build and Push Image
continue-on-error: true
run: |
export SHELL=/bin/bash
eval $(minikube -p minikube docker-env)
docker buildx create --use --name draft-builder || docker buildx use draft-builder
docker buildx inspect --bootstrap

Copilot uses AI. Check for mistakes.
docker build -f ./langtest/Dockerfile -t testapp ./langtest/
docker tag testapp host.minikube.internal:5001/testapp
echo -n "verifying images:"
docker images
docker push host.minikube.internal:5001/testapp
docker buildx build \
--cache-from=type=gha,scope=${{ inputs.language }}-kustomize \
--cache-to=type=gha,scope=${{ inputs.language }}-kustomize,mode=max \
-f ./langtest/Dockerfile \
-t host.minikube.internal:5001/testapp \
--push \
./langtest/
echo 'Curling host.minikube.internal test app images from minikube'
minikube ssh "curl http://host.minikube.internal:5001/v2/testapp/tags/list"
# Deploys application based on manifest files from previous step
Expand Down Expand Up @@ -326,14 +353,29 @@ jobs:
kubectl get svc
echo 'Starting minikube tunnel'
minikube tunnel > /dev/null 2>&1 & tunnelPID=$!
sleep 120
trap 'kill $tunnelPID' EXIT
Copy link

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same teardown concern: trap 'kill $tunnelPID' EXIT can fail the step if kill returns non-zero (PID already exited). Make the kill non-fatal so it can’t override the curl loop’s success.

Suggested change
trap 'kill $tunnelPID' EXIT
trap 'kill $tunnelPID 2>/dev/null || true' EXIT

Copilot uses AI. Check for mistakes.
echo 'Waiting for service IP'
for i in {1..30}; do
SERVICEIP=$(kubectl get svc -o jsonpath={'.items[1].status.loadBalancer.ingress[0].ip'})
Copy link

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Service IP polling is still using .items[1] which is order-dependent. Use kubectl get svc <expected-service-name> -o jsonpath=... (or a label selector) to make this deterministic.

Suggested change
SERVICEIP=$(kubectl get svc -o jsonpath={'.items[1].status.loadBalancer.ingress[0].ip'})
SERVICEIP=$(kubectl get svc -l app=testapp -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}')

Copilot uses AI. Check for mistakes.
if [ -n "$SERVICEIP" ]; then
break
fi
sleep 4
done
kubectl get svc
SERVICEIP=$(kubectl get svc -o jsonpath={'.items[1].status.loadBalancer.ingress[0].ip'})
echo "SERVICEIP: $SERVICEIP"
echo 'Curling service IP'
curl -m 3 $SERVICEIP:${{env.serviceport}}
sleep 5
kill $tunnelPID
if [ -z "$SERVICEIP" ]; then
echo 'Service IP not ready'
exit 1
fi
echo 'Curling service IP with retries'
for i in {1..20}; do
if curl -m 3 "http://$SERVICEIP:${{env.serviceport}}"; then
exit 0
fi
sleep 3
done
exit 1
- run: |
./draft -v generate-workflow \
-d ./langtest/ \
Expand Down Expand Up @@ -458,16 +500,22 @@ jobs:
uses: medyagh/setup-minikube@master
with:
insecure-registry: "host.minikube.internal:5001,10.0.0.0/24"
- name: Set up buildx
run: |
Copy link

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Buildx setup occurs before switching Docker to Minikube (eval $(minikube ... docker-env) in the next step). This can lead to the builder being created on the wrong daemon. Recommend setting up (or re-selecting) the buildx builder after switching to Minikube’s Docker environment.

Suggested change
run: |
run: |
eval $(minikube -p minikube docker-env)

Copilot uses AI. Check for mistakes.
docker buildx create --use --name draft-builder || docker buildx use draft-builder
docker buildx inspect --bootstrap
- name: Build and Push Image
continue-on-error: true
run: |
export SHELL=/bin/bash
eval $(minikube -p minikube docker-env)
docker build -f ./langtest/Dockerfile -t testapp ./langtest/
docker tag testapp host.minikube.internal:5001/testapp
echo -n "verifying images:"
docker images
docker push host.minikube.internal:5001/testapp
docker buildx build \
--cache-from=type=gha,scope=${{ inputs.language }}-manifests \
--cache-to=type=gha,scope=${{ inputs.language }}-manifests,mode=max \
-f ./langtest/Dockerfile \
-t host.minikube.internal:5001/testapp \
--push \
./langtest/
echo 'Curling host.minikube.internal test app images from minikube'
minikube ssh "curl http://host.minikube.internal:5001/v2/testapp/tags/list"
# Deploys application based on manifest files from previous step
Expand Down Expand Up @@ -497,14 +545,29 @@ jobs:
kubectl get svc
echo 'Starting minikube tunnel'
minikube tunnel > /dev/null 2>&1 & tunnelPID=$!
sleep 120
trap 'kill $tunnelPID' EXIT
Copy link

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

trap 'kill $tunnelPID' EXIT should ignore errors; if minikube tunnel exits early, kill may fail and cause the step to be marked failed under bash -e. Make tunnel cleanup best-effort.

Suggested change
trap 'kill $tunnelPID' EXIT
trap 'kill "$tunnelPID" >/dev/null 2>&1 || true' EXIT

Copilot uses AI. Check for mistakes.
echo 'Waiting for service IP'
for i in {1..30}; do
SERVICEIP=$(kubectl get svc -o jsonpath={'.items[1].status.loadBalancer.ingress[0].ip'})
Copy link

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The JSONPath .items[1]... approach for finding the Service IP is non-deterministic and may break if Service ordering changes. Prefer querying the exact Service (name/selector) before extracting .status.loadBalancer.ingress[0].ip.

Suggested change
SERVICEIP=$(kubectl get svc -o jsonpath={'.items[1].status.loadBalancer.ingress[0].ip'})
SERVICEIP=$(kubectl get svc -l app=testapp -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}')

Copilot uses AI. Check for mistakes.
if [ -n "$SERVICEIP" ]; then
break
fi
sleep 4
done
kubectl get svc
SERVICEIP=$(kubectl get svc -o jsonpath={'.items[1].status.loadBalancer.ingress[0].ip'})
echo "SERVICEIP: $SERVICEIP"
echo 'Curling service IP'
curl -m 3 $SERVICEIP:${{env.serviceport}}
sleep 5
kill $tunnelPID
if [ -z "$SERVICEIP" ]; then
echo 'Service IP not ready'
exit 1
fi
echo 'Curling service IP with retries'
for i in {1..20}; do
if curl -m 3 "http://$SERVICEIP:${{env.serviceport}}"; then
exit 0
fi
sleep 3
done
exit 1
- run: |
./draft -v generate-workflow \
-d ./langtest/ \
Expand Down Expand Up @@ -570,13 +633,21 @@ jobs:
- name: start minikube
id: minikube
uses: medyagh/setup-minikube@master
- name: Set up buildx
run: |
Copy link

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In manifest-update, buildx is configured before switching to the Minikube Docker daemon (eval $(minikube ... docker-env) happens later). With --load, building against the wrong daemon would leave the image unavailable to the cluster. Move buildx setup after switching to Minikube’s Docker env (or re-create/use the builder after the switch).

Suggested change
run: |
run: |
export SHELL=/bin/bash
eval $(minikube -p minikube docker-env)

Copilot uses AI. Check for mistakes.
docker buildx create --use --name draft-builder || docker buildx use draft-builder
docker buildx inspect --bootstrap
- name: Build image
run: |
export SHELL=/bin/bash
eval $(minikube -p minikube docker-env)
docker build -f ./langtest/Dockerfile -t testapp ./langtest/
echo -n "verifying images:"
docker images
docker buildx build \
--cache-from=type=gha,scope=${{ inputs.language }}-manifest-update \
--cache-to=type=gha,scope=${{ inputs.language }}-manifest-update,mode=max \
--load \
-f ./langtest/Dockerfile \
-t testapp \
./langtest/
# Deploys application based on manifest files from previous step
- name: Deploy application
run: kubectl apply -f ./langtest/manifests/
Expand Down
Loading