ROSAENG-726: Spike request-serving isolation on EKS (design + Kata/Firecracker POC) by typeid · Pull Request #738 · openshift-online/rosa-hyperfleet

typeid · 2026-08-10T07:46:38Z

Summary

Design doc comparing isolation models for request-serving workloads on EKS (docs/design/request-serving-isolation-eks.md)
- Option A: Karpenter dedicated nodes (A1: per-HCP subnets, A2: shared subnets)
- Option B: Kata Containers microVM isolation (B1: Firecracker, B2: QEMU)
- Security analysis with attack chain comparison across all options
- Covers FIPS compliance, subnet isolation compatibility, VPA scaling overrides
Working POC of Option B1 (Kata/Firecracker) on AL2023 — validated with full KAS pods running under kata-fc

Related branches

The POC implementation spans three repos. The design doc covers multiple variants; this implementation is B1 (Firecracker) only:

Repo	Branch	Changes
rosa-hyperfleet (this PR)	`spike_request_serving_nodes`	kata-deploy Helm chart, Karpenter NodePool with nested virt + devmapper, design doc
rosa-hyperfleet-api	`request_serving_nodes`	Kata topology annotation on HostedClusters, CPO image update
hypershift	`request_serving_nodes`	Kata RuntimeClass for request-serving pods, aws-iam-authenticator KAS sidecar

Open questions

Does AWS accept shared-subnet Nitro isolation (Option A2) for the ROSA HCP security bar?
Does Kata deploy + run on RHCOS's immutable OSTree root? (FIPS requirement for production)
Karpenter node pairing mechanism for Option A
Full FIPS validation of RHCOS → KVM → VMM → guest kernel stack

Test plan

KAS pods boot and serve under kata-fc with all ConfigMap/Secret mounts
RHCOS AMI with Kata (not yet attempted)
QEMU variant (B2) validation
VPA minAllowed scaling override

🤖 Generated with Claude Code

…s management Replace EKS Auto Mode with OSS Karpenter for MC and RC workload node provisioning. Karpenter is managed via ArgoCD (argocd/config/<cluster-type>/karpenter/) with Bottlerocket AMIs. AWS Load Balancer Controller added as a dependency with corrected TargetGroupBinding API versions across all services. Key changes: - Add Karpenter controller IAM role and SQS interruption queue (Terraform) - Move kms:CreateGrant from node role to Karpenter controller role - Remove unused IAMInstanceProfileCreate/Modify permissions (not needed with Bottlerocket) - Fix ec2nodeclasses CRD name in verify-fips.sh (was nodeclasses.karpenter.k8s.aws) - Fix hypershift-install Job: replace kubectl with curl for in-cluster API polling - Fix monitoring bootstrap: disable tls.enabled alongside admissionWebhooks.enabled - Correct TargetGroupBinding apiVersion to elbv2.k8s.aws/v1beta1 across all services - Update monitoring e2e to explicit ginkgo invocation with 15m timeout Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…-error check On nonzero exit from hypershift install, namespace existence alone is a weak signal when backoffLimit=1: on retry the namespace may already exist from a prior failed attempt, causing a real failure to be treated as the known benign post-apply error. Capture stdout+stderr into _hs_out, print it, and check the captured output for resource-apply failure strings before falling back to the namespace check. The output check is the primary gate; namespace presence remains a secondary corroborating signal. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…ecksum verification Replace get-helm-3 script (which lacks retry logic) with direct download using curl retry parameters. This matches the pattern used for Terraform and other tools, ensuring transient network failures don't break CI builds. - Add architecture detection (amd64/arm64) for multi-arch support - Download Helm tarball and SHA-256 checksum with retry/timeout settings - Verify checksum before extraction (fails build on mismatch) - Add helm version verification step at end

…t redis-ha - Restore argocd/config/regional-cluster/hyperfleet/values.yaml to match upstream/main - Revert to openshift-online/rosa-hyperfleet-api repository - Use pinned commit f7ec7d59ed8523bc66277cce3f4814e0dee5fa83 - Fix chart path to hyperfleet-operator/charts - Add explicit redis-ha.enabled: true in argocd/config/shared/argocd/values.yaml - Improves clarity (though implied by ha.enabled: true) - Preserves existing HA configuration and tolerations - Strengthen regression tests in scripts/test_render.py - Add real Jinja variables to both Go-template quote tests - Assert complete expected scan results - Verify all Go-template keywords (if/else/end) are ignored - Include full if/else/end block in double-quoted fixture Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Critical fixes: - HyperShift Job: capture curl errors in _crd_ready() (curl -sSf), fail-by-default error detection - Terraform destroy: change to on_failure=fail with preflight checks for aws/kubectl - Remove unused KARPENTER_VERSION and KARPENTER_QUEUE_URL variables Code quality improvements: - Fix if/else indentation in ecs-bootstrap ArgoCD installation block - Create shared Karpenter chart directory (argocd/config/shared/karpenter) - Replace MC/RC karpenter directories with symlinks to shared Documentation updates: - Correct FIPS documentation to reflect current state (standard Bottlerocket) - Note FIPS-validated compute coming with RHEL AMI work - Remove references to removed Karpenter variables from README Net: -119 lines, improved error handling, eliminated duplicate code Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

openshift-ci · 2026-08-10T07:46:43Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

openshift-ci · 2026-08-10T07:46:45Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign slopezz for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci · 2026-08-10T07:46:47Z

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

coderabbitai · 2026-08-10T07:46:47Z

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 14ad1397-7f6d-4f96-8a08-577ca8a8cf0b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

_{Comment @coderabbitai help to get the list of available commands.}

openshift-ci-robot · 2026-08-10T07:52:40Z

@typeid: This pull request references ROSAENG-726 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

Design doc comparing isolation models for request-serving workloads on EKS (docs/design/request-serving-isolation-eks.md)

Option A: Karpenter dedicated nodes (A1: per-HCP subnets, A2: shared subnets)

Option B: Kata Containers microVM isolation (B1: Firecracker, B2: QEMU)

Security analysis with attack chain comparison across all options

Covers FIPS compliance, subnet isolation compatibility, VPA scaling overrides

Working POC of Option B1 (Kata/Firecracker) on AL2023 — validated with full KAS pods running under kata-fc

Related branches

The POC implementation spans three repos. The design doc covers multiple variants; this implementation is B1 (Firecracker) only:

Repo Branch Changes

rosa-hyperfleet (this PR) spike_request_serving_nodes kata-deploy Helm chart, Karpenter NodePool with nested virt + devmapper, design doc

rosa-hyperfleet-api request_serving_nodes Kata topology annotation on HostedClusters, CPO image update

hypershift request_serving_nodes Kata RuntimeClass for request-serving pods, aws-iam-authenticator KAS sidecar

Open questions

Does AWS accept shared-subnet Nitro isolation (Option A2) for the ROSA HCP security bar?

Does Kata deploy + run on RHCOS's immutable OSTree root? (FIPS requirement for production)

Karpenter node pairing mechanism for Option A

Full FIPS validation of RHCOS → KVM → VMM → guest kernel stack

Test plan

KAS pods boot and serve under kata-fc with all ConfigMap/Secret mounts

RHCOS AMI with Kata (not yet attempted)

QEMU variant (B2) validation

VPA minAllowed scaling override

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

…cker POC) Design doc comparing two isolation models for request-serving workloads on EKS: - Option A: Karpenter dedicated nodes (A1: per-HCP subnets, A2: shared subnets) - Option B: Kata Containers microVM isolation (B1: Firecracker, B2: QEMU) Working POC of Option B1 (Kata/Firecracker) on AL2023: - kata-deploy Helm chart (upstream 3.27.0) with Firecracker runtime - Karpenter NodePool with nested virtualization and devmapper thin-pool - Validated with full KAS pods running under kata-fc Related branches: - rosa-hyperfleet-api: request_serving_nodes - hypershift: request_serving_nodes Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

theautoroboto and others added 7 commits August 5, 2026 14:24

documentation update, update helm version

d1f3f9f

fix: add SHA-256 checksum verification for Helm installation

178a327

openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 10, 2026

openshift-ci Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 10, 2026

typeid force-pushed the spike_request_serving_nodes branch 2 times, most recently from 9ded404 to 34279f5 Compare August 10, 2026 07:50

typeid changed the title ~~spike: request-serving isolation on EKS — design + Kata/Firecracker POC~~ Spike: request-serving isolation on EKS (design + Kata/Firecracker POC) Aug 10, 2026

typeid changed the title ~~Spike: request-serving isolation on EKS (design + Kata/Firecracker POC)~~ ROSAENG-726: Spike request-serving isolation on EKS (design + Kata/Firecracker POC) Aug 10, 2026

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Aug 10, 2026

typeid force-pushed the spike_request_serving_nodes branch from 34279f5 to 9ba7735 Compare August 12, 2026 07:55

Navigation Menu

Uh oh!

ROSAENG-726: Spike request-serving isolation on EKS (design + Kata/Firecracker POC) - #738

ROSAENG-726: Spike request-serving isolation on EKS (design + Kata/Firecracker POC)#738
typeid wants to merge 8 commits into
openshift-online:mainfrom
typeid:spike_request_serving_nodes

typeid commented Aug 10, 2026 •

edited

Loading

Uh oh!

openshift-ci Bot commented Aug 10, 2026

Uh oh!

openshift-ci Bot commented Aug 10, 2026

Uh oh!

openshift-ci Bot commented Aug 10, 2026

Uh oh!

coderabbitai Bot commented Aug 10, 2026 •

edited

Loading

Review skipped

Uh oh!

openshift-ci-robot commented Aug 10, 2026 •

edited by openshift-ci Bot

Loading

Summary

Related branches

Open questions

Test plan

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

typeid commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related branches

Open questions

Test plan

Uh oh!

openshift-ci Bot commented Aug 10, 2026

Uh oh!

openshift-ci Bot commented Aug 10, 2026

Uh oh!

openshift-ci Bot commented Aug 10, 2026

Uh oh!

coderabbitai Bot commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

openshift-ci-robot commented Aug 10, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related branches

Open questions

Test plan

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

typeid commented Aug 10, 2026 •

edited

Loading

coderabbitai Bot commented Aug 10, 2026 •

edited

Loading

openshift-ci-robot commented Aug 10, 2026 •

edited by openshift-ci Bot

Loading