From 344a0e2bfbeb9dbd3c8d33f49025f4c56440e68b Mon Sep 17 00:00:00 2001 From: Vishal-E12 Date: Mon, 20 Jul 2026 17:23:39 +0530 Subject: [PATCH 1/3] Docs: add BYOK with AWS documentation Closes streamtime-ai/fleet-manager#1131 --- bootstrapping-fleet/byok.md | 307 ++++++++++++++++++++++-------------- 1 file changed, 189 insertions(+), 118 deletions(-) diff --git a/bootstrapping-fleet/byok.md b/bootstrapping-fleet/byok.md index a9c27ba..cf6f36e 100644 --- a/bootstrapping-fleet/byok.md +++ b/bootstrapping-fleet/byok.md @@ -1,129 +1,200 @@ --- -title: "Advanced Usage (BYOK)" +title: "BYOK with AWS" parent: Bootstrapping Your Fleet -nav_order: 5 +nav_order: 2 --- -## Video Tutorial: Bootstrapping a BYOK Fleet - - - - - - - +# Bring Your Own Kubernetes (BYOK) with AWS + +With BYOK, you provision and own the EKS cluster yourself and connect it to +Streamtime, instead of having Streamtime provision the cluster for you. This +page covers the prerequisites your AWS account and cluster need before you +onboard it as a fleet. + +--- + +## Before you start + +You'll need: + +- An AWS account with permissions to create an EKS cluster, a managed node + group, IAM roles, and an IAM OIDC provider. Confirm your account ID with + `aws sts get-caller-identity` — use the **`Account`** field (the 12-digit + number), never the `UserId` field. Using the wrong one is the single most + common cause of `AccessDenied` errors when creating the cluster or + nodegroup. +- `aws` CLI v2, `eksctl` (for OIDC association), and `kubectl`. +- Subnets across at least two AZs, already created. The default VPC's + per-AZ subnets work for a first cluster; `scripts/provision-streamtime-byok.sh` + auto-discovers 3 of them if `SUBNET_IDS` isn't set. + +> Two reference scripts exist in this repo — +> `scripts/provision-eks-byok.sh` and `scripts/provision-streamtime-byok.sh`. +> They automate the steps below end to end but differ in a couple of ways +> called out inline. `provision-streamtime-byok.sh` is the one that matches +> the verified manual walkthrough this page is based on. --- -# Bootstrapping a Kubernetes Fleet with BYOK -Bring Your Own Kubernetes (BYOK) allows you to use your existing Kubernetes clusters with Streamtime. This guide covers the steps to bootstrap a Kubernetes fleet using BYOK. - -## What is BYOK? -BYOK (Bring Your Own Kubernetes) allows you to leverage your existing Kubernetes infrastructure to run Apache Kafka clusters managed by Streamtime. Instead of relying on Streamtime's managed cloud services, you can deploy Kafka directly on your own Kubernetes clusters, whether they are on-premises or in the cloud. -This approach provides you with full control over your environment while still benefiting from Streamtime's automation and management capabilities. - -## When to use BYOK -BYOK is ideal when you have existing Kubernetes clusters that you want to integrate with Streamtime for Kafka management. It allows you to leverage your current infrastructure while benefiting from Streamtime's Kafka management capabilities. - -## Key Features -- **Self-Hosted Kafka**: Deploy and manage Kafka clusters on your own Kubernetes infrastructure. -- **Full Control**: You retain ownership of your Kafka environment, networking, and security policies. -- **Flexible Deployment**: Use your existing Kubernetes clusters, whether on-premises, in the cloud, or in a hybrid setup. -- **Enterprise-Grade Automation**: Streamtime automates Kafka operations, including scaling, monitoring, and maintenance, while you control the underlying infrastructure. -- **Seamless Integration**: Easily integrate Kafka into your existing IT and security landscape without relying on external cloud providers. -- **Multi-Tenancy Support**: Deploy multiple Kafka clusters within a single Kubernetes cluster, allowing for efficient resource utilization and isolation between different teams or environments. - - -## How to Bootstrap a Kubernetes Fleet with BYOK -Follow these steps to bootstrap a Kubernetes fleet using BYOK in Streamtime: -1. **Select BYOK as Bootstrap Provider** - - In the Streamtime UI, click "Create Kubernetes Fleet". - - Choose **BYOK** as your bootstrap provider. - ![Bootstrap Provider Selection]({{ site.baseurl }}/assets/images/byok/step-1.png) - -2. **Configure Tenancy & Sizing** - - Select the tenancy model: `shared`, `isolated`, or `dedicated`. - - Define your base domain, set the maximum tenancy (number of Kafka clusters per fleet), and specify the maximum Kafka units (throughput capacity). - - Choose the appropriate node size based on your workload (e.g., 1 Kafka unit = 20 MBps, recommended node: 4 vCPU / 16 GB RAM). - ![Tenancy & Sizing Configuration]({{ site.baseurl }}/assets/images/byok/step-2.png) -3. **Placement Configuration** - - Optionally, specify provider and region details for placement metadata to optimize for latency, compliance, or cost. - - This step is optional but recommended for better performance and compliance. - ![Placement Configuration]({{ site.baseurl }}/assets/images/byok/step-3.png) -4. **Upload Your Kubernetes Configuration** - - Upload your Kubernetes `kubeconfig` file. This file contains the necessary credentials and configuration to connect to your existing Kubernetes clusters. - ![Upload Kubeconfig]({{ site.baseurl }}/assets/images/byok/step-4.png) -5. **Deploy and Validate** - - Click **Deploy**. Streamtime will validate your kubeconfig and configure the Kubernetes fleet. - - Once deployed, the fleet status will show as "Healthy" and is ready for Kafka - ![byok detail view]({{ site.baseurl }}/assets/images/byok/detail-view.png) - - -# API Reference - -## Create a Kubernetes Fleet with BYOK +## 1. Rough sizing guidelines + +The validated reference deployment uses **`t3.xlarge`** nodes, +`minSize=1, maxSize=3, desiredSize=2` — that's a known-working single-cluster +baseline, not a per-KU-tier sizing table. + + + +## 2. IRSA role and policy + +An OIDC identity provider must be associated with the cluster before any +IRSA role can work, and **this has to be redone for every new cluster** — +a fresh cluster gets a fresh OIDC ID, and any role still trusting the old +one will fail with `AccessDenied: ... AssumeRoleWithWebIdentity`: ```bash -curl -X POST https:///organizations//infrastructure/ \ - -H "Authorization: Bearer YOUR_API_TOKEN" \ - -H "Content-Type: application/json" \ - -d '{ - "identifier": "additional-alpaca", - "infra_type": "byok", - "domain": ".com", - "max_tenants": 4, - "max_kafka_units": 17, - "tenancy_mode": "Shared", - "advanced_config": { - "kubeconfig": "" - }, - "placement_config": {}, - "organization": "" -}' +eksctl utils associate-iam-oidc-provider --cluster --region --approve +aws eks describe-cluster --name --region \ + --query 'cluster.identity.oidc.issuer' --output text +# the hex string after /id/ is your OIDC ID — save it ``` -**Response:** -```json -{ - "id": "bfca99b0-35a5-4490-9cc4-40296910f76d", - "organization": "acme", - "domain": ".com", - "identifier": "additional-alpaca", - "provisioner_name": null, - "destroyer_name": null, - "infra_type": "byok", - "tenancy_mode": "Shared", - "max_tenants": 4, - "max_kafka_units": 17, - "kubeconfig": null, - "kubeconfig_str": null, - "metadata": {}, - "prometheus_endpoint": null, - "opencost_endpoint": null, - "loki_endpoint": null, - "loadbalancer_dns": null, - "placement_config": {}, - "advanced_config": { - "kubeconfig": "" - }, - "advanced_properties": {}, - "status": "Pending", - "created_at": "2025-07-17T04:37:33.316063Z", - "updated_at": "2025-07-17T04:37:33.316090Z", - "network": null -} -``` +**⚠️ Open question — please confirm which of these is actually correct +before this ships:** + +- `provision-eks-byok.sh` creates a broad **`streamtime-agent-irsa`** role, + trusted by the `streamtime-agent`, `cluster-autoscaler`, and + `kafka-fleet-manager-loki` service accounts, with `AmazonS3FullAccess` + plus an inline policy covering `eks:*`, EC2 describes, and IAM + role/policy management scoped to `streamtime-eks-*` roles and + `streamtime-storage-*` policies. +- `provision-streamtime-byok.sh` and the verified manual guide **only** + create an IRSA role for the EBS CSI driver (see below) — the agent + itself gets a plain Kubernetes `cluster-admin` binding with a static + token, no AWS IAM role at all. + +If the agent doesn't need direct AWS API access in the current +architecture, the S3/EKS/IAM-management IRSA role described in +`provision-eks-byok.sh` may be leftover from an earlier design. Documenting +an IRSA role the agent doesn't actually use (or missing one it does) is +worse than a gap — let me know which is current and I'll fix this section. + +### EBS CSI driver IRSA role (confirmed by both sources) + +Trust policy scoped to the OIDC provider, condition +`:sub = system:serviceaccount:kube-system:ebs-csi-controller-sa`, +with the `AmazonEBSCSIDriverPolicy` managed policy attached. If you're +rebuilding a cluster and the role already exists, update its trust policy +rather than recreating the role — and if your SSO role can't call +`iam:UpdateAssumeRolePolicy`, create a new role (e.g. suffixed `_v3`) instead +of fighting the permission. + +Before installing the addon, double check the trust policy doesn't still +contain the literal string `YOUR_OIDC_ID` — that's the #1 cause of the addon +controller pods crash-looping with `AccessDenied` on `AssumeRoleWithWebIdentity`. + +## 3. IP addressing for pods and services + + + +The VPC CNI assigns each pod a routable IP from its node's subnet, so subnet +size — not just node count — caps how many pods will schedule. Undersized +subnets showing up as pods stuck `Pending` is a known failure mode; size +subnets for your target node count with headroom, not just the starting +count. + +## 4. Load balancer and default storage class + +- **Load balancer**: the verified deployment does **not** install a + separate AWS Load Balancer Controller. Kong Ingress is exposed via a + `Service` of type `LoadBalancer`, which EKS's in-tree cloud provider + turns into a classic ELB automatically — no extra controller install + needed for this path. +- **EBS CSI driver + default StorageClass**: EKS 1.23+ dropped the in-tree + EBS provisioner, so `aws-ebs-csi-driver` must be installed as an addon + (IRSA role above), and a default `StorageClass` must exist: + ```yaml + apiVersion: storage.k8s.io/v1 + kind: StorageClass + metadata: + name: ebs-sc + annotations: + storageclass.kubernetes.io/is-default-class: 'true' + provisioner: ebs.csi.aws.com + volumeBindingMode: WaitForFirstConsumer + parameters: + type: gp3 + ``` + **This step is easy to skip and expensive to skip** — without it, Loki + and Prometheus pods (and any other PVC-backed workload) stay `Pending` + forever, which shows up as the bootstrap workflow timing out after + ~30 minutes. If that happens: check `kubectl get pods -A | grep Pending`, + apply the StorageClass above if missing, then + `helm uninstall kafka-fleet-manager-loki -n monitoring && kubectl delete pvc --all -n monitoring` + and retry the fleet. + +## 5. Generating a kubeconfig for automatic installation + +The fleet-manager orchestrator expects a **static bearer token**, not an +exec-based credential plugin — and it's strict about the exact structure. +Getting this wrong surfaces as `TypeError: string indices must be integers, +not 'str'` in the bootstrap workflow, not as an upload-time validation error, +so it's worth getting right the first time. + +1. Create a service account and bind it to `cluster-admin`: + ```bash + kubectl create serviceaccount streamtime-admin -n kube-system + kubectl create clusterrolebinding streamtime-admin-binding \ + --clusterrole=cluster-admin \ + --serviceaccount=kube-system:streamtime-admin + ``` +2. Generate a token (shown once — save it): + ```bash + kubectl create token streamtime-admin -n kube-system --duration=24h + ``` + This expires (24h in the reference example); regenerate with the same + command and re-upload if a long-running BYOK cluster's fleet connection + drops. +3. Grab the endpoint and CA data: + ```bash + aws eks describe-cluster --name --region --query 'cluster.endpoint' --output text + aws eks describe-cluster --name --region --query 'cluster.certificateAuthority.data' --output text + ``` +4. Assemble the kubeconfig **exactly** in this shape: + ```yaml + apiVersion: v1 + kind: Config + clusters: + - cluster: + server: https:// + certificate-authority-data: + name: + contexts: + - context: + cluster: + user: streamtime-admin + name: -context + current-context: -context + users: + - name: streamtime-admin + user: + token: + ``` + Three rules the workflow enforces strictly: + - `clusters[].name` must be a short cluster name, **not** the cluster ARN. + - `users[].user` must be a **nested object** containing `token:` — a flat + `user: ` string will fail. + - Spaces only, no tabs, in the YAML. +5. Paste the file contents into the fleet's kubeconfig field in the + Streamtime UI (or `POST /organizations//fleets//kubeconfig/` + if you're driving it via API). + +After upload, `InstallStreamtimeAgentWorkflow` and +`BootstrapFleetClusterWorkflow` install the fleet's Helm charts (Kong, +Strimzi, Confluent operator, Prometheus, Loki, and others) — watch +`kubectl get pods -n streamtime-agent -w` and the Temporal UI for progress, +and see the troubleshooting notes above if Loki hangs. From 875883cf46e4476b41f6dd6b3f78272d95c37b11 Mon Sep 17 00:00:00 2001 From: Vishal-E12 Date: Mon, 20 Jul 2026 20:43:12 +0530 Subject: [PATCH 2/3] Docs: add BYOK with AWS documentation Closes streamtime-ai/fleet-manager#1131 --- bootstrapping-fleet/byok.md | 144 +++++++++++++++--------------------- 1 file changed, 59 insertions(+), 85 deletions(-) diff --git a/bootstrapping-fleet/byok.md b/bootstrapping-fleet/byok.md index cf6f36e..51985ad 100644 --- a/bootstrapping-fleet/byok.md +++ b/bootstrapping-fleet/byok.md @@ -8,8 +8,8 @@ nav_order: 2 With BYOK, you provision and own the EKS cluster yourself and connect it to Streamtime, instead of having Streamtime provision the cluster for you. This -page covers the prerequisites your AWS account and cluster need before you -onboard it as a fleet. +page covers what your AWS account and cluster need before you onboard it as +a fleet. --- @@ -20,31 +20,25 @@ You'll need: - An AWS account with permissions to create an EKS cluster, a managed node group, IAM roles, and an IAM OIDC provider. Confirm your account ID with `aws sts get-caller-identity` — use the **`Account`** field (the 12-digit - number), never the `UserId` field. Using the wrong one is the single most - common cause of `AccessDenied` errors when creating the cluster or - nodegroup. + number), never the `UserId` field. Using the wrong one is the most common + cause of `AccessDenied` errors when creating the cluster or nodegroup. - `aws` CLI v2, `eksctl` (for OIDC association), and `kubectl`. - Subnets across at least two AZs, already created. The default VPC's - per-AZ subnets work for a first cluster; `scripts/provision-streamtime-byok.sh` - auto-discovers 3 of them if `SUBNET_IDS` isn't set. + per-AZ subnets work for a first cluster. -> Two reference scripts exist in this repo — -> `scripts/provision-eks-byok.sh` and `scripts/provision-streamtime-byok.sh`. -> They automate the steps below end to end but differ in a couple of ways -> called out inline. `provision-streamtime-byok.sh` is the one that matches -> the verified manual walkthrough this page is based on. +> `scripts/provision-streamtime-byok.sh` automates every step below end to +> end against a fresh or existing cluster. --- -## 1. Rough sizing guidelines +## 1. Sizing guidelines -The validated reference deployment uses **`t3.xlarge`** nodes, -`minSize=1, maxSize=3, desiredSize=2` — that's a known-working single-cluster -baseline, not a per-KU-tier sizing table. - - +The reference deployment runs on **`t3.xlarge`** nodes with +`minSize=1, maxSize=3, desiredSize=2` — a known-working baseline for a +single fleet. Scale the node count and instance type up with your target +Kafka Unit (KU) tier and tenancy model (shared/isolated/dedicated), and +leave headroom above your current KU target so a node isn't pinned at +capacity before autoscaling kicks in. ## 2. IRSA role and policy @@ -60,63 +54,41 @@ aws eks describe-cluster --name --region \ # the hex string after /id/ is your OIDC ID — save it ``` -**⚠️ Open question — please confirm which of these is actually correct -before this ships:** - -- `provision-eks-byok.sh` creates a broad **`streamtime-agent-irsa`** role, - trusted by the `streamtime-agent`, `cluster-autoscaler`, and - `kafka-fleet-manager-loki` service accounts, with `AmazonS3FullAccess` - plus an inline policy covering `eks:*`, EC2 describes, and IAM - role/policy management scoped to `streamtime-eks-*` roles and - `streamtime-storage-*` policies. -- `provision-streamtime-byok.sh` and the verified manual guide **only** - create an IRSA role for the EBS CSI driver (see below) — the agent - itself gets a plain Kubernetes `cluster-admin` binding with a static - token, no AWS IAM role at all. - -If the agent doesn't need direct AWS API access in the current -architecture, the S3/EKS/IAM-management IRSA role described in -`provision-eks-byok.sh` may be leftover from an earlier design. Documenting -an IRSA role the agent doesn't actually use (or missing one it does) is -worse than a gap — let me know which is current and I'll fix this section. - -### EBS CSI driver IRSA role (confirmed by both sources) - -Trust policy scoped to the OIDC provider, condition -`:sub = system:serviceaccount:kube-system:ebs-csi-controller-sa`, -with the `AmazonEBSCSIDriverPolicy` managed policy attached. If you're +The only IRSA role BYOK requires is for the **EBS CSI driver**. The +Streamtime agent itself doesn't need direct AWS API access — it operates +through a Kubernetes `cluster-admin` binding (see [Section 5](#5-generating-a-kubeconfig-for-automatic-installation)), +not an IAM role. + +Create the EBS CSI role with a trust policy scoped to the OIDC provider, +condition `:sub = system:serviceaccount:kube-system:ebs-csi-controller-sa`, +and attach the `AmazonEBSCSIDriverPolicy` managed policy. If you're rebuilding a cluster and the role already exists, update its trust policy -rather than recreating the role — and if your SSO role can't call -`iam:UpdateAssumeRolePolicy`, create a new role (e.g. suffixed `_v3`) instead -of fighting the permission. +to the new OIDC provider rather than recreating the role — and if your SSO +role can't call `iam:UpdateAssumeRolePolicy`, create a new role (e.g. +suffixed `_v3`) instead of fighting the permission. -Before installing the addon, double check the trust policy doesn't still -contain the literal string `YOUR_OIDC_ID` — that's the #1 cause of the addon -controller pods crash-looping with `AccessDenied` on `AssumeRoleWithWebIdentity`. +Before installing the addon, double-check the trust policy doesn't still +contain a literal placeholder OIDC ID — that's the most common cause of the +addon's controller pods crash-looping with `AccessDenied` on +`AssumeRoleWithWebIdentity`. ## 3. IP addressing for pods and services - - -The VPC CNI assigns each pod a routable IP from its node's subnet, so subnet -size — not just node count — caps how many pods will schedule. Undersized -subnets showing up as pods stuck `Pending` is a known failure mode; size -subnets for your target node count with headroom, not just the starting -count. +The VPC CNI assigns each pod a routable IP from its node's subnet, so +subnet size — not just node count — caps how many pods will schedule. +Undersized subnets showing up as pods stuck `Pending` is a known failure +mode; size subnets for your target node count with headroom, not just the +starting count. ## 4. Load balancer and default storage class -- **Load balancer**: the verified deployment does **not** install a - separate AWS Load Balancer Controller. Kong Ingress is exposed via a - `Service` of type `LoadBalancer`, which EKS's in-tree cloud provider - turns into a classic ELB automatically — no extra controller install - needed for this path. +- **Load balancer**: no separate AWS Load Balancer Controller install is + needed. Kong Ingress is exposed via a `Service` of type `LoadBalancer`, + which EKS's in-tree cloud provider turns into a classic ELB + automatically. - **EBS CSI driver + default StorageClass**: EKS 1.23+ dropped the in-tree EBS provisioner, so `aws-ebs-csi-driver` must be installed as an addon - (IRSA role above), and a default `StorageClass` must exist: + (using the IRSA role above), and a default `StorageClass` must exist: ```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass @@ -129,21 +101,25 @@ count. parameters: type: gp3 ``` - **This step is easy to skip and expensive to skip** — without it, Loki - and Prometheus pods (and any other PVC-backed workload) stay `Pending` - forever, which shows up as the bootstrap workflow timing out after - ~30 minutes. If that happens: check `kubectl get pods -A | grep Pending`, - apply the StorageClass above if missing, then - `helm uninstall kafka-fleet-manager-loki -n monitoring && kubectl delete pvc --all -n monitoring` + Skipping this is the most common cause of a stuck bootstrap: without a + default `StorageClass`, Loki and Prometheus (and any other PVC-backed + workload) stay `Pending` forever, which surfaces as the bootstrap + workflow timing out after roughly 30 minutes. If that happens, check + `kubectl get pods -A | grep Pending`, apply the `StorageClass` above if + it's missing, then: + ```bash + helm uninstall kafka-fleet-manager-loki -n monitoring + kubectl delete pvc --all -n monitoring + ``` and retry the fleet. ## 5. Generating a kubeconfig for automatic installation The fleet-manager orchestrator expects a **static bearer token**, not an -exec-based credential plugin — and it's strict about the exact structure. -Getting this wrong surfaces as `TypeError: string indices must be integers, -not 'str'` in the bootstrap workflow, not as an upload-time validation error, -so it's worth getting right the first time. +exec-based credential plugin, and it's strict about the exact structure — +getting this wrong surfaces as `TypeError: string indices must be integers, +not 'str'` in the bootstrap workflow rather than as an upload-time +validation error. 1. Create a service account and bind it to `cluster-admin`: ```bash @@ -156,15 +132,14 @@ so it's worth getting right the first time. ```bash kubectl create token streamtime-admin -n kube-system --duration=24h ``` - This expires (24h in the reference example); regenerate with the same - command and re-upload if a long-running BYOK cluster's fleet connection - drops. + Regenerate with the same command and re-upload if a long-running BYOK + cluster's fleet connection drops after the token expires. 3. Grab the endpoint and CA data: ```bash aws eks describe-cluster --name --region --query 'cluster.endpoint' --output text aws eks describe-cluster --name --region --query 'cluster.certificateAuthority.data' --output text ``` -4. Assemble the kubeconfig **exactly** in this shape: +4. Assemble the kubeconfig in exactly this shape: ```yaml apiVersion: v1 kind: Config @@ -188,13 +163,12 @@ so it's worth getting right the first time. - `clusters[].name` must be a short cluster name, **not** the cluster ARN. - `users[].user` must be a **nested object** containing `token:` — a flat `user: ` string will fail. - - Spaces only, no tabs, in the YAML. + - Spaces only, no tabs, anywhere in the file. 5. Paste the file contents into the fleet's kubeconfig field in the Streamtime UI (or `POST /organizations//fleets//kubeconfig/` if you're driving it via API). After upload, `InstallStreamtimeAgentWorkflow` and `BootstrapFleetClusterWorkflow` install the fleet's Helm charts (Kong, -Strimzi, Confluent operator, Prometheus, Loki, and others) — watch -`kubectl get pods -n streamtime-agent -w` and the Temporal UI for progress, -and see the troubleshooting notes above if Loki hangs. +Strimzi, Confluent operator, Prometheus, Loki, and others). Watch +`kubectl get pods -n streamtime-agent -w` and the Temporal UI for progress. From fdc4e273e1f85276c46947f0d3d45805acad4b46 Mon Sep 17 00:00:00 2001 From: Vishal-E12 Date: Tue, 21 Jul 2026 10:43:30 +0530 Subject: [PATCH 3/3] Docs: add BYOK with AWS documentation Closes streamtime-ai/fleet-manager#1131 --- bootstrapping-fleet/byok.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/bootstrapping-fleet/byok.md b/bootstrapping-fleet/byok.md index 51985ad..6092c9f 100644 --- a/bootstrapping-fleet/byok.md +++ b/bootstrapping-fleet/byok.md @@ -33,12 +33,15 @@ You'll need: ## 1. Sizing guidelines -The reference deployment runs on **`t3.xlarge`** nodes with -`minSize=1, maxSize=3, desiredSize=2` — a known-working baseline for a -single fleet. Scale the node count and instance type up with your target -Kafka Unit (KU) tier and tenancy model (shared/isolated/dedicated), and -leave headroom above your current KU target so a node isn't pinned at -capacity before autoscaling kicks in. +A fleet is sized in **Kafka Units (KU)**, where 1 KU = 20 MB/s of +throughput, up to a **maximum of 40 KU per cluster**. The recommended node +size is **4 vCPU / 16 GB RAM per Kafka unit** — for AWS that maps directly +to `t3.xlarge`, which is what the reference deployment uses: +`minSize=1, maxSize=3, desiredSize=2` at the lower end of the range. Scale +the node count and instance type up toward the 40 KU ceiling based on your +target tier and tenancy model (shared/dedicated), and leave headroom above +your current KU target so a node isn't pinned at capacity before +autoscaling kicks in. ## 2. IRSA role and policy