From 506d3ad11a650e43140f80ac6a8e74959bc38aa7 Mon Sep 17 00:00:00 2001 From: Rodney Osodo Date: Mon, 6 Jul 2026 15:57:26 +0300 Subject: [PATCH 1/3] docs: replace Magistrala with Atom and update related docs Signed-off-by: Rodney Osodo --- content/docs/tee.mdx | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/content/docs/tee.mdx b/content/docs/tee.mdx index 9e75293..e59911f 100644 --- a/content/docs/tee.mdx +++ b/content/docs/tee.mdx @@ -290,12 +290,12 @@ sudo apt-get install -y \ Set the required variables before running the script: -| Variable | Description | Example | -| -------------------- | ---------------------------- | ------------------- | -| `PROPLET_TENANT_ID` | Atom tenant identifier | `my-tenant-123` | -| `PROPLET_ENTITY_ID` | Unique client identifier | `proplet-worker-01` | -| `PROPLET_API_KEY` | Authentication key | `secret-key-here` | -| `PROPLET_CHANNEL_ID` | Communication channel ID | `channel-456` | +| Variable | Description | Example | +| -------------------- | ------------------------ | ------------------- | +| `PROPLET_TENANT_ID` | Atom tenant identifier | `my-tenant-123` | +| `PROPLET_ENTITY_ID` | Unique client identifier | `proplet-worker-01` | +| `PROPLET_API_KEY` | Authentication key | `secret-key-here` | +| `PROPLET_CHANNEL_ID` | Communication channel ID | `channel-456` | Optional variables: From 0e8ed3f9c2473f50f6ca5ccc950e06a6131eee5d Mon Sep 17 00:00:00 2001 From: Rodney Osodo Date: Tue, 21 Jul 2026 13:48:05 +0300 Subject: [PATCH 2/3] docs: use tenant/entity/api-key terminology and add plugin docs Signed-off-by: Rodney Osodo --- content/docs/architecture.mdx | 11 +- content/docs/developer-guide.mdx | 45 ++- content/docs/k8s/deploy-without-operator.mdx | 18 +- content/docs/k8s/deployment.mdx | 145 ++++++-- content/docs/k8s/example.mdx | 80 ++--- content/docs/k8s/features.mdx | 342 +++++++++---------- content/docs/k8s/operator.mdx | 51 ++- content/docs/monitoring.mdx | 4 +- content/docs/proxy.mdx | 19 +- content/docs/reference.mdx | 2 + content/docs/tee.mdx | 8 +- 11 files changed, 405 insertions(+), 320 deletions(-) diff --git a/content/docs/architecture.mdx b/content/docs/architecture.mdx index 7e26059..07d02a0 100644 --- a/content/docs/architecture.mdx +++ b/content/docs/architecture.mdx @@ -112,9 +112,16 @@ Proplet includes a built-in process monitoring subsystem for observing task and The Proxy service handles downloading and distributing WebAssembly modules from OCI registries and distributes them to proplets. -### Atom +### Plugin System + +Propeller includes two independent WebAssembly plugin systems that extend behaviour at runtime without recompiling core binaries: + +- **Manager Plugins**: Written with [`propeller-plugin-sdk`](https://github.com/absmach/propeller/tree/main/crates/propeller-plugin-sdk) (Rust, compiled to `wasm32-wasip1`). Plugins intercept task operations (authorize, enrich, on_before_proplet_select, on_before_dispatch, on_task_start, on_task_complete) before they reach the scheduler. +- **Proplet Plugins**: Written with [`propeller-proplet-plugin-sdk`](https://github.com/absmach/propeller/tree/main/crates/propeller-proplet-plugin-sdk) (Rust, WIT component model). Plugins intercept tasks on the edge node just before execution. -[Atom](https://github.com/absmach/atom) is an open-source identity and authorization platform that provides entity management, tenant isolation, access policies, and API key management. +Both plugin types run inside a sandboxed Wasmtime instance with WASI. See the [Plugins](/plugins) documentation for details. + +### Atom It uses [FluxMQ](https://github.com/absmach/fluxmq) as its message broker and provides a GraphQL API for managing: diff --git a/content/docs/developer-guide.mdx b/content/docs/developer-guide.mdx index a2766fe..1972704 100644 --- a/content/docs/developer-guide.mdx +++ b/content/docs/developer-guide.mdx @@ -211,6 +211,23 @@ make addition This compiles the example to WebAssembly format. The compiled `.wasm` file is located in the `build` directory. +### Building the Plugin SDK Crates + +Propeller includes two Rust crate SDKs for extending behaviour via WebAssembly plugins: + +- `crates/propeller-plugin-sdk` — SDK for writing Manager plugins +- `crates/propeller-proplet-plugin-sdk` — SDK for writing Proplet plugins + +Build both with: + +```bash +cargo build --release --workspace --manifest-path crates/Cargo.toml +``` + +For authoring plugins, see the [Plugins](/plugins) documentation and the [plugin-auth](https://github.com/absmach/propeller/tree/main/examples/plugin-auth) and [proplet-plugin-example](https://github.com/absmach/propeller/tree/main/examples/proplet-plugin-example) examples. + +### Testing Examples + To test the compiled Wasm example locally with Wasmtime (requires Wasmtime installed): ```bash @@ -336,24 +353,16 @@ Example output: ```bash docker compose -f docker/compose.yaml --env-file docker/.env up -d -[+] Running 17/17 - ✔ Container atom-db Started 2.5s - ✔ Container atom-redis Started 2.1s - ✔ Container atom-jaeger Started 3.0s - ✔ Container atom-http Started 5.1s - ✔ Container atom-grpc Started 5.2s - ✔ Container atom-domains-db Started 2.8s - ✔ Container atom-domains Started 5.0s - ✔ Container atom-policies-db Started 3.0s - ✔ Container atom-policies Started 5.4s - ✔ Container atom-tokens Started 5.3s - ✔ Container atom-providers Started 5.6s - ✔ Container fluxmq-node1 Started 5.9s - ✔ Container fluxmq-node2 Started 6.1s - ✔ Container fluxmq-node3 Started 6.0s - ✔ Container fluxmq-auth Started 5.5s - ✔ Container base-net Created 0.1s - ✔ Container nginx Started 6.5s +[+] Running 9/9 + ✔ Network propeller-base-net Created 0.1s + ✔ Volume propeller-atom-db-volume Created 0.1s + ✔ Volume propeller-fluxmq-node1-volume Created 0.1s + ✔ Container propeller-atom-db Started 2.5s + ✔ Container propeller-atom Started 5.1s + ✔ Container propeller-atom-ui Started 5.3s + ✔ Container propeller-fluxmq-auth Started 5.5s + ✔ Container propeller-fluxmq-node1 Started 5.9s + ✔ Container propeller-nginx Started 6.5s ``` - Docker must be installed and running diff --git a/content/docs/k8s/deploy-without-operator.mdx b/content/docs/k8s/deploy-without-operator.mdx index 953bee3..a6b04f3 100644 --- a/content/docs/k8s/deploy-without-operator.mdx +++ b/content/docs/k8s/deploy-without-operator.mdx @@ -45,9 +45,9 @@ kubectl create namespace propeller ```bash kubectl create secret generic propeller-credentials \ --namespace propeller \ - --from-literal=MANAGER_API_KEY= \ - --from-literal=PROPLET_API_KEY= \ - --from-literal=PROXY_API_KEY= + --from-literal=MANAGER_API_KEY= \ + --from-literal=PROPLET_API_KEY= \ + --from-literal=PROXY_API_KEY= ``` ### 3. Apply Deployments @@ -75,11 +75,11 @@ spec: image: ghcr.io/absmach/propeller/manager:latest env: - name: MANAGER_TENANT_ID - value: "" + value: "" - name: MANAGER_CHANNEL_ID value: "" - name: MANAGER_ENTITY_ID - value: "" + value: "" - name: MANAGER_API_KEY valueFrom: secretKeyRef: @@ -108,11 +108,11 @@ spec: image: ghcr.io/absmach/propeller/proplet:latest env: - name: PROPLET_TENANT_ID - value: "" + value: "" - name: PROPLET_CHANNEL_ID value: "" - name: PROPLET_ENTITY_ID - value: "" + value: "" - name: PROPLET_API_KEY valueFrom: secretKeyRef: @@ -141,11 +141,11 @@ spec: image: ghcr.io/absmach/propeller/proxy:latest env: - name: PROXY_TENANT_ID - value: "" + value: "" - name: PROXY_CHANNEL_ID value: "" - name: PROXY_ENTITY_ID - value: "" + value: "" - name: PROXY_API_KEY valueFrom: secretKeyRef: diff --git a/content/docs/k8s/deployment.mdx b/content/docs/k8s/deployment.mdx index effe66f..f401298 100644 --- a/content/docs/k8s/deployment.mdx +++ b/content/docs/k8s/deployment.mdx @@ -15,15 +15,15 @@ The operator is configured through command-line flags and one environment variab ### MQTT Flags -| Flag | Default | Description | -| ---------------- | ------- | ----------------------------------------------------- | -| `--mqtt-address` | — | MQTT broker URL (required, e.g. `tcp://mqtt:1883`) | -| `--domain-id` | — | Atom domain ID (required) | -| `--channel-id` | — | Atom channel ID (required) | -| `--client-id` | — | Manager client ID for MQTT authentication (required) | -| `--client-key` | — | Manager client key for MQTT authentication (required) | -| `--mqtt-qos` | `0` | MQTT QoS level (0–2) | -| `--mqtt-timeout` | `30s` | Broker operation timeout | +| Flag | Default | Description | +| ---------------- | ------- | ---------------------------------------------------- | +| `--mqtt-address` | — | MQTT broker URL (required, e.g. `tcp://mqtt:1883`) | +| `--tenant-id` | — | Atom domain / tenant ID (required) | +| `--channel-id` | — | Atom channel ID (required) | +| `--entity-id` | — | Manager entity ID for MQTT authentication (required) | +| `--api-key` | — | Manager API key for MQTT authentication (required) | +| `--mqtt-qos` | `0` | MQTT QoS level (0–2) | +| `--mqtt-timeout` | `30s` | Broker operation timeout | ### Proplet Monitoring Flags @@ -84,10 +84,10 @@ Set `imagePullPolicy: Never` in `config/manager/manager.yaml` when using a local ```bash go run ./cmd/main.go \ --mqtt-address="tcp://localhost:1883" \ - --domain-id="" \ + --tenant-id="" \ --channel-id="" \ - --client-id="" \ - --client-key="" \ + --entity-id="" \ + --api-key="" \ --metrics-secure=false ``` @@ -97,16 +97,16 @@ The operator uses the current `kubectl` context for cluster access. The operator requires a `ClusterRole` with the following permissions: -| API Group | Resources | Verbs | -| ----------------------------------------- | ----------------------------------------------------------------------- | ----------------------------------------------- | -| `""` (core) | `configmaps` | create, delete, get, list, patch, update | -| `""` (core) | `pods` | get, list | -| `""` (core) | `secrets` | get | -| `apps` | `deployments` | create, delete, get, list, patch, update, watch | -| `batch` | `jobs` | create, delete, get, list, patch, update, watch | -| `propeller.propeller.abstractmachines.fr` | `tasks`, `proplets`, `propellerjobs`, `federatedjobs`, `trainingrounds` | create, delete, get, list, patch, update, watch | -| `propeller.propeller.abstractmachines.fr` | `*/status` | get, patch, update | -| `propeller.propeller.abstractmachines.fr` | `*/finalizers` | update | +| API Group | Resources | Verbs | +| -------------------------------- | ----------------------------------------------------------------------- | ----------------------------------------------- | +| `""` (core) | `configmaps` | create, delete, get, list, patch, update | +| `""` (core) | `pods` | get, list | +| `""` (core) | `secrets` | get | +| `apps` | `deployments` | create, delete, get, list, patch, update, watch | +| `batch` | `jobs` | create, delete, get, list, patch, update, watch | +| `propeller.propeller.absmach.eu` | `tasks`, `proplets`, `propellerjobs`, `federatedjobs`, `trainingrounds` | create, delete, get, list, patch, update, watch | +| `propeller.propeller.absmach.eu` | `*/status` | get, patch, update | +| `propeller.propeller.absmach.eu` | `*/finalizers` | update | ### Container Security @@ -144,3 +144,104 @@ make setup-test-e2e # creates a Kind cluster make test-e2e # runs Ginkgo e2e suite make cleanup-test-e2e # tears down the Kind cluster ``` + +### Sample Configurations + +Sample YAML files covering all CRDs are in `config/samples/`: + +| File | CRD | Description | +| ------------------------------------ | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | +| `propeller_v1_proplet.yaml` | Proplet (k8s) | K8s-backed proplet managed as a Deployment | +| `propeller_v1_proplet_external.yaml` | Proplet (external) | External device proplet via MQTT | +| `propeller_v1_task.yaml` | Task (file) | WASM file dispatched via MQTT | +| `propeller_v1_task_with_image.yaml` | Task (registry) | WASM OCI image dispatched via MQTT | +| `propeller_v1_task_k8s_job.yaml` | Task (k8s Job) | Container image run as K8s Job (referenced in `kustomization.yaml` but not yet shipped — create one from the inline example below) | +| `propeller_v1_task_broadcast.yaml` | Task (broadcast) | WASM sent to all proplets | +| `propeller_v1_task_recurring.yaml` | Task (cron) | Cron-scheduled recurring task | +| `propeller_v1_task_monitoring.yaml` | Task (monitoring) | Task with metrics collection | +| `propeller_v1_task_dag_a.yaml` | Task (DAG target) | DAG dependency target | +| `propeller_v1_task_dag.yaml` | Task (DAG dependent) | Task with dependsOn | +| `propeller_v1_propellerjob.yaml` | PropellerJob | Batch of parallel tasks | +| `propeller_v1_federatedjob.yaml` | FederatedJob | Multi-round FL experiment | + +### E2E Tests + +The operator ships with a Ginkgo-based e2e test suite. Two modes are available: + +```bash +# Kind (creates a temporary cluster automatically) +make test-e2e + +# k3d (uses your existing k3d cluster) +make test-e2e-k3d +``` + +### Manual Testing on k3d + +1. Create a k3d cluster: + + ```bash + k3d cluster create propeller + ``` + +2. Install CRDs: + + ```bash + make install + ``` + +3. Start the operator: + + ```bash + make run ARGS="--mqtt-address='tcp://your-mqtt:1883' \ + --tenant-id='' \ + --channel-id='' \ + --entity-id='' \ + --api-key=''" + ``` + +4. Create a proplet and run a task: + + ```bash + # k8s proplet + kubectl apply -f config/samples/propeller_v1_proplet.yaml + + # WASM task via MQTT + kubectl apply -f config/samples/propeller_v1_task.yaml + + # Container task via K8s Job — if the sample file is missing, create inline: + cat < The propeller-manager, propeller-proplet, and propeller-proxy containers start but will fail to connect until you provision credentials. That is expected at this stage. @@ -491,10 +473,10 @@ args: - --mqtt-address=tcp://host.k3d.internal:1883 - --mqtt-qos=2 - --mqtt-timeout=30s - - --domain-id=3053156a-1994-4776-9e18-8c5d8883647c + - --tenant-id=3053156a-1994-4776-9e18-8c5d8883647c - --channel-id=c19ded40-eeec-448e-86e9-42f490c766a4 - - --client-id=0a96e62c-da03-45eb-9699-b7d8dfa843c9 - - --client-key=8bd64edf-6cec-4cd8-aebd-a8e9695c8d90 + - --entity-id=0a96e62c-da03-45eb-9699-b7d8dfa843c9 + - --api-key=8bd64edf-6cec-4cd8-aebd-a8e9695c8d90 ``` Also add `imagePullPolicy: Never` immediately after the `image:` line so k3d uses the locally-loaded image without trying to pull it from a registry: @@ -569,11 +551,11 @@ Your output should look like this: cd config/manager && ./bin/kustomize edit set image controller=propeller-k8s-operator:latest ./bin/kustomize build config/default | kubectl apply -f - namespace/propeller-k8s-operator-system created -customresourcedefinition.apiextensions.k8s.io/federatedjobs.propeller.propeller.abstractmachines.fr unchanged -customresourcedefinition.apiextensions.k8s.io/propellerjobs.propeller.propeller.abstractmachines.fr unchanged -customresourcedefinition.apiextensions.k8s.io/proplets.propeller.propeller.abstractmachines.fr unchanged -customresourcedefinition.apiextensions.k8s.io/tasks.propeller.propeller.abstractmachines.fr unchanged -customresourcedefinition.apiextensions.k8s.io/trainingrounds.propeller.propeller.abstractmachines.fr unchanged +customresourcedefinition.apiextensions.k8s.io/federatedjobs.propeller.propeller.absmach.eu unchanged +customresourcedefinition.apiextensions.k8s.io/propellerjobs.propeller.propeller.absmach.eu unchanged +customresourcedefinition.apiextensions.k8s.io/proplets.propeller.propeller.absmach.eu unchanged +customresourcedefinition.apiextensions.k8s.io/tasks.propeller.propeller.absmach.eu unchanged +customresourcedefinition.apiextensions.k8s.io/trainingrounds.propeller.propeller.absmach.eu unchanged serviceaccount/propeller-k8s-operator-controller-manager created ... deployment.apps/propeller-k8s-operator-controller-manager created @@ -620,7 +602,7 @@ Register the running docker proplet as an external Proplet CRD. The `spec.connec ```bash kubectl apply -n propeller-workloads -f - <<'EOF' -apiVersion: propeller.propeller.abstractmachines.fr/v1 +apiVersion: propeller.propeller.absmach.eu/v1 kind: Proplet metadata: name: docker-proplet @@ -645,7 +627,7 @@ EOF Your output should look like this: ```bash -proplet.propeller.propeller.abstractmachines.fr/docker-proplet created +proplet.propeller.propeller.absmach.eu/docker-proplet created ``` > Replace `domainId`, `channelId`, `clientId`, and `clientKey` with your provisioned values. @@ -744,7 +726,7 @@ Create a Task that targets the docker proplet. This uses the addition WASM modul ```bash kubectl apply -n propeller-workloads -f - <<'EOF' -apiVersion: propeller.propeller.abstractmachines.fr/v1 +apiVersion: propeller.propeller.absmach.eu/v1 kind: Task metadata: name: addition-wasm @@ -764,7 +746,7 @@ EOF Your output should look like this: ```bash -task.propeller.propeller.abstractmachines.fr/addition-wasm created +task.propeller.propeller.absmach.eu/addition-wasm created ``` Check task status: diff --git a/content/docs/k8s/features.mdx b/content/docs/k8s/features.mdx index d0a4214..789b260 100644 --- a/content/docs/k8s/features.mdx +++ b/content/docs/k8s/features.mdx @@ -17,7 +17,7 @@ A `Proplet` resource represents one worker node. The `spec.type` field selects b ![Proplet Lifecycle State Machine](../images/kubernetes-operator/proplet-lifecycle.svg) -*Figure: Proplet state machine showing transitions between Initializing, Running, and Offline phases. A proplet starts in Initializing, moves to Running when heartbeats are received (external) or when the Deployment is ready (k8s), and transitions to Offline when heartbeats stop. When heartbeats resume, the proplet returns to Running.* +_Figure: Proplet state machine showing transitions between Initializing, Running, and Offline phases. A proplet starts in Initializing, moves to Running when heartbeats are received (external) or when the Deployment is ready (k8s), and transitions to Offline when heartbeats stop. When heartbeats resume, the proplet returns to Running._ - **Initializing** — default phase on creation; no heartbeats received yet, or k8s Deployment not yet ready. - **Running** — heartbeat received within `--last-seen-threshold` (default 30 s) for external proplets, or all Deployment replicas ready for k8s proplets. @@ -25,32 +25,32 @@ A `Proplet` resource represents one worker node. The `spec.type` field selects b ### Proplet Conditions -| Condition | Meaning | -| ------------- | ---------------------------------------------------------------- | -| `Ready` | Proplet can accept tasks | -| `Connected` | MQTT heartbeat is current (external) or pod is available (k8s) | -| `Healthy` | Proplet is processing tasks without errors | +| Condition | Meaning | +| ----------- | -------------------------------------------------------------- | +| `Ready` | Proplet can accept tasks | +| `Connected` | MQTT heartbeat is current (external) or pod is available (k8s) | +| `Healthy` | Proplet is processing tasks without errors | ### Proplet Spec Reference -| Field | Type | Required | Default | Description | -| ---------------------------------------- | ------------------------------------------------ | ----------- | --------- | ----------------------------------------------------------------------------------------------- | -| `type` | `k8s` \ | `external` | Yes | — | -| `k8s.image` | string | Yes (k8s) | — | Container image for the proplet pod | -| `k8s.logLevel` | `debug` \ | `info` \ | `warn` \ | `error` | -| `k8s.replicas` | int32 | No | `1` | Number of pod replicas (0–100) | -| `k8s.pluginDir` | string | No | — | Path inside the container where WASM plugin files are loaded from (`PROPLET_PLUGIN_DIR`) | -| `external.deviceType` | string | No | — | Device category label used for task matching (e.g. `raspberry-pi-4`) | -| `external.capabilities` | []string | No | — | Capability strings the device advertises (e.g. `wasm`, `gpu`) | -| `resources` | ResourceList | No | — | Kubernetes CPU and memory requests/limits applied to the proplet pod | -| `connectionConfig.mqttAddress` | string | Yes | — | MQTT broker URL (e.g. `tcp://mqtt:1883`) | -| `connectionConfig.domainId` | string | Yes | — | Atom domain ID | -| `connectionConfig.channelId` | string | Yes | — | Atom channel ID | -| `connectionConfig.clientId` | string | Yes | — | Atom client credential ID; must match `proplet_id` in alive messages for external proplets | -| `connectionConfig.clientKey` | string | No* | — | Atom client secret (inline). Exactly one of `clientKey` or `clientKeySecretRef` must be set | -| `connectionConfig.clientKeySecretRef` | SecretKeySelector | No* | — | Reference to a Kubernetes Secret holding the client secret. Mutually exclusive with `clientKey` | -| `connectionConfig.mqttQos` | uint8 | No | `0` | MQTT QoS level (0–2) | -| `connectionConfig.mqttTimeout` | duration | No | `30s` | Broker operation timeout | +| Field | Type | Required | Default | Description | +| ---------------------------------- | -------------------------------------- | --------- | ------- | ------------------------------------------------------------------------------------------ | +| `type` | `k8s` \| `external` | Yes | — | Proplet type: K8s Deployment or external device | +| `k8s.image` | string | Yes (k8s) | — | Container image for the proplet pod | +| `k8s.logLevel` | `debug` \| `info` \| `warn` \| `error` | No | `info` | Log level for the proplet pod | +| `k8s.replicas` | int32 | No | `1` | Number of pod replicas (0–100) | +| `k8s.pluginDir` | string | No | — | Path inside the container where WASM plugin files are loaded from (`PROPLET_PLUGIN_DIR`) | +| `external.deviceType` | string | No | — | Device category label used for task matching (e.g. `raspberry-pi-4`) | +| `external.capabilities` | []string | No | — | Capability strings the device advertises (e.g. `wasm`, `gpu`) | +| `resources` | ResourceList | No | — | Kubernetes CPU and memory requests/limits applied to the proplet pod | +| `connectionConfig.mqttAddress` | string | Yes | — | MQTT broker URL (e.g. `tcp://mqtt:1883`) | +| `connectionConfig.tenantId` | string | Yes | — | Atom tenant ID | +| `connectionConfig.channelId` | string | Yes | — | Atom channel ID | +| `connectionConfig.entityId` | string | Yes | — | Atom entity credential ID; must match `proplet_id` in alive messages for external proplets | +| `connectionConfig.apiKey` | string | No\* | — | Atom API key (inline). Exactly one of `apiKey` or `apiKeySecretRef` must be set | +| `connectionConfig.apiKeySecretRef` | SecretKeySelector | No\* | — | Reference to a Kubernetes Secret holding the API key. Mutually exclusive with `apiKey` | +| `connectionConfig.mqttQos` | uint8 | No | `0` | MQTT QoS level (0–2) | +| `connectionConfig.mqttTimeout` | duration | No | `30s` | Broker operation timeout | ### Proplet Status Reference @@ -70,7 +70,6 @@ A `Proplet` resource represents one worker node. The `spec.type` field selects b | `metadata` | Device metadata reported in alive messages: `description`, `tags`, `location`, `ip`, `environment`, `os`, `hostname`, `cpuArch`, `totalMemoryBytes`, `propletVersion`, `wasmRuntime` | | `latestMetrics` | Most recent metrics sample: `cpuMilliPercent` (CPU × 1000), `memoryMilliPercent` (memory × 1000), `memoryBytes`, `timestamp` | - ## Task A `Task` resource defines a single WASM workload. The `TaskReconciler` drives it through a phase machine, selecting a proplet, dispatching execution, and capturing the result. @@ -79,27 +78,28 @@ A `Task` resource defines a single WASM workload. The `TaskReconciler` drives it ![Task Lifecycle State Machine](../images/kubernetes-operator/task-lifecycle.svg) -*Figure: Task state machine showing all valid transitions. Tasks begin in pending and move directly to running when dispatched to a proplet. From running, tasks can complete successfully (completed), fail (failed), or be stopped (interrupted). The operator enforces valid transitions and supports retry/restart flows from terminal states back to pending.* +_Figure: Task state machine showing all valid transitions. Tasks begin in pending and move directly to running when dispatched to a proplet. From running, tasks can complete successfully (completed), fail (failed), or be stopped (interrupted). The operator enforces valid transitions and supports retry/restart flows from terminal states back to pending._ Valid state transitions are enforced by the operator. Task completion and failure arrive via `control/proplet/results` MQTT messages; the operator validates each proposed transition before applying it. -| From | Allowed next states | -| ------------- | ---------------------------------------------------------- | -| `pending` | `scheduled`, `running`, `completed`, `failed`, `skipped` | -| `scheduled` | `running`, `completed`, `failed`, `skipped` | -| `running` | `completed`, `failed`, `interrupted` | -| `completed` | `pending` (for restart or recurring tasks) | -| `failed` | `pending` (for retry) | -| `interrupted` | `pending` (for resume) | -| `skipped` | — (terminal) | +| From | Allowed next states | +| ------------- | -------------------------------------------------------- | +| `pending` | `scheduled`, `running`, `completed`, `failed`, `skipped` | +| `scheduled` | `running`, `completed`, `failed`, `skipped` | +| `running` | `completed`, `failed`, `interrupted` | +| `completed` | `pending` (for restart or recurring tasks) | +| `failed` | `pending` (for retry) | +| `interrupted` | `pending` (for resume) | +| `skipped` | — (terminal) | ### Execution Paths ![Task Execution Paths](../images/kubernetes-operator/task-execution-paths.svg) -*Figure: Comparison of task execution paths for external vs k8s proplets. External proplets receive tasks via MQTT and return results through the broker. K8s proplets run as Kubernetes Jobs with ConfigMaps for task configuration and multiple result extraction methods (Job annotations, ConfigMaps, Secrets, or Pod annotations).* +_Figure: Comparison of task execution paths for external vs k8s proplets. External proplets receive tasks via MQTT and return results through the broker. K8s proplets run as Kubernetes Jobs with ConfigMaps for task configuration and multiple result extraction methods (Job annotations, ConfigMaps, Secrets, or Pod annotations)._ **External proplet path:** + 1. `TaskReconciler` picks up the new Task and resolves the target proplet by `spec.propletSelector.propletId`. 2. The reconciler looks up the Proplet CR and confirms its type is `external`. 3. It publishes an MQTT message to `control/manager/start` containing the full task spec: function name, WASM file or image URL, inputs, environment variables, scheduling parameters, monitoring profile, and the assigned proplet ID. @@ -108,6 +108,7 @@ Valid state transitions are enforced by the operator. Task completion and failur 6. The `TaskReconciler`'s MQTT result handler receives the message, stores the output in `status.results`, and sets `status.phase = completed`. **k8s proplet path:** + 1. `TaskReconciler` resolves the proplet and confirms its type is `k8s`. 2. It creates a `ConfigMap` containing task environment variables (function name, inputs, monitoring settings, etc.) and an optional `wasm_file_provided` key if a WASM file is embedded. 3. It creates a Kubernetes `Job` whose pod runs the image specified in `spec.imageUrl`. @@ -131,68 +132,67 @@ If no proplet satisfies the filter, the task remains in `pending` and is requeue ### Task Spec Reference -| Field | Type | Required | Default | Description | -| ------------------------------------------ | ----------------------------- | ------------ | ---------- | ------------------------------------------------------------------------------------------------ | -| `name` | string | Yes | — | Task name (max 253 chars) | -| `functionName` | string | No | — | Exported WASM function to invoke; defaults to the task name if omitted | -| `kind` | `standard` \ | `federated` | No | `standard` | -| `file` | bytes | No | — | WASM binary encoded as a base64 string in YAML | -| `imageUrl` | string | No | — | OCI image reference containing the WASM module; required for k8s proplets | -| `cliArgs` | []string | No | — | Command-line arguments passed to the WASM runtime | -| `inputs` | []string | No | — | Inputs passed to the function; numeric values are accepted and coerced | -| `env` | map[string]string | No | — | Environment variables available in the module | -| `broadcast` | bool | No | `false` | Send the task to all available proplets simultaneously instead of one | -| `mode` | `infer` \ | `train` | No | — | -| `metadata` | object | No | — | Arbitrary key-value JSON attached to the task and forwarded to the proplet | -| `propletSelector.propletId` | string | No | — | Target a specific Proplet by its Kubernetes resource name | -| `propletSelector.matchLabels` | map[string]string | No | — | Target Proplets matching all labels | -| `propletSelector.matchDeviceTypes` | []string | No | — | Target Proplets with a matching device type | -| `propletSelector.matchCapabilities` | []string | No | — | Target Proplets advertising all listed capabilities | -| `preferredPropletType` | `k8s` \ | `external` \ | `any` | No | -| `resourceRequirements.cpu` | string | No | — | CPU resource request/limit applied to the k8s Job pod (e.g. `100m`); not used for scheduling | -| `resourceRequirements.memory` | string | No | — | Memory resource request/limit applied to the k8s Job pod (e.g. `256Mi`); not used for scheduling | -| `resourceRequirements.custom` | map[string]string | No | — | Custom resource constraints passed to the k8s Job pod | -| `daemon` | bool | No | `false` | Run indefinitely; sets Kubernetes RestartPolicy to Always | -| `restartPolicy` | Kubernetes RestartPolicy | No | — | Override pod restart policy for k8s tasks | -| `encrypted` | bool | No | `false` | Signal that the WASM binary is encrypted | -| `kbsResourcePath` | string | No | — | Key Broker Service path for decrypting the module | -| `dependsOn` | []string | No | — | Task IDs that must complete before this task is scheduled | -| `runIf` | `success` \ | `failure` | No | — | -| `workflowId` | string | No | — | Groups tasks into a workflow | -| `jobId` | string | No | — | Associates this task with a PropellerJob | -| `schedule` | string | No | — | Cron expression for recurring execution | -| `isRecurring` | bool | No | `false` | Enable recurring execution via `schedule` | -| `timezone` | string | No | `UTC` | Timezone for cron evaluation | -| `priority` | int | No | `50` | Scheduling priority (0–100; higher = more urgent) | -| `monitoringProfile.enabled` | bool | No | `false` | Enable metrics collection during execution | -| `monitoringProfile.interval` | duration | No | — | Metrics collection interval (e.g. `10s`, `1m`) | -| `monitoringProfile.collectCpu` | bool | No | — | Collect CPU usage | -| `monitoringProfile.collectMemory` | bool | No | — | Collect memory usage | -| `monitoringProfile.collectDiskIo` | bool | No | — | Collect disk I/O | -| `monitoringProfile.collectThreads` | bool | No | — | Collect thread count | -| `monitoringProfile.collectFileDescriptors` | bool | No | — | Collect file descriptor count | -| `monitoringProfile.historySize` | int | No | — | Number of history entries to retain | -| `monitoringProfile.exportToMqtt` | bool | No | — | Stream metrics to MQTT | -| `monitoringProfile.retainHistory` | bool | No | — | Retain metrics history after task completes | +| Field | Type | Required | Default | Description | +| ------------------------------------------ | ------------------------ | ------------- | ------- | ------------------------------------------------------------------------------------------------ | +| `name` | string | Yes | — | Task name (max 253 chars) | +| `functionName` | string | No | — | Exported WASM function to invoke; defaults to the task name if omitted | +| `kind` | `standard` \ | `federated` | No | `standard` | +| `file` | bytes | No | — | WASM binary encoded as a base64 string in YAML | +| `imageUrl` | string | No | — | OCI image reference containing the WASM module; required for k8s proplets | +| `cliArgs` | []string | No | — | Command-line arguments passed to the WASM runtime | +| `inputs` | []string | No | — | Inputs passed to the function; numeric values are accepted and coerced | +| `env` | map[string]string | No | — | Environment variables available in the module | +| `broadcast` | bool | No | `false` | Send the task to all available proplets simultaneously instead of one | +| `mode` | `infer` \ | `train` | No | — | +| `metadata` | object | No | — | Arbitrary key-value JSON attached to the task and forwarded to the proplet | +| `propletSelector.propletId` | string | No | — | Target a specific Proplet by its Kubernetes resource name | +| `propletSelector.matchLabels` | map[string]string | No | — | Target Proplets matching all labels | +| `propletSelector.matchDeviceTypes` | []string | No | — | Target Proplets with a matching device type | +| `propletSelector.matchCapabilities` | []string | No | — | Target Proplets advertising all listed capabilities | +| `preferredPropletType` | `k8s` \ | `external` \ | `any` | No | +| `resourceRequirements.cpu` | string | No | — | CPU resource request/limit applied to the k8s Job pod (e.g. `100m`); not used for scheduling | +| `resourceRequirements.memory` | string | No | — | Memory resource request/limit applied to the k8s Job pod (e.g. `256Mi`); not used for scheduling | +| `resourceRequirements.custom` | map[string]string | No | — | Custom resource constraints passed to the k8s Job pod | +| `daemon` | bool | No | `false` | Run indefinitely; sets Kubernetes RestartPolicy to Always | +| `restartPolicy` | Kubernetes RestartPolicy | No | — | Override pod restart policy for k8s tasks | +| `encrypted` | bool | No | `false` | Signal that the WASM binary is encrypted | +| `kbsResourcePath` | string | No | — | Key Broker Service path for decrypting the module | +| `dependsOn` | []string | No | — | Task IDs that must complete before this task is scheduled | +| `runIf` | `success` \ | `failure` | No | — | +| `workflowId` | string | No | — | Groups tasks into a workflow | +| `jobId` | string | No | — | Associates this task with a PropellerJob | +| `schedule` | string | No | — | Cron expression for recurring execution | +| `isRecurring` | bool | No | `false` | Enable recurring execution via `schedule` | +| `timezone` | string | No | `UTC` | Timezone for cron evaluation | +| `priority` | int | No | `50` | Scheduling priority (0–100; higher = more urgent) | +| `monitoringProfile.enabled` | bool | No | `false` | Enable metrics collection during execution | +| `monitoringProfile.interval` | duration | No | — | Metrics collection interval (e.g. `10s`, `1m`) | +| `monitoringProfile.collectCpu` | bool | No | — | Collect CPU usage | +| `monitoringProfile.collectMemory` | bool | No | — | Collect memory usage | +| `monitoringProfile.collectDiskIo` | bool | No | — | Collect disk I/O | +| `monitoringProfile.collectThreads` | bool | No | — | Collect thread count | +| `monitoringProfile.collectFileDescriptors` | bool | No | — | Collect file descriptor count | +| `monitoringProfile.historySize` | int | No | — | Number of history entries to retain | +| `monitoringProfile.exportToMqtt` | bool | No | — | Stream metrics to MQTT | +| `monitoringProfile.retainHistory` | bool | No | — | Retain metrics history after task completes | ### Task Status Reference -| Field | Description | -| ------------------------- | --------------------------------------------------------------------------------------- | -| `phase` | `pending`, `scheduled`, `running`, `completed`, `failed`, `skipped`, or `interrupted` | -| `assignedProplet` | Name of the Proplet executing this task | -| `createdAt` | When the Task was first processed | -| `startedAt` | When execution began | -| `finishedAt` | When execution ended | -| `updatedAt` | Last status update timestamp | -| `nextRun` | Next scheduled execution time (recurring tasks) | -| `results` | Return value from the WASM function | -| `error` | Error message when `phase` is `failed` | -| `conditions[Scheduled]` | `True` when a proplet has been assigned | -| `conditions[Started]` | `True` when execution began | -| `conditions[Completed]` | `True` when execution finished successfully | -| `latestMetrics` | Most recent CPU and memory sample from the proplet during execution | - +| Field | Description | +| ----------------------- | ------------------------------------------------------------------------------------- | +| `phase` | `pending`, `scheduled`, `running`, `completed`, `failed`, `skipped`, or `interrupted` | +| `assignedProplet` | Name of the Proplet executing this task | +| `createdAt` | When the Task was first processed | +| `startedAt` | When execution began | +| `finishedAt` | When execution ended | +| `updatedAt` | Last status update timestamp | +| `nextRun` | Next scheduled execution time (recurring tasks) | +| `results` | Return value from the WASM function | +| `error` | Error message when `phase` is `failed` | +| `conditions[Scheduled]` | `True` when a proplet has been assigned | +| `conditions[Started]` | `True` when execution began | +| `conditions[Completed]` | `True` when execution finished successfully | +| `latestMetrics` | Most recent CPU and memory sample from the proplet during execution | ## PropellerJob @@ -202,33 +202,32 @@ A `PropellerJob` groups multiple Tasks into a single managed batch. The `Propell ![PropellerJob Lifecycle State Machine](../images/kubernetes-operator/propellerjob-lifecycle.svg) -*Figure: PropellerJob state machine. Jobs start in Pending, transition to Running when child Tasks are created, and finally move to Completed when all tasks finish successfully or to Failed if any task fails.* +_Figure: PropellerJob state machine. Jobs start in Pending, transition to Running when child Tasks are created, and finally move to Completed when all tasks finish successfully or to Failed if any task fails._ ### PropellerJob Spec Reference -| Field | Type | Required | Default | Description | -| --------------- | ------------------------------------------------- | -------------- | -------------- | ------------------------------------------- | -| `name` | string | Yes | — | Job name | -| `executionMode` | `parallel` \ | `sequential` \ | `configurable` | No | -| `tasks` | []TaskSpec | No | — | Inline task definitions to create | -| `taskRefs` | []string | No | — | Names of existing Task resources to include | +| Field | Type | Required | Default | Description | +| --------------- | ------------- | --------------- | -------------- | ------------------------------------------- | +| `name` | string | Yes | — | Job name | +| `executionMode` | `parallel` \ | `sequential` \ | `configurable` | No | +| `tasks` | []TaskSpec | No | — | Inline task definitions to create | +| `taskRefs` | []string | No | — | Names of existing Task resources to include | ### PropellerJob Status Reference -| Field | Description | -| ------------------- | ------------------------------------------------ | -| `phase` | `Pending`, `Running`, `Completed`, or `Failed` | -| `startTime` | When the job began | -| `finishTime` | When the job completed | -| `createdAt` | When the PropellerJob was first processed | -| `updatedAt` | Last status update timestamp | -| `taskCount` | Total number of tasks | -| `completedCount` | Tasks in `completed` phase | -| `failedCount` | Tasks in `failed` phase | -| `skippedCount` | Tasks in `skipped` phase | -| `interruptedCount` | Tasks in `interrupted` phase | -| `conditions` | Kubernetes condition array for detailed state | - +| Field | Description | +| ------------------ | ---------------------------------------------- | +| `phase` | `Pending`, `Running`, `Completed`, or `Failed` | +| `startTime` | When the job began | +| `finishTime` | When the job completed | +| `createdAt` | When the PropellerJob was first processed | +| `updatedAt` | Last status update timestamp | +| `taskCount` | Total number of tasks | +| `completedCount` | Tasks in `completed` phase | +| `failedCount` | Tasks in `failed` phase | +| `skippedCount` | Tasks in `skipped` phase | +| `interruptedCount` | Tasks in `interrupted` phase | +| `conditions` | Kubernetes condition array for detailed state | ## FederatedJob @@ -238,35 +237,34 @@ A `FederatedJob` coordinates a multi-round federated learning experiment across ![FederatedJob Lifecycle State Machine](../images/kubernetes-operator/federatedjob-lifecycle.svg) -*Figure: FederatedJob state machine for multi-round federated learning experiments. Jobs begin in Pending, move to Running as TrainingRounds are created and executed sequentially, and complete when all rounds finish aggregation. Jobs fail if any round times out or encounters an aggregation error.* +_Figure: FederatedJob state machine for multi-round federated learning experiments. Jobs begin in Pending, move to Running as TrainingRounds are created and executed sequentially, and complete when all rounds finish aggregation. Jobs fail if any round times out or encounters an aggregation error._ ### FederatedJob Spec Reference -| Field | Type | Required | Description | -| ------------------------ | ------------------- | ---------- | ------------------------------------------------------------------- | -| `experimentId` | string | No | Identifier for this federated learning experiment | -| `modelRef` | string | No | Reference to the initial global model | -| `taskWasmImage` | string | Yes | OCI image containing the WASM training task | -| `participants` | []ParticipantSpec | Yes | List of proplet IDs that participate in each round | -| `hyperparams` | object | No | Arbitrary hyperparameter values passed to each participant | -| `kOfN` | int | Yes | Minimum number of participant updates required before aggregation | -| `timeoutSeconds` | int | No | Maximum seconds per round before failing | -| `rounds.total` | int | Yes | Total number of training rounds | -| `rounds.strategy` | string | No | Round execution strategy | -| `aggregator.algorithm` | string | No | Aggregation algorithm (`fedavg`, `concat`) | -| `aggregator.config` | object | No | Algorithm-specific configuration | +| Field | Type | Required | Description | +| ---------------------- | ----------------- | -------- | ----------------------------------------------------------------- | +| `experimentId` | string | No | Identifier for this federated learning experiment | +| `modelRef` | string | No | Reference to the initial global model | +| `taskWasmImage` | string | Yes | OCI image containing the WASM training task | +| `participants` | []ParticipantSpec | Yes | List of proplet IDs that participate in each round | +| `hyperparams` | object | No | Arbitrary hyperparameter values passed to each participant | +| `kOfN` | int | Yes | Minimum number of participant updates required before aggregation | +| `timeoutSeconds` | int | No | Maximum seconds per round before failing | +| `rounds.total` | int | Yes | Total number of training rounds | +| `rounds.strategy` | string | No | Round execution strategy | +| `aggregator.algorithm` | string | No | Aggregation algorithm (`fedavg`, `concat`) | +| `aggregator.config` | object | No | Algorithm-specific configuration | ### FederatedJob Status Reference -| Field | Description | -| ---------------------- | -------------------------------------------------- | -| `phase` | `Pending`, `Running`, `Completed`, or `Failed` | -| `currentRound` | Index of the round currently executing | -| `completedRounds` | Number of rounds that have completed aggregation | -| `aggregatedModelRef` | Reference to the final aggregated model | -| `participants` | Per-participant status across all rounds | -| `conditions` | Kubernetes condition array | - +| Field | Description | +| -------------------- | ------------------------------------------------ | +| `phase` | `Pending`, `Running`, `Completed`, or `Failed` | +| `currentRound` | Index of the round currently executing | +| `completedRounds` | Number of rounds that have completed aggregation | +| `aggregatedModelRef` | Reference to the final aggregated model | +| `participants` | Per-participant status across all rounds | +| `conditions` | Kubernetes condition array | ## TrainingRound @@ -276,33 +274,32 @@ A `TrainingRound` represents a single round within a `FederatedJob`. The `Traini ![TrainingRound Lifecycle State Machine](../images/kubernetes-operator/traininground-lifecycle.svg) -*Figure: TrainingRound state machine showing the four-phase lifecycle. Rounds start in Pending, move to Running when participant Tasks are created, transition to Aggregating once k-of-n updates are received, and finally reach Completed after successful model aggregation. Rounds fail on timeout or aggregation errors.* +_Figure: TrainingRound state machine showing the four-phase lifecycle. Rounds start in Pending, move to Running when participant Tasks are created, transition to Aggregating once k-of-n updates are received, and finally reach Completed after successful model aggregation. Rounds fail on timeout or aggregation errors._ ### TrainingRound Spec Reference -| Field | Type | Description | -| ------------------- | ---------------------- | ---------------------------------------------------- | -| `roundId` | string | Round identifier | -| `federatedJobRef` | LocalObjectReference | Parent FederatedJob | -| `modelRef` | string | Global model reference for this round | -| `taskWasmImage` | string | OCI image containing the participant training task | -| `participants` | []string | Proplet IDs to include in this round | -| `hyperparams` | object | Hyperparameter values forwarded to tasks | -| `kOfN` | int | Minimum updates needed before aggregation | -| `timeoutSeconds` | int | Round timeout in seconds | +| Field | Type | Description | +| ----------------- | -------------------- | -------------------------------------------------- | +| `roundId` | string | Round identifier | +| `federatedJobRef` | LocalObjectReference | Parent FederatedJob | +| `modelRef` | string | Global model reference for this round | +| `taskWasmImage` | string | OCI image containing the participant training task | +| `participants` | []string | Proplet IDs to include in this round | +| `hyperparams` | object | Hyperparameter values forwarded to tasks | +| `kOfN` | int | Minimum updates needed before aggregation | +| `timeoutSeconds` | int | Round timeout in seconds | ### TrainingRound Status Reference -| Field | Description | -| ------------------------ | --------------------------------------------------------------------- | -| `phase` | `Pending`, `Running`, `Aggregating`, `Completed`, or `Failed` | -| `startTime`, `endTime` | Round timing | -| `updatesReceived` | Number of participant updates collected so far | -| `updatesRequired` | The k-of-n threshold | -| `participants` | Per-participant status (propletId, taskRef, status, updateReceived) | -| `aggregatedModelRef` | Reference to the round's aggregated result | -| `conditions` | Kubernetes condition array | - +| Field | Description | +| ---------------------- | ------------------------------------------------------------------- | +| `phase` | `Pending`, `Running`, `Aggregating`, `Completed`, or `Failed` | +| `startTime`, `endTime` | Round timing | +| `updatesReceived` | Number of participant updates collected so far | +| `updatesRequired` | The k-of-n threshold | +| `participants` | Per-participant status (propletId, taskRef, status, updateReceived) | +| `aggregatedModelRef` | Reference to the round's aggregated result | +| `conditions` | Kubernetes condition array | ## Scheduler @@ -314,28 +311,27 @@ The scheduler runs inside the `TaskReconciler` to select a proplet for each task ![Scheduler Algorithm](../images/kubernetes-operator/scheduler-algorithm.svg) -*Figure: Three-phase scheduler algorithm. First, the Filter phase removes proplets that don't meet selector criteria or aren't in Running phase. Then, the Score phase assigns numeric scores to candidates (the default round-robin scorer assigns 0.1 to the next-in-line proplet and 1.0 to others). Finally, the Pick phase selects the lowest-scoring candidate for task assignment.* +_Figure: Three-phase scheduler algorithm. First, the Filter phase removes proplets that don't meet selector criteria or aren't in Running phase. Then, the Score phase assigns numeric scores to candidates (the default round-robin scorer assigns 0.1 to the next-in-line proplet and 1.0 to others). Finally, the Pick phase selects the lowest-scoring candidate for task assignment._ The default implementation is **round-robin**: the proplet immediately after the last-selected one receives a score of 0.1; all others receive 1.0. This distributes tasks evenly across available proplets over time. Errors returned by the scheduler: -| Error | Meaning | -| ------------------ | ------------------------------------------------ | -| `ErrNoProplet` | No proplets exist in the namespace | -| `ErrNoCandidates` | No proplets satisfy the task's selector criteria | - +| Error | Meaning | +| ----------------- | ------------------------------------------------ | +| `ErrNoProplet` | No proplets exist in the namespace | +| `ErrNoCandidates` | No proplets satisfy the task's selector criteria | ## Owner References and Garbage Collection The operator sets Kubernetes owner references on all resources it creates: -| Parent | Owned resources | -| ---------------------- | ------------------------------------------- | -| `PropellerJob` | `Task` CRs created from inline task specs | -| `FederatedJob` | `TrainingRound` CRs | -| `TrainingRound` | `Task` CRs (one per participant) | -| `Proplet` (k8s type) | `Deployment` | -| `Task` (k8s proplet) | `Job`, `ConfigMap` | +| Parent | Owned resources | +| -------------------- | ----------------------------------------- | +| `PropellerJob` | `Task` CRs created from inline task specs | +| `FederatedJob` | `TrainingRound` CRs | +| `TrainingRound` | `Task` CRs (one per participant) | +| `Proplet` (k8s type) | `Deployment` | +| `Task` (k8s proplet) | `Job`, `ConfigMap` | When a parent resource is deleted, Kubernetes garbage-collects all owned children automatically. For Proplet and Task resources, the operator also adds a finalizer to ensure it can clean up resources before the object is removed from the API server. diff --git a/content/docs/k8s/operator.mdx b/content/docs/k8s/operator.mdx index ec8ac7d..d13c052 100644 --- a/content/docs/k8s/operator.mdx +++ b/content/docs/k8s/operator.mdx @@ -7,7 +7,6 @@ The Propeller Kubernetes Operator extends any Kubernetes cluster with five Custo The operator source code is available at [github.com/absmach/propeller-k8s-operator](https://github.com/absmach/propeller-k8s-operator). - ## Architecture ### How the Operator Fits into Propeller @@ -18,31 +17,31 @@ The operator connects to a Atom MQTT broker on startup using the manager's crede ![Propeller Kubernetes Operator Architecture Overview](../images/kubernetes-operator/architecture.svg) -*Figure: High-level architecture of the Propeller Kubernetes Operator showing the relationship between the operator, Kubernetes API, Atom MQTT broker, and proplet workers (both in-cluster k8s pods and external devices). The operator watches Custom Resources and reconciles state through MQTT communication.* +_Figure: High-level architecture of the Propeller Kubernetes Operator showing the relationship between the operator, Kubernetes API, Atom MQTT broker, and proplet workers (both in-cluster k8s pods and external devices). The operator watches Custom Resources and reconciles state through MQTT communication._ ### Controllers The operator runs five controllers concurrently. Each controller owns one Custom Resource kind and drives it through a phase-based state machine. -| Controller | Custom Resource | Responsibility | -| --------------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `PropletReconciler` | `Proplet` | Creates and reconciles Kubernetes Deployments for in-cluster workers; monitors external devices via MQTT heartbeats | -| `TaskReconciler` | `Task` | Schedules a WASM workload on the selected proplet; creates a Kubernetes Job for k8s proplets or publishes an MQTT message for external proplets | -| `PropellerJobReconciler` | `PropellerJob` | Groups a set of Tasks into a single batch; creates child Task resources and aggregates their outcomes | -| `FederatedJobReconciler` | `FederatedJob` | Orchestrates a multi-round federated learning experiment by creating and advancing `TrainingRound` resources | -| `TrainingRoundReconciler` | `TrainingRound` | Creates per-participant Task resources for one training round, collects model updates via MQTT, and aggregates them once k-of-n participants complete | +| Controller | Custom Resource | Responsibility | +| ------------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | +| `PropletReconciler` | `Proplet` | Creates and reconciles Kubernetes Deployments for in-cluster workers; monitors external devices via MQTT heartbeats | +| `TaskReconciler` | `Task` | Schedules a WASM workload on the selected proplet; creates a Kubernetes Job for k8s proplets or publishes an MQTT message for external proplets | +| `PropellerJobReconciler` | `PropellerJob` | Groups a set of Tasks into a single batch; creates child Task resources and aggregates their outcomes | +| `FederatedJobReconciler` | `FederatedJob` | Orchestrates a multi-round federated learning experiment by creating and advancing `TrainingRound` resources | +| `TrainingRoundReconciler` | `TrainingRound` | Creates per-participant Task resources for one training round, collects model updates via MQTT, and aggregates them once k-of-n participants complete | ### Custom Resource Definitions The operator installs five CRDs into the cluster: -| CRD | API Version | Description | -| ----------------- | ---------------------------------------------------- | -------------------------------------------------------------- | -| `Task` | `propeller.propeller.abstractmachines.fr/v1` | A single WASM workload to execute on a proplet | -| `Proplet` | `propeller.propeller.abstractmachines.fr/v1` | A worker node, either a Kubernetes pod or an external device | -| `PropellerJob` | `propeller.propeller.abstractmachines.fr/v1` | A named batch of Tasks executed as a group | -| `FederatedJob` | `propeller.propeller.abstractmachines.fr/v1` | A federated learning experiment spanning multiple proplets | -| `TrainingRound` | `propeller.propeller.abstractmachines.fr/v1` | One round within a FederatedJob | +| CRD | API Version | Description | +| --------------- | ----------------------------------- | ------------------------------------------------------------ | +| `Task` | `propeller.propeller.absmach.eu/v1` | A single WASM workload to execute on a proplet | +| `Proplet` | `propeller.propeller.absmach.eu/v1` | A worker node, either a Kubernetes pod or an external device | +| `PropellerJob` | `propeller.propeller.absmach.eu/v1` | A named batch of Tasks executed as a group | +| `FederatedJob` | `propeller.propeller.absmach.eu/v1` | A federated learning experiment spanning multiple proplets | +| `TrainingRound` | `propeller.propeller.absmach.eu/v1` | One round within a FederatedJob | ### MQTT Communication @@ -54,17 +53,17 @@ m/{tenant_id}/c/{channel_id}/{sub-topic} The operator subscribes to the wildcard `m/{tenant_id}/c/{channel_id}/#` and dispatches incoming messages to the appropriate handler by matching the topic suffix. -| Sub-topic | Direction | Purpose | -| -------------------------------- | -------------------- | --------------------------------------------------------------------------------------- | -| `control/proplet/alive` | proplet → operator | Heartbeat published by every proplet every 10 seconds; carries optional device metadata | -| `control/proplet/metrics` | proplet → operator | Proplet system metrics (CPU utilisation, memory usage) | -| `control/proplet/results` | proplet → operator | Task result payload after WASM execution completes | -| `control/proplet/task_metrics` | proplet → operator | Per-task CPU and memory metrics during execution | -| `control/manager/start` | operator → proplet | Task execution instruction including the full task spec | -| `control/manager/stop` | operator → proplet | Instruction to stop a running task | +| Sub-topic | Direction | Purpose | +| ------------------------------ | ------------------ | --------------------------------------------------------------------------------------- | +| `control/proplet/alive` | proplet → operator | Heartbeat published by every proplet every 10 seconds; carries optional device metadata | +| `control/proplet/metrics` | proplet → operator | Proplet system metrics (CPU utilisation, memory usage) | +| `control/proplet/results` | proplet → operator | Task result payload after WASM execution completes | +| `control/proplet/task_metrics` | proplet → operator | Per-task CPU and memory metrics during execution | +| `control/manager/start` | operator → proplet | Task execution instruction including the full task spec | +| `control/manager/stop` | operator → proplet | Instruction to stop a running task | ![MQTT Communication Flow](../images/kubernetes-operator/mqtt-communication.svg) -*Figure: MQTT message flow between the Propeller Kubernetes Operator and proplets via Atom. Shows the bidirectional communication pattern including heartbeats, task dispatch, status updates, and result collection across domain and channel scoped topics.* +_Figure: MQTT message flow between the Propeller Kubernetes Operator and proplets via Atom. Shows the bidirectional communication pattern including heartbeats, task dispatch, status updates, and result collection across domain and channel scoped topics._ -The MQTT client ID used by the operator is `propeller-controller`. Proplets identify themselves in alive messages using their Atom `entity_id`. The operator matches this value against `spec.connectionConfig.clientId` in Proplet CRs to route heartbeats and results to the correct resource. +The MQTT client ID used by the operator is `propeller-controller`. Proplets identify themselves in alive messages using their Atom `entity_id`. The operator matches this value against `spec.connectionConfig.entityId` in Proplet CRs to route heartbeats and results to the correct resource. diff --git a/content/docs/monitoring.mdx b/content/docs/monitoring.mdx index 44bf5bb..ef8987f 100644 --- a/content/docs/monitoring.mdx +++ b/content/docs/monitoring.mdx @@ -291,11 +291,11 @@ The `aggregated` field is only present when `retain_history: true` in the monito ### Subscribing to Metrics -**Subscribe to per-task metrics for a specific domain/channel:** +**Subscribe to per-task metrics for a specific tenant/channel:** ```bash mosquitto_sub -h localhost -p 1883 \ - -t "m/domain-123/c/channel-456/control/proplet/task_metrics" -v + -t "m/tenant_id-123/c/channel_id-456/control/proplet/task_metrics" -v ``` **Subscribe to proplet metrics:** diff --git a/content/docs/proxy.mdx b/content/docs/proxy.mdx index f00d41a..e398bf3 100644 --- a/content/docs/proxy.mdx +++ b/content/docs/proxy.mdx @@ -244,29 +244,18 @@ If you're curious about the internals or need to debug issues, here's how the Pr The Proxy runs two goroutines that communicate via channels: -**HTTP Stream**: Handles registry communication. When a module request arrives, it authenticates with the registry, resolves the OCI manifest, downloads the WASM layer, and chunks it. Limits concurrent downloads to 50 to prevent overwhelming the registry. +**Registry Stream**: Handles registry communication via ORAS (OCI Registry as Storage). When a module request arrives, it authenticates with the registry, resolves the OCI manifest, downloads the WASM layer, and chunks it. Limits concurrent downloads to 50 to prevent overwhelming the registry. -**MQTT Stream**: Takes chunks from the HTTP stream and publishes them to MQTT. Tracks which chunks have been sent for each module and logs completion. +**MQTT Stream**: Takes chunks from the Registry stream and publishes them to MQTT. Tracks which chunks have been sent for each module and logs completion. The diagram below shows the internal architecture of the Proxy, illustrating how these two streams interact: ![Proxy Architecture](./images/proxy/architecture.svg) -### OpenTelemetry Tracing - -The Proxy supports distributed tracing via OpenTelemetry. When configured, it creates spans for registry fetches, chunking, and MQTT delivery operations, propagating trace context across service boundaries. - -| Variable | Description | Default | -| --------------------- | ------------------------------------------------ | --------- | -| `PROXY_OTEL_ENABLED` | Enable OpenTelemetry tracing | `false` | -| `PROXY_OTEL_ENDPOINT` | OTLP HTTP endpoint for trace export | — | - -Trace context is propagated via `traceparent` and `baggage` headers in HTTP requests and through MQTT application messages. - ### MQTT Topics Used -| Topic Pattern | Direction | Purpose | -| ----------------------------------------- | --------- | -------------------------------------- | +| Topic Pattern | Direction | Purpose | +| ----------------------------------------------- | --------- | -------------------------------------- | | `m/{tenant_id}/c/{channel_id}/registry/proplet` | Subscribe | Receives module requests from Proplets | | `m/{tenant_id}/c/{channel_id}/registry/server` | Publish | Sends chunks back to Proplets | diff --git a/content/docs/reference.mdx b/content/docs/reference.mdx index 9faa18c..3fcd730 100644 --- a/content/docs/reference.mdx +++ b/content/docs/reference.mdx @@ -79,7 +79,9 @@ propeller-cli provision [--file-name ] | `job_id` | string (UUID) | ID of the job this task belongs to. | | `results` | any | Output stored after the task completes successfully. | | `error` | string | Error message if the task failed. | +| `broadcast` | boolean | If `true`, send the task to all available proplets simultaneously instead of a single selected one. | | `monitoring_profile` | object | Per-task monitoring configuration (see [Monitoring](monitoring)). | +| `metadata` | object | Free-form JSON key-value metadata attached to the task (max 1 MB serialized). | | `schedule` | string | Cron expression for recurring execution (e.g. `"0 * * * *"`). | | `timezone` | string | Timezone for cron schedule evaluation (default: `UTC`). | | `is_recurring` | boolean | `true` if a `schedule` is set. | diff --git a/content/docs/tee.mdx b/content/docs/tee.mdx index e59911f..77f49dd 100644 --- a/content/docs/tee.mdx +++ b/content/docs/tee.mdx @@ -354,8 +354,8 @@ A full example with credentials: ```bash export PROPLET_TENANT_ID=your-tenant-id -export PROPLET_ENTITY_ID=your-client-id -export PROPLET_API_KEY=your-client-key +export PROPLET_ENTITY_ID=your-entity-id +export PROPLET_API_KEY=your-api-key export PROPLET_CHANNEL_ID=your-channel-id export PROPLET_MQTT_ADDRESS="tcp://10.0.2.2:1883" export KBS_URL="http://10.0.2.2:8082" @@ -517,8 +517,8 @@ Set these environment variables before starting Proplet: ```bash export PROPLET_TENANT_ID=your-tenant-id -export PROPLET_ENTITY_ID=your-client-id -export PROPLET_API_KEY=your-client-key +export PROPLET_ENTITY_ID=your-entity-id +export PROPLET_API_KEY=your-api-key export PROPLET_CHANNEL_ID=your-channel-id export PROPLET_MQTT_ADDRESS="tcp://10.0.2.2:1883" export KBS_URL="http://10.0.2.2:8082" From 4c344b8d02d9016737c5cca336ef7832da7df2e0 Mon Sep 17 00:00:00 2001 From: Rodney Osodo Date: Tue, 21 Jul 2026 14:06:52 +0300 Subject: [PATCH 3/3] docs: add proxy provisioning steps, tweak examples and defaults Signed-off-by: Rodney Osodo --- content/docs/getting-started.mdx | 33 +++++++++++++++++++++++--------- content/docs/k8s/deployment.mdx | 5 +---- content/docs/manager.mdx | 10 +++++----- content/docs/reference.mdx | 3 +-- 4 files changed, 31 insertions(+), 20 deletions(-) diff --git a/content/docs/getting-started.mdx b/content/docs/getting-started.mdx index 08d082e..7c1c3d2 100644 --- a/content/docs/getting-started.mdx +++ b/content/docs/getting-started.mdx @@ -7,13 +7,13 @@ This guide walks you through installing Propeller and running your first WASM wo ## Prerequisites -| Tool | Version | Purpose | -| ------------------------------------------ | ------- | ---------------------------- | -| [Go](https://go.dev/doc/install) | 1.26.0+ | Building Propeller binaries | -| [Rust](https://rustup.rs/) | Latest | Building the proplet runtime | -| [Docker](https://docs.docker.com/) | 20.10+ | Running base services (Atom, FluxMQ) | -| [Make](https://www.gnu.org/software/make/) | 3.81+ | Build automation | -| [Mosquitto Tools](https://mosquitto.org/) | Latest | MQTT CLI utilities | +| Tool | Version | Purpose | +| ------------------------------------------ | ------- | ------------------------------------ | +| [Go](https://go.dev/doc/install) | 1.26.0+ | Building Propeller binaries | +| [Rust](https://rustup.rs/) | Latest | Building the proplet runtime | +| [Docker](https://docs.docker.com/) | 20.10+ | Running base services (Atom, FluxMQ) | +| [Make](https://www.gnu.org/software/make/) | 3.81+ | Build automation | +| [Mosquitto Tools](https://mosquitto.org/) | Latest | MQTT CLI utilities | These are only needed if you plan to build WASM modules locally: @@ -200,8 +200,8 @@ channel_id = "8c6e1e6c-fc89-43b4-b00b-884a690c7419" [proxy] tenant_id = "182c0907-002c-4bfd-8bf3-e4f40c58dde6" -entity_id = "fa407362-9c5f-41b8-9a09-9d0c0b039287" -api_key = "991c4d03-2f2c-4ba5-97a6-45bead85457e" +entity_id = "a3d7e901-2345-6789-abcd-ef0123456789" +api_key = "a1b2c3d4-e5f6-7890-abcd-ef1234567890" channel_id = "8c6e1e6c-fc89-43b4-b00b-884a690c7419" ``` @@ -531,6 +531,21 @@ make stop-propeller make start-propeller ``` +### Adding or Replacing a Proxy Entity + +If your `config.toml` was created before the proxy was part of Propeller, or you need to regenerate the proxy credentials, use the `add-proxy` subcommand: + +```bash +propeller-cli provision add-proxy +``` + +This command: + +1. Reads your existing `config.toml` to extract `tenant_id` and `channel_id` +2. Creates a new Atom service entity with an API key for the proxy +3. Connects the entity to the existing channel +4. Updates the `[proxy]` section in your `config.toml` + ## Postman Collection A [Postman collection](../postman_collection.json) of all API calls is available for interacting with the Propeller system. diff --git a/content/docs/k8s/deployment.mdx b/content/docs/k8s/deployment.mdx index f401298..283db1f 100644 --- a/content/docs/k8s/deployment.mdx +++ b/content/docs/k8s/deployment.mdx @@ -166,14 +166,11 @@ Sample YAML files covering all CRDs are in `config/samples/`: ### E2E Tests -The operator ships with a Ginkgo-based e2e test suite. Two modes are available: +The operator ships with a Ginkgo-based e2e test suite that uses Kind: ```bash # Kind (creates a temporary cluster automatically) make test-e2e - -# k3d (uses your existing k3d cluster) -make test-e2e-k3d ``` ### Manual Testing on k3d diff --git a/content/docs/manager.mdx b/content/docs/manager.mdx index e9c43ef..136a06f 100644 --- a/content/docs/manager.mdx +++ b/content/docs/manager.mdx @@ -126,8 +126,8 @@ channel_id = "9856d353-88d3-4eb6-82bd-117eadf69c74" [proxy] tenant_id = "73cc3c8f-2c6c-4c08-903b-f1f426d5d9bf" -entity_id = "ec182939-7940-4b25-869e-47b245ddec09" -api_key = "21076163-f7b9-439e-94b9-f17f69ce3d28" +entity_id = "f3c4d5e6-a7b8-9012-cdef-345678901234" +api_key = "87654321-4321-6543-2109-876543210987" channel_id = "9856d353-88d3-4eb6-82bd-117eadf69c74" ``` @@ -135,9 +135,9 @@ Each section contains: | Field | Purpose | | ------------ | ------------------------------------------------ | -| `tenant_id` | Atom tenant for multi-tenant isolation | -| `entity_id` | Unique identifier for MQTT entity identification | -| `api_key` | Secret API key for MQTT authentication | +| `tenant_id` | Atom tenant for multi-tenant isolation | +| `entity_id` | Unique identifier for MQTT entity identification | +| `api_key` | Secret API key for MQTT authentication | | `channel_id` | Shared channel for Manager-Proplet communication | The Manager, Proplets, and Proxy all share the same `tenant_id` and `channel_id` but have unique `entity_id`/`api_key` pairs. This allows them to communicate securely on a shared channel while maintaining distinct identities. diff --git a/content/docs/reference.mdx b/content/docs/reference.mdx index 3fcd730..a1d7c32 100644 --- a/content/docs/reference.mdx +++ b/content/docs/reference.mdx @@ -15,8 +15,7 @@ The `propeller-cli` binary is installed to `$GOBIN` by `make install`. All comma | -------------------- | ----- | ----------------------- | ----------------------------------- | | `--manager-url` | `-m` | `http://localhost:7070` | URL of the Manager HTTP API | | `--tls-verification` | `-v` | — | Enable TLS certificate verification | -| `--atom-url` | `-a` | `http://localhost:9000` | Atom base service URL | -| `--content-type` | `-t` | — | HTTP Content-Type header value | +| `--atom-url` | `-a` | `http://localhost:8080` | Atom base service URL | ### Tasks