Add together-volcano and together-kueue skills

AGENTS.md

@@ -1,6 +1,6 @@
 # AGENTS.md
 
-This repository contains 12 agent skills for the Together AI platform. Each skill is a self-contained directory following the [Agent Skills specification](https://agentskills.io/specification).
+This repository contains 14 agent skills for the Together AI platform. Each skill is a self-contained directory following the [Agent Skills specification](https://agentskills.io/specification).
 
 ## Skill registry
 
@@ -17,6 +17,8 @@ This repository contains 12 agent skills for the Together AI platform. Each skil
 - **together-dedicated-endpoints**: Single-tenant GPU endpoints on Together AI with autoscaling and no rate limits. Deploy fine-tuned or uploaded models, size hardware, and manage endpoint lifecycle. Reach for it whenever the user needs predictable always-on hosting rather than serverless inference, custom containers, or raw clusters.
 - **together-dedicated-containers**: Custom Dockerized inference workers on Together AI's managed GPU infrastructure. Build with Sprocket SDK, configure with Jig CLI, submit async queue jobs, and poll results. Reach for it whenever the user needs container-level control rather than a standard model endpoint or raw cluster.
 - **together-gpu-clusters**: On-demand and reserved GPU clusters (H100, H200, B200) on Together AI with Kubernetes or Slurm orchestration, shared storage, credential management, and cluster scaling for ML and HPC jobs. Reach for it when the user needs multi-node compute or infrastructure control rather than a managed model endpoint.
+- **together-volcano**: Install and use the Volcano batch scheduler on a Together AI Kubernetes GPU cluster for gang scheduling. Covers installing Volcano, creating queues, submitting all-or-nothing gang-scheduled jobs (vcjobs), and verifying placement. Reach for it when a job on a Together cluster needs its pods scheduled all-at-once or not at all, such as distributed multi-node training, rather than per-pod best-effort scheduling. Pins Volcano v1.15.0.
+- **together-kueue**: Install and use the Kueue job-queueing controller on a Together AI Kubernetes GPU cluster to gate jobs on quota. Covers installing Kueue, defining ResourceFlavor, ClusterQueue, and LocalQueue quota, submitting jobs to a queue, and watching quota admit or suspend them. Reach for it when a Together cluster's GPU pool must be shared across teams or workloads by quota, admitting jobs only when capacity is free, rather than letting every job start immediately. Pins Kueue v0.18.3 (API kueue.x-k8s.io/v1beta2).
 
 </skills>
 

README.md

@@ -34,6 +34,8 @@ Each skill contains:
 | **together-dedicated-endpoints** | Single-tenant GPU endpoints on Together AI with autoscaling and no rate limits. | `deploy_finetuned.py`, `manage_endpoint.py`, `upload_custom_model.py` |
 | **together-dedicated-containers** | Custom Dockerized inference workers on Together AI's managed GPU infrastructure. | `queue_client.py`, `sprocket_hello_world.py` |
 | **together-gpu-clusters** | On-demand and reserved GPU clusters (H100, H200, B200) on Together AI with Kubernetes or Slurm orchestration, shared ... | `manage_cluster.py`, `manage_storage.py` |
+| **together-volcano** | Install and use the Volcano batch scheduler on a Together AI Kubernetes GPU cluster for gang scheduling. | — |
+| **together-kueue** | Install and use the Kueue job-queueing controller on a Together AI Kubernetes GPU cluster to gate jobs on quota. | — |
 <!-- END_SKILLS_TABLE -->
 
 ## Installation

quality/trigger-evals/together-kueue.json

@@ -0,0 +1,26 @@
+[
+  {
+    "query": "Install Kueue on my Together AI cluster and gate GPU jobs on quota so they queue until capacity frees up.",
+    "should_trigger": true
+  },
+  {
+    "query": "Share a fixed GPU pool across teams on a Together cluster; admit jobs only when quota is available and suspend the rest.",
+    "should_trigger": true
+  },
+  {
+    "query": "Set up a Kueue ClusterQueue, ResourceFlavor, and LocalQueue for GPU quota on Together AI and submit a queued job.",
+    "should_trigger": true
+  },
+  {
+    "query": "I need all pods of my Together cluster training job scheduled all-at-once or not at all with gang scheduling.",
+    "should_trigger": false
+  },
+  {
+    "query": "Run high-volume offline inference at lower cost with the Together AI Batch API.",
+    "should_trigger": false
+  },
+  {
+    "query": "Fine-tune a model on Together AI with LoRA.",
+    "should_trigger": false
+  }
+]

quality/trigger-evals/together-volcano.json

@@ -0,0 +1,26 @@
+[
+  {
+    "query": "Install Volcano on my Together AI Kubernetes cluster and run a gang-scheduled multi-node training job.",
+    "should_trigger": true
+  },
+  {
+    "query": "My distributed training on a Together cluster keeps starting some workers while others stay pending. I need all pods scheduled together or none.",
+    "should_trigger": true
+  },
+  {
+    "query": "Create a Volcano queue and a vcjob that requests 8 GPUs with minAvailable on Together AI.",
+    "should_trigger": true
+  },
+  {
+    "query": "I want to share my Together cluster's GPUs across teams by quota and have jobs wait in a queue until capacity is free.",
+    "should_trigger": false
+  },
+  {
+    "query": "Deploy a supported chat model to a managed dedicated endpoint on Together AI.",
+    "should_trigger": false
+  },
+  {
+    "query": "Generate an image with FLUX on Together AI.",
+    "should_trigger": false
+  }
+]

scripts/generate_agents.py

@@ -36,6 +36,8 @@
     "together-dedicated-endpoints",
     "together-dedicated-containers",
     "together-gpu-clusters",
+    "together-volcano",
+    "together-kueue",
 ]
 
 README_TABLE_BEGIN = "<!-- BEGIN_SKILLS_TABLE -->"

skills/together-kueue/SKILL.md

@@ -0,0 +1,146 @@
+---
+name: together-kueue
+description: Install and use the Kueue job-queueing controller on a Together AI Kubernetes GPU cluster to gate jobs on quota. Covers installing Kueue, defining ResourceFlavor, ClusterQueue, and LocalQueue quota, submitting jobs to a queue, and watching quota admit or suspend them. Reach for it when a Together cluster's GPU pool must be shared across teams or workloads by quota, admitting jobs only when capacity is free, rather than letting every job start immediately. Pins Kueue v0.18.3 (API kueue.x-k8s.io/v1beta2).
+---
+
+# Kueue on Together GPU clusters
+
+Kueue is a Kubernetes-native job queueing controller. It holds jobs in a queue and admits them only when their quota is free, suspending the rest. Use it to share a fixed GPU pool across teams or workloads without overcommitting it. Unlike Volcano, Kueue does not replace the scheduler; it gates when jobs start by toggling their `suspend` flag.
+
+Public cookbook: https://docs.together.ai/docs/kueue-on-gpu-clusters. Pair this skill with the `together-gpu-clusters` skill, which covers creating the cluster and configuring `kubectl`.
+
+## Preconditions
+
+- A Together Kubernetes GPU cluster in the `Ready` state, with `kubectl` pointed at it (`tg beta clusters get-credentials <cluster_id> --set-default-context`).
+- GPU nodes expose `nvidia.com/gpu` (NVIDIA device plugin preinstalled on Together clusters).
+
+## Install
+
+Pin the version. The API group is `kueue.x-k8s.io/v1beta2` in v0.18.x; older examples using `v1beta1` will not apply.
+
+```bash
+kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.18.3/manifests.yaml
+kubectl -n kueue-system wait --for=condition=Available deployment/kueue-controller-manager --timeout=240s
+```
+
+`--server-side` is required; the bundled CRDs exceed the client-side apply annotation limit. Helm is an alternative: `helm install kueue oci://registry.k8s.io/kueue/charts/kueue --version 0.18.3 --namespace kueue-system --create-namespace`.
+
+## Define quotas
+
+Three objects: `ResourceFlavor` (a node class), `ClusterQueue` (the quota pool), and `LocalQueue` (the namespaced entry point users submit to).
+
+```yaml
+apiVersion: kueue.x-k8s.io/v1beta2
+kind: ResourceFlavor
+metadata:
+  name: gpu-flavor
+---
+apiVersion: kueue.x-k8s.io/v1beta2
+kind: ClusterQueue
+metadata:
+  name: gpu-cluster-queue
+spec:
+  namespaceSelector: {}            # accept jobs from every namespace
+  resourceGroups:
+    - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]  # MUST list every resource jobs request
+      flavors:
+        - name: gpu-flavor         # must match the ResourceFlavor
+          resources:
+            - name: "cpu"
+              nominalQuota: 64
+            - name: "memory"
+              nominalQuota: 512Gi
+            - name: "nvidia.com/gpu"
+              nominalQuota: 8       # admit at most 8 GPUs worth of jobs at once
+---
+apiVersion: kueue.x-k8s.io/v1beta2
+kind: LocalQueue
+metadata:
+  namespace: default
+  name: gpu-queue
+spec:
+  clusterQueue: gpu-cluster-queue
+```
+
+## Attach shared storage (optional)
+
+Together provisions a static PersistentVolume named after the cluster's shared volume. Bind a `ReadWriteMany` PVC to it so queued jobs share datasets and checkpoints, then mount it in the job (below). Storage is not a quota resource, so it does not affect admission.
+
+```yaml
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: shared-pvc
+spec:
+  accessModes: ["ReadWriteMany"]
+  storageClassName: shared-wekafs   # Together default shared storage class
+  volumeName: <shared-volume-name>  # the static PV named after your shared volume
+  resources:
+    requests:
+      storage: 100Gi
+```
+
+## Submit a job to a queue
+
+Add the queue-name label and start the job suspended. Kueue flips `suspend` to `false` on admission. The `volumes`/`volumeMounts` blocks are optional; drop them if the job needs no shared storage.
+
+```yaml
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: gpu-job
+  namespace: default
+  labels:
+    kueue.x-k8s.io/queue-name: gpu-queue   # route to the LocalQueue
+spec:
+  parallelism: 1
+  completions: 1
+  suspend: true                            # Kueue unsuspends on admission
+  template:
+    spec:
+      restartPolicy: Never
+      containers:
+        - name: worker
+          image: nvidia/cuda:12.4.0-base-ubuntu22.04
+          command: ["bash", "-c", "nvidia-smi -L; sleep 180"]
+          resources:
+            requests:
+              cpu: "2"
+              memory: "8Gi"
+            limits:
+              nvidia.com/gpu: 6
+          volumeMounts:
+            - name: shared
+              mountPath: /mnt/shared
+      volumes:
+        - name: shared
+          persistentVolumeClaim:
+            claimName: shared-pvc
+```
+
+## Verify
+
+- `kubectl get workloads` shows `ADMITTED True` once the job fits.
+- An over-quota job stays `suspend: true` with no pod. This is correct queueing, not a failure. Read the reason:
+
+```bash
+WORKLOAD=$(kubectl get workloads -o name | grep <job-name>)
+kubectl get "$WORKLOAD" -o jsonpath='{.status.conditions[?(@.type=="QuotaReserved")].message}'
+```
+
+- Freeing quota (a running job finishes or is deleted) auto-admits the next queued job. Do not resubmit.
+
+## Rules and gotchas
+
+- Pin the version, and use API `kueue.x-k8s.io/v1beta2` for v0.18.x.
+- The ClusterQueue's `coveredResources` MUST include every resource a job requests. A GPU job that also requests CPU and memory is never admitted if the ClusterQueue only covers `nvidia.com/gpu`. This is the most common failure.
+- `flavors[].name` must exactly match a `ResourceFlavor` name, or the ClusterQueue admits nothing.
+- The `kueue.x-k8s.io/queue-name` label must name a `LocalQueue` in the job's own namespace. A missing or wrong label means the job runs immediately, bypassing quota.
+- Volcano and Kueue can coexist: Kueue gates jobs on the default scheduler; Volcano owns pods with `schedulerName: volcano`. Do not point one job at both.
+
+## Reference
+
+- Kueue docs: https://kueue.sigs.k8s.io/docs/
+- Installation: https://kueue.sigs.k8s.io/docs/installation/
+- Concepts (ResourceFlavor, ClusterQueue, LocalQueue, Workload): https://kueue.sigs.k8s.io/docs/concepts/
+- Running jobs (Job, JobSet, RayJob, MPIJob): https://kueue.sigs.k8s.io/docs/tasks/run/jobs/

skills/together-kueue/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Together Kueue Queueing"
+  short_description: "Gate GPU jobs on quota with Kueue"
+  default_prompt: "Use $together-kueue to install Kueue and queue GPU jobs by quota on a Together AI Kubernetes cluster."

skills/together-volcano/SKILL.md

@@ -0,0 +1,124 @@
+---
+name: together-volcano
+description: Install and use the Volcano batch scheduler on a Together AI Kubernetes GPU cluster for gang scheduling. Covers installing Volcano, creating queues, submitting all-or-nothing gang-scheduled jobs (vcjobs), and verifying placement. Reach for it when a job on a Together cluster needs its pods scheduled all-at-once or not at all, such as distributed multi-node training, rather than per-pod best-effort scheduling. Pins Volcano v1.15.0.
+---
+
+# Volcano on Together GPU clusters
+
+Volcano is a Kubernetes-native batch scheduler. Its key property is gang scheduling: a group of pods is placed all-at-once or not at all, so distributed training never starts with only some workers running. Use it when a job needs N pods to run together or not at all.
+
+Public cookbook: https://docs.together.ai/docs/volcano-on-gpu-clusters. Pair this skill with the `together-gpu-clusters` skill, which covers creating the cluster and configuring `kubectl`.
+
+## Preconditions
+
+- A Together Kubernetes GPU cluster in the `Ready` state, with `kubectl` pointed at it (`tg beta clusters get-credentials <cluster_id> --set-default-context`).
+- GPU nodes expose `nvidia.com/gpu`; the NVIDIA device plugin is preinstalled on Together clusters. Confirm with:
+
+```bash
+kubectl get nodes -o custom-columns='NODE:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'
+```
+
+## Install
+
+Pin the version. Do not use `master`.
+
+```bash
+kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.15.0/installer/volcano-development.yaml
+kubectl -n volcano-system wait --for=condition=Available deployment --all --timeout=180s
+```
+
+A `default` queue is created automatically (`kubectl get queue`). Helm is an alternative: `helm repo add volcano-sh https://volcano-sh.github.io/helm-charts && helm install volcano volcano-sh/volcano -n volcano-system --create-namespace`.
+
+## Create a queue
+
+A queue caps and weights the resources a set of jobs can use.
+
+```yaml
+apiVersion: scheduling.volcano.sh/v1beta1
+kind: Queue
+metadata:
+  name: research
+spec:
+  reclaimable: true          # give idle capacity back to other queues
+  weight: 1                  # relative share under contention
+  capability:                # hard ceiling for this queue
+    nvidia.com/gpu: 8
+```
+
+## Attach shared storage (optional)
+
+Together provisions a static PersistentVolume named after the cluster's shared volume. Bind a `ReadWriteMany` PVC to it so every worker shares datasets and checkpoints, then mount it in the job (below).
+
+```yaml
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: shared-pvc
+spec:
+  accessModes: ["ReadWriteMany"]
+  storageClassName: shared-wekafs   # Together default shared storage class
+  volumeName: <shared-volume-name>  # the static PV named after your shared volume
+  resources:
+    requests:
+      storage: 100Gi
+```
+
+## Run a gang-scheduled job
+
+The gang guarantee comes from `minAvailable` on a Volcano `Job` (`vcjob`). Set `schedulerName: volcano` and a `queue`. The `volumes`/`volumeMounts` blocks are optional; drop them if the job needs no shared storage.
+
+```yaml
+apiVersion: batch.volcano.sh/v1alpha1
+kind: Job
+metadata:
+  name: gpu-gang
+spec:
+  minAvailable: 4            # schedule all 4 pods together or none
+  schedulerName: volcano
+  queue: research
+  policies:
+    - event: PodEvicted
+      action: RestartJob     # restart the whole gang if a pod is evicted
+  tasks:
+    - replicas: 4
+      name: worker
+      template:
+        spec:
+          restartPolicy: OnFailure
+          containers:
+            - name: worker
+              image: nvidia/cuda:12.4.0-base-ubuntu22.04
+              command: ["bash", "-c", "nvidia-smi -L; sleep 300"]
+              resources:
+                limits:
+                  nvidia.com/gpu: 2   # 4 x 2 = 8 GPUs
+              volumeMounts:
+                - name: shared
+                  mountPath: /mnt/shared
+          volumes:
+            - name: shared
+              persistentVolumeClaim:
+                claimName: shared-pvc
+```
+
+## Verify
+
+- `kubectl get vcjob <name>` should show `STATUS Running` and `RUNNINGS` equal to `minAvailable`.
+- `kubectl get podgroup` shows the gang; a `Running` phase means the whole group is placed.
+- A gang that cannot fit stays `Inqueue` with **zero** pods running (all-or-nothing). This is the signal Volcano is working, not a failure. Never "fix" it by lowering `minAvailable` below the job's real parallelism.
+
+## Rules and gotchas
+
+- Always pin the Volcano version in the install URL.
+- `schedulerName: volcano` must be on the pod template (on a `vcjob` it goes under `spec`). Without it, the default scheduler grabs the pods and there is no gang guarantee.
+- A job whose total request exceeds the queue's `capability` is rejected. Raise the cap or split the job.
+- To gang-schedule non-vcjob workloads (for example MPIJobs from the preinstalled MPI Operator), set `schedulerName: volcano` and attach a `PodGroup`. See https://volcano.sh/en/docs/podgroup/.
+- Volcano and Kueue can coexist on one cluster: Volcano owns pods with `schedulerName: volcano`; Kueue admits jobs on the default scheduler. Pick one per workload; do not point a single job at both.
+
+## Reference
+
+- Volcano docs: https://volcano.sh/en/docs/
+- Installation: https://volcano.sh/en/docs/installation/
+- Gang scheduling: https://volcano.sh/en/docs/gang_scheduling/
+- Queue: https://volcano.sh/en/docs/queue/
+- vcjob: https://volcano.sh/en/docs/vcjob/

skills/together-volcano/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Together Volcano Scheduler"
+  short_description: "Gang-schedule GPU jobs on Together clusters"
+  default_prompt: "Use $together-volcano to install Volcano and run gang-scheduled GPU jobs on a Together AI Kubernetes cluster."