Add together-rl skill (GRPO with sandboxed code-execution rewards)

AGENTS.md

@@ -1,6 +1,6 @@
 # AGENTS.md
 
-This repository contains 12 agent skills for the Together AI platform. Each skill is a self-contained directory following the [Agent Skills specification](https://agentskills.io/specification).
+This repository contains 13 agent skills for the Together AI platform. Each skill is a self-contained directory following the [Agent Skills specification](https://agentskills.io/specification).
 
 ## Skill registry
 
@@ -17,6 +17,7 @@ This repository contains 12 agent skills for the Together AI platform. Each skil
 - **together-dedicated-endpoints**: Single-tenant GPU endpoints on Together AI with autoscaling and no rate limits. Deploy fine-tuned or uploaded models, size hardware, and manage endpoint lifecycle. Reach for it whenever the user needs predictable always-on hosting rather than serverless inference, custom containers, or raw clusters.
 - **together-dedicated-containers**: Custom Dockerized inference workers on Together AI's managed GPU infrastructure. Build with Sprocket SDK, configure with Jig CLI, submit async queue jobs, and poll results. Reach for it whenever the user needs container-level control rather than a standard model endpoint or raw cluster.
 - **together-gpu-clusters**: On-demand and reserved GPU clusters (H100, H200, B200) on Together AI with Kubernetes or Slurm orchestration, shared storage, credential management, and cluster scaling for ML and HPC jobs. Reach for it when the user needs multi-node compute or infrastructure control rather than a managed model endpoint.
+- **together-rl**: GRPO and policy-gradient reinforcement-learning post-training on Together AI via low-level training sessions: create a session, sample token-level rollouts, score them, and apply forward_backward plus optim_step. Pairs the RL training API with the together-sandbox SDK to compute rewards by executing model-generated code in isolated sandboxes (run a test suite; pass/fail becomes the reward). Reach for it whenever the user wants to train a model with reinforcement learning, run a GRPO loop, or compute verifiable code-execution rewards rather than managed supervised fine-tuning or plain inference.
 
 </skills>
 

README.md

@@ -34,6 +34,7 @@ Each skill contains:
 | **together-dedicated-endpoints** | Single-tenant GPU endpoints on Together AI with autoscaling and no rate limits. | `deploy_finetuned.py`, `manage_endpoint.py`, `upload_custom_model.py` |
 | **together-dedicated-containers** | Custom Dockerized inference workers on Together AI's managed GPU infrastructure. | `queue_client.py`, `sprocket_hello_world.py` |
 | **together-gpu-clusters** | On-demand and reserved GPU clusters (H100, H200, B200) on Together AI with Kubernetes or Slurm orchestration, shared ... | `manage_cluster.py`, `manage_storage.py` |
+| **together-rl** | GRPO and policy-gradient reinforcement-learning post-training on Together AI via low-level training sessions: create ... | `grpo_sandbox_reward.py` |
 <!-- END_SKILLS_TABLE -->
 
 ## Installation

quality/trigger-evals/together-rl.json

@@ -0,0 +1,26 @@
+[
+  {
+    "query": "Run a GRPO reinforcement-learning loop on Together AI where the reward comes from running the model's generated code against a test suite.",
+    "should_trigger": true
+  },
+  {
+    "query": "I want to post-train a model with policy gradients using together-rl: sample rollouts, compute advantages, and call forward_backward and optim_step.",
+    "should_trigger": true
+  },
+  {
+    "query": "Train a coding model with RL where each candidate solution is executed in a sandbox and pass/fail is the reward.",
+    "should_trigger": true
+  },
+  {
+    "query": "Fine-tune Llama 3.1 8B on my instruction dataset with LoRA on Together AI.",
+    "should_trigger": false
+  },
+  {
+    "query": "Just run a chat completion with Llama 3.3 70B and stream the tokens.",
+    "should_trigger": false
+  },
+  {
+    "query": "Create an isolated sandbox from a Docker image and run some bash commands.",
+    "should_trigger": false
+  }
+]

skills/together-rl/SKILL.md

@@ -0,0 +1,94 @@
+---
+name: together-rl
+description: "GRPO and policy-gradient reinforcement-learning post-training on Together AI via low-level training sessions: create a session, sample token-level rollouts, score them, and apply forward_backward plus optim_step. Pairs the RL training API with the together-sandbox SDK to compute rewards by executing model-generated code in isolated sandboxes (run a test suite; pass/fail becomes the reward). Reach for it whenever the user wants to train a model with reinforcement learning, run a GRPO loop, or compute verifiable code-execution rewards rather than managed supervised fine-tuning or plain inference."
+---
+
+# Together RL
+
+## Overview
+
+Use Together RL when the user wants to post-train a model with reinforcement learning — driving the
+training loop themselves through low-level training sessions rather than submitting a managed job.
+
+Typical fits:
+
+- GRPO / policy-gradient post-training with a custom reward
+- Coding or agentic RL where the reward comes from **executing** the model's output (run tests, check exit code)
+- Verifiable-reward RL (math, code, tool use) with token-level sampling and advantages
+- Multi-turn rollout training where each rollout is scored out-of-band
+
+## When This Skill Wins
+
+- The user wants to run the GRPO loop directly: sample → score → `forward_backward` → `optim_step`
+- The reward requires running untrusted, model-generated code (so it must execute in a sandbox)
+- The user references GRPO, RL fine-tuning, policy gradients, advantages, reward functions, or verifiers
+- Token-level control is needed (logprobs, advantages, loss masks) rather than a managed dataset job
+
+## Hand Off To Another Skill
+
+- Use `together-sandbox` for the sandbox API itself — this skill calls it to execute reward code, but the
+  snapshot/sandbox/exec surface lives there
+- Use `together-fine-tuning` for managed supervised fine-tuning, LoRA, or DPO jobs (no hand-written loop)
+- Use `together-chat-completions` if the user only needs inference, not training
+- Use `together-evaluations` to score a model with an LLM judge rather than a programmatic reward
+
+## Quick Routing
+
+- **Run a GRPO loop with code-execution rewards**
+  - Start with [scripts/grpo_sandbox_reward.py](scripts/grpo_sandbox_reward.py)
+- **Training-session lifecycle, the three operations, and the GRPO sample schema**
+  - Read [references/grpo-loop.md](references/grpo-loop.md)
+- **Compute rewards by running code in a sandbox (the integration)**
+  - Read [references/sandbox-rewards.md](references/sandbox-rewards.md)
+
+## Workflow
+
+1. Create a training session with `client.beta.rl.sessions.create(base_model=...)` and wait for status
+   `TRAINING_SESSION_STATUS_RUNNING`.
+2. For each step, sample a group of rollouts per prompt with `client.beta.rl.training.sample(...)`
+   (`num_samples` = group size); each call returns an async **operation** you poll to completion.
+3. Decode the sampled token sequences and **compute rewards out-of-band** — for code tasks, hand off to
+   `together-sandbox`: run each candidate's test suite in an isolated sandbox, `exit 0` → reward `1.0`.
+4. Compute GRPO advantages: `advantage = reward − mean(group_rewards)` (center within each prompt's group).
+5. Assemble GRPO samples and call `client.beta.rl.training.forward_backward(...)` with `LOSS_TYPE_GRPO`.
+6. Apply `client.beta.rl.training.optim_step(session_id, learning_rate=...)`.
+7. Repeat; always `client.beta.rl.sessions.stop(session_id)` in a `finally`.
+
+## High-Signal Rules
+
+- **RL calls return operations, not results.** `sample` / `forward_backward` / `optim_step` each return an
+  operation handle — poll it (status `TRAINING_OPERATION_STATUS_*`) until complete before using the output.
+- **Rewards are computed out-of-band, never inside the RL call.** Sampling returns tokens + logprobs; you
+  score them yourself and feed advantages back in `forward_backward`.
+- **The RL SDK is synchronous; the sandbox SDK is async.** Bridge them with a single `asyncio.run(...)` per
+  batch that fans out sandboxes with `asyncio.gather` (one sandbox per candidate), then return rewards into
+  the sync loop. Do not create a sandbox per token or per RL call.
+- **Center advantages within the group.** GRPO subtracts the per-prompt group-mean reward as the baseline;
+  a group where every sample gets the same reward contributes zero gradient.
+- **GRPO sample schema** (see [references/grpo-loop.md](references/grpo-loop.md)): `model_input.chunks[].
+  encoded_text.tokens`, and `loss_inputs.{loss_mask, target_tokens, grpo_inputs.{advantages,
+  generator_logprobs}}`. Mask the prompt tokens out of the loss; only response tokens carry advantage.
+- **Pre-build the reward environment as a sandbox snapshot.** Bake the task harness, dependencies, and
+  tests into a snapshot once, then create ephemeral sandboxes from it per rollout — don't `pip install`
+  inside every rollout.
+
+## Status
+
+The RL training API (`client.beta.rl`) is in **beta** and is **not yet in the public `together` package**
+(public `together` exposes `beta.clusters` / `beta.jig` only). It also requires a service-specific
+`base_url`. This skill targets the public SDK surface; the reward half (`together-sandbox`) is usable today.
+Until the public RL release ships, install the preview SDK per the RL API docs and pass the RL `base_url`
+explicitly.
+
+## Resource Map
+
+- **GRPO loop mechanics**: [references/grpo-loop.md](references/grpo-loop.md)
+- **Sandbox reward integration**: [references/sandbox-rewards.md](references/sandbox-rewards.md)
+- **End-to-end script**: [scripts/grpo_sandbox_reward.py](scripts/grpo_sandbox_reward.py)
+- **Sandbox API** (hand-off): the `together-sandbox` skill
+
+## Official Docs
+
+- Together RL training API — `/rl/training-sessions` (create, `:forward-backward`, `:optim-step`, `:stop`)
+- [Together Sandbox SDK](https://github.com/togethercomputer/together-sandbox/blob/main/docs/python-sdk.md)
+- Reference loop: [rl-cookbook/grpo_train.py](https://github.com/togethercomputer/rl-cookbook/blob/main/grpo_train.py)

skills/together-rl/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Together RL"
+  short_description: "GRPO reinforcement-learning post-training with sandboxed rewards"
+  default_prompt: "Use $together-rl to run a GRPO training loop on Together AI with rewards computed by executing code in a sandbox."

skills/together-rl/references/grpo-loop.md

@@ -0,0 +1,135 @@
+# GRPO Training Loop Reference
+
+## Contents
+
+- [Overview](#overview)
+- [Session Lifecycle](#session-lifecycle)
+- [Operations and Polling](#operations-and-polling)
+- [The Three Training Operations](#the-three-training-operations)
+- [GRPO Sample Schema](#grpo-sample-schema)
+- [Advantages](#advantages)
+- [Batch Padding](#batch-padding)
+
+## Overview
+
+GRPO (Group Relative Policy Optimization) post-training on Together runs as a loop over the low-level RL
+training API. Each step: sample a group of completions per prompt, score them into rewards, center the
+rewards within each group to get advantages, then push one `forward_backward` + `optim_step`.
+
+This doc covers the **training mechanics** (the request/response shapes you assemble). It does not restate
+the full API — for authoritative request/response definitions see the RL training API docs and the
+`/rl/training-sessions` family in the OpenAPI spec. The canonical reference implementation is
+[`rl-cookbook/grpo_train.py`](https://github.com/togethercomputer/rl-cookbook/blob/main/grpo_train.py);
+the only thing this skill changes is **where the reward comes from** (a sandbox — see
+[sandbox-rewards.md](sandbox-rewards.md)).
+
+## Session Lifecycle
+
+```python
+session = client.beta.rl.sessions.create(base_model="meta-llama/Llama-3.2-1B-Instruct")
+session_id = session.session_id or getattr(session, "id", None)
+# Poll client.beta.rl.sessions.retrieve(session_id).status until:
+#   TRAINING_SESSION_STATUS_RUNNING
+# ... run training steps ...
+client.beta.rl.sessions.stop(session_id)   # always, in a finally block
+```
+
+A session holds the live model weights on the training service. Create once, run many steps, stop when
+done (stopping releases the GPU — orphaned sessions keep billing).
+
+## Operations and Polling
+
+`sample`, `forward_backward`, and `optim_step` are **asynchronous operations**: each returns an operation
+handle, not a finished result. Poll until complete before reading the output. The cookbook's helpers in
+`together.lib.beta.rl.operations` (`retrieve_operation`, `get_operation_status`, `is_operation_complete`,
+`get_operation_error`) wrap this:
+
+```python
+def wait_for_operation(client, *, session_id, operation, kind, timeout, interval):
+    deadline = time.monotonic() + timeout
+    while True:
+        operation = rl_ops.retrieve_operation(client, session_id=session_id, operation=operation, kind=kind)
+        if rl_ops.is_operation_complete(operation):
+            if rl_ops.get_operation_status(operation) == "TRAINING_OPERATION_STATUS_FAILED":
+                raise RuntimeError(rl_ops.get_operation_error(operation))
+            return operation
+        if time.monotonic() >= deadline:
+            raise TimeoutError(f"operation {kind} timed out")
+        time.sleep(interval)
+```
+
+## The Three Training Operations
+
+```python
+# 1. Sample a group of completions for one prompt (num_samples = group size)
+sample_op = client.beta.rl.training.sample(
+    session_id=session_id,
+    prompt={"chunks": [{"encoded_text": {"tokens": prompt_tokens}}]},
+    num_samples=group_size,
+    sampling_params={"max_tokens": max_sample_tokens},
+)
+result = wait_for_operation(...)            # result.output.sequences[i].tokens / .logprobs
+
+# 2. One gradient pass over the assembled, advantage-weighted batch
+fb_op = client.beta.rl.training.forward_backward(
+    session_id=session_id,
+    loss={"type": "LOSS_TYPE_GRPO",
+          "grpo_params": {"beta": 0.0, "agg_type": "GRPO_LOSS_AGGREGATION_TYPE_TOKEN_MEAN"}},
+    samples=samples,
+)
+loss = wait_for_operation(...).output.loss
+
+# 3. Optimizer step
+opt_op = client.beta.rl.training.optim_step(session_id=session_id, learning_rate=8e-5)
+wait_for_operation(...)
+```
+
+## GRPO Sample Schema
+
+Each training sample concatenates prompt + response tokens, masks the prompt out of the loss, and attaches
+the per-token advantage and the generator's logprobs:
+
+```python
+def build_grpo_sample(prompt_tokens, response_tokens, response_logprobs, advantage, max_model_length):
+    available = max(max_model_length - len(prompt_tokens), 0)
+    response_tokens = response_tokens[:available]
+    response_logprobs = response_logprobs[:available]
+    if not response_tokens:
+        return None
+    model_tokens = prompt_tokens + response_tokens
+    loss_mask  = [0] * len(prompt_tokens) + [1] * len(response_tokens)   # train on response only
+    targets    = model_tokens[1:] + [0]                                  # next-token targets
+    advantages = [0.0] * len(prompt_tokens) + [advantage] * len(response_tokens)
+    logprobs   = [0.0] * len(prompt_tokens) + [float(x) for x in response_logprobs]
+    return {
+        "model_input": {"chunks": [{"encoded_text": {"tokens": model_tokens}}]},
+        "loss_inputs": {
+            "loss_mask":     {"data": loss_mask,  "dtype": "D_TYPE_INT64"},
+            "target_tokens": {"data": targets,    "dtype": "D_TYPE_INT64"},
+            "grpo_inputs": {
+                "advantages":         {"data": advantages, "dtype": "D_TYPE_FLOAT32"},
+                "generator_logprobs": {"data": logprobs,    "dtype": "D_TYPE_FLOAT32"},
+            },
+        },
+    }
+```
+
+## Advantages
+
+GRPO centers rewards **within each prompt's group**:
+
+```python
+baseline = sum(group_rewards) / len(group_rewards)
+advantage = reward - baseline
+```
+
+A group where every sample earns the same reward has zero advantage everywhere and contributes no
+gradient — reward diversity within a group is what drives learning. `forward_backward` needs a minimum
+batch (the cookbook requires ≥ 8 assembled samples).
+
+## Batch Padding
+
+All samples in a `forward_backward` batch must be padded to the longest sequence. Pad `tokens`,
+`loss_mask`, and `target_tokens` with `0`, and `advantages` / `generator_logprobs` with `0.0`. See
+`_pad_samples` in [rl-cookbook/grpo_train.py](https://github.com/togethercomputer/rl-cookbook/blob/main/grpo_train.py)
+for the exact padding (note: the integer fields are serialized as strings in the padded request).