Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions skills/together-dedicated-endpoints/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,8 @@ Typical fits:
- Read [references/dedicated-models.md](references/dedicated-models.md)
- **Hardware and sizing choices**
- Read [references/hardware-options.md](references/hardware-options.md)
- **Serve multiple LoRA adapters on one endpoint**
- Read the Multi-LoRA Adapters section in [references/api-reference.md](references/api-reference.md)

## Workflow

Expand All @@ -62,6 +64,7 @@ Typical fits:
- Endpoint management uses endpoint IDs, while inference usually uses the endpoint name as `model`.
- Autoscaling, auto-shutdown, prompt caching, and speculative decoding materially affect operations and cost.
- For custom or fine-tuned models, do not skip the intermediate verification steps before deployment.
- One LoRA-enabled endpoint can serve multiple adapters that share its base model (beta): attach/list/remove them on the running endpoint via `client.endpoints.adapters` and select by model name, instead of deploying separate hardware per adapter.

## Resource Map

Expand Down
54 changes: 54 additions & 0 deletions skills/together-dedicated-endpoints/references/api-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
- [Upload Model](#upload-model)
- [List Models](#list-models)
- [Using the Endpoint](#using-the-endpoint)
- [Multi-LoRA Adapters](#multi-lora-adapters)
- [Auto-Shutdown](#auto-shutdown)
- [Speculative Decoding](#speculative-decoding)
- [Prompt Caching](#prompt-caching)
Expand Down Expand Up @@ -460,6 +461,59 @@ curl -X POST "https://api.together.xyz/v1/chat/completions" \
}'
```

## Multi-LoRA Adapters

A single LoRA-enabled dedicated endpoint can serve multiple LoRA adapters that share the same base
model. Attach, list, and remove adapters on a running endpoint (beta) without redeploying, then
select among them by model name at inference time.

Requirements: the endpoint must be a private dedicated endpoint with LoRA enabled, running a base
model compatible with the adapter, and the adapter and endpoint must be owned by the same account.

| Method | Path | Description |
|--------|------|-------------|
| `POST /endpoints/{id}/adapters` | Add adapter | Bind a LoRA adapter to the endpoint |
| `GET /endpoints/{id}/adapters` | List adapters | List adapters bound to the endpoint |
| `DELETE /endpoints/{id}/adapters` | Remove adapter | Unbind an adapter from the endpoint |

Each adapter is identified by a combined `model_id` of the form `endpoint_name:adapter_model_name`,
where `endpoint_name` must match the endpoint resolved from the endpoint ID and `adapter_model_name`
is the adapter's uploaded `model_name`.

```python
from together import Together

client = Together()

# Attach
result = client.endpoints.adapters.add(
"endpoint-abc123",
model_id="my-endpoint-name:my-adapter-model",
)

# List
adapters = client.endpoints.adapters.list("endpoint-abc123")
for adapter in adapters.data or []:
print(adapter.api_model_id, adapter.adapter_name, adapter.endpoint_name)

# Remove
client.endpoints.adapters.remove(
"endpoint-abc123",
model_id="my-endpoint-name:my-adapter-model",
)
```

```shell
together endpoints adapters add <ENDPOINT_ID> <ENDPOINT_NAME>:<ADAPTER_MODEL_NAME>
together endpoints adapters list <ENDPOINT_ID>
together endpoints adapters remove <ENDPOINT_ID> <ENDPOINT_NAME>:<ADAPTER_MODEL_NAME>
```

Once attached, send inference requests using the adapter model name (or the full
`endpoint_name:adapter_model_name`) as the `model` parameter; requests route to the endpoint
automatically. Note: the Python SDK exposes the combined identifier as `api_model_id` on the
response object, while the raw API and TypeScript SDK use `model_id`.

## Auto-Shutdown

Endpoints auto-stop after 1 hour of inactivity by default. Set `inactive_timeout` in minutes to
Expand Down
Loading