Skip to content

DX-596-together-dedicated-endpoints: sync with mintlify-docs#1215 #51

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
zainhas wants to merge 1 commit into
base: main
Choose a base branch
from
Open

DX-596-together-dedicated-endpoints: sync with mintlify-docs#1215 #51

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions skills/together-dedicated-endpoints/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,8 @@ Typical fits:
- Read [references/dedicated-models.md](references/dedicated-models.md)
- **Hardware and sizing choices**
- Read [references/hardware-options.md](references/hardware-options.md)
- **LoRA-enabled base model support (target modules, max loaded adapters)**
- Read the LoRA-Enabled Base Models section in [references/api-reference.md](references/api-reference.md)

## Workflow

Expand Down
32 changes: 32 additions & 0 deletions skills/together-dedicated-endpoints/references/api-reference.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
- [Upload Model](#upload-model)
- [List Models](#list-models)
- [Using the Endpoint](#using-the-endpoint)
- [LoRA-Enabled Base Models](#lora-enabled-base-models)
- [Auto-Shutdown](#auto-shutdown)
- [Speculative Decoding](#speculative-decoding)
- [Prompt Caching](#prompt-caching)
Expand Down Expand Up @@ -460,6 +461,37 @@ curl -X POST "https://api.together.xyz/v1/chat/completions" \
}'
```

## LoRA-Enabled Base Models

LoRA-enabled dedicated endpoints are available for the base models below. Each adapter uploaded or
attached to the endpoint must target the same base model and train only from that model's supported
target modules. `Max loaded adapters` is the maximum number of adapters that can stay attached to a
single endpoint for that base model.

| Model | Supported target modules | Max loaded adapters |
|-------|--------------------------|---------------------|
| `google/gemma-3-270m-it-lora` | `k_proj`, `up_proj`, `o_proj`, `q_proj`, `down_proj`, `v_proj`, `gate_proj` | 16 |
| `google/gemma-3-27b-it-lora` | `k_proj`, `up_proj`, `o_proj`, `q_proj`, `down_proj`, `v_proj`, `gate_proj` | 16 |
| `google/gemma-4-31B-it-lora` | `k_proj`, `up_proj`, `o_proj`, `q_proj`, `down_proj`, `v_proj`, `gate_proj` | 16 |
| `mistralai/Mixtral-8x7B-Instruct-v0.1-FP8-Lora` | `k_proj`, `o_proj`, `q_proj`, `v_proj` | 16 |
| `meta-llama/Llama-3.3-70B-Instruct-FP8-Lora` | `k_proj`, `up_proj`, `o_proj`, `q_proj`, `down_proj`, `v_proj`, `gate_proj` | 16 |
| `meta-llama/Llama-4-Scout-17B-16E-Instruct-FP8-Lora` | `k_proj`, `o_proj`, `q_proj`, `v_proj`, `shared_expert.gate_proj`, `shared_expert.up_proj`, `shared_expert.down_proj`, `feed_forward.gate_proj`, `feed_forward.up_proj`, `feed_forward.down_proj` | 16 |
| `Qwen/Qwen3.5-0.8B-Lora` | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` † | 5 |
| `Qwen/Qwen3.5-2B-Lora` | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` † | 5 |
| `Qwen/Qwen3.5-4B-Lora` | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` † | 5 |
| `Qwen/Qwen3.5-9B-Lora` | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` † | 5 |
| `Qwen/Qwen3.5-27B-Lora` | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` † | 5 |
| `Qwen/Qwen3.5-35B-A3B-Base-Lora` | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` † | 5 |
| `Qwen/Qwen3.5-35B-A3B-Lora` | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` † | 5 |
| `Qwen/Qwen3.5-122B-A10B-Lora` | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` † | 5 |
| `Qwen/Qwen3.5-397B-A17B-Lora` | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` † | 5 |
| `Qwen/Qwen3.6-27B-Lora` | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` † | 5 |
| `Qwen/Qwen3.6-35B-A3B-Lora` | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` † | 5 |

† For the Qwen3.5 and Qwen3.6 models, adapters target the standard attention (`q_proj`, `k_proj`,
`v_proj`, `o_proj`) and MLP (`gate_proj`, `up_proj`, `down_proj`) projections. Adapters that target
other layers, such as the vision encoder, may not load for serving.

## Auto-Shutdown

Endpoints auto-stop after 1 hour of inactivity by default. Set `inactive_timeout` in minutes to
Expand Down
Loading