feat: add LiteLLM gateway support for chat + embeddings

[prodmanpd]

Summary

Adds optional support for routing notesGPT's LLM calls through a
LiteLLM proxy (or any
OpenAI-compatible gateway), so users can point the chat + embedding calls at
100+ providers (OpenAI, Azure, Anthropic, Bedrock, Gemini, ...) behind a single
URL. Together AI stays the default - existing deployments keep working with
only TOGETHER_API_KEY set. This is purely additive: no behavior changes unless
you set the new env vars.

Motivation

Today the provider is hardcoded in convex/together.ts
(baseURL: 'https://api.together.xyz/v1' + hardcoded model strings), so there's
no way to run notesGPT against a different LLM without editing source. Routing
through a LiteLLM proxy is the standard way to swap providers, add spend
limits/fallbacks, or use models you already have keys for - without touching app
code.

Changes

• convex/llm.ts (new) - central provider config read from env with Together AI
  defaults (LLM_BASE_URL, LLM_API_KEY -> falls back to TOGETHER_API_KEY,
  LLM_CHAT_MODEL, LLM_EMBEDDING_MODEL, LLM_INSTRUCTOR_MODE) + a
  createLLMClient() factory.
• convex/together.ts - uses the shared client + config instead of the hardcoded
  new OpenAI({ baseURL: '...together...' }) and hardcoded model literals. The
  extraction and both embedding call sites now honor the config.
• README.md - documents the optional vars and a copy-pasteable LiteLLM proxy
  example.

Transcription (convex/whisper.ts) intentionally stays on Together's Whisper
endpoint: it relies on Together's file-URL upload extension (handles >1GB /
30-min audio without chunking), which the generic OpenAI transcription contract
doesn't cover. This PR gateways the chat + embedding inference.

Cross-provider fix caught while testing

Instructor's mode: 'JSON_SCHEMA' (previously hardcoded) emits Together's
proprietary response_format.schema field, which other providers reject
(Azure returned Unknown parameter: 'response_format.schema'). So a naive base-
URL swap would 400 on OpenAI/Azure/Anthropic. The fix: LLM_INSTRUCTOR_MODE
(default JSON_SCHEMA for Together back-compat) - set it to TOOLS to drive
structured output through cross-provider tool calling. Verified below.

Tests

This repo has no test harness, so here's typecheck + formatter + a real
end-to-end run.

1. Typecheck - npx tsc --noEmit -> exit 0 (clean).

2. Formatting - npx prettier --check convex/llm.ts convex/together.ts README.md -> all files use Prettier code style.

3. Live E2E - configured client -> local LiteLLM proxy -> Azure (gpt via
Azure AI Foundry), structured extraction with LLM_INSTRUCTOR_MODE=TOOLS:

base URL   : http://localhost:4000
model      : chat

=== Structured extraction (tool call through LiteLLM -> Azure) ===
finish_reason: tool_calls
usage        : {"completion_tokens":102,"prompt_tokens":186,"total_tokens":288,...}
arguments    :
{
  "title": "Tomorrow's Tasks and Errands",
  "summary": "Reschedule Thursday's dentist appointment, complete the Q3 budget spreadsheet before the 2 p.m. standup, and pick up oat milk on the way home.",
  "actionItems": [
    "Call the dentist tomorrow to reschedule Thursday's appointment.",
    "Finish the Q3 budget spreadsheet before the 2 p.m. standup tomorrow.",
    "Buy oat milk on the way home tomorrow."
  ]
}

Proxy config used (Azure behind a chat alias):

model_list:
  - model_name: chat
    litellm_params:
      model: azure_ai/<deployment>
      api_base: https://<resource>.services.ai.azure.com
      api_key: os.environ/AZURE_API_KEY
litellm_settings:
  drop_params: true

This proves the full chain: createLLMClient() -> OpenAI SDK -> LiteLLM proxy ->
Azure -> structured JSON matching the existing NoteSchema. Embeddings run
through the same configured client (togetherai.embeddings.create) and the same
base URL; I didn't have an embedding deployment on the test resource, so that
path wasn't exercised live, but it's the identical client + config.

Risk / Compatibility

• Additive only. With no new env vars set, behavior is byte-for-byte the same
  (Together AI, same models, JSON_SCHEMA mode).
• No new dependencies - reuses the existing openai SDK.
• Transcription path untouched.

Example usage

1. Define a LiteLLM proxy config (litellm.config.yaml) that fronts whatever
providers you want behind one endpoint:

model_list:
  - model_name: chat # -> LLM_CHAT_MODEL
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY
  - model_name: chat-claude # swap in with one env change
    litellm_params:
      model: anthropic/claude-3-5-sonnet-latest
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: embed # -> LLM_EMBEDDING_MODEL
    litellm_params:
      model: openai/text-embedding-3-small
      api_key: os.environ/OPENAI_API_KEY
litellm_settings:
  drop_params: true

2. Start the proxy (Python) - exposes an OpenAI-compatible API on :4000:

# pip install "litellm[proxy]"
# litellm --config litellm.config.yaml --port 4000
#
# Sanity-check it the same way notesGPT talks to it - plain OpenAI SDK:
from openai import OpenAI

client = OpenAI(base_url="http://localhost:4000", api_key="sk-litellm-anything")
resp = client.chat.completions.create(
    model="chat",
    messages=[{"role": "user", "content": "Say OK"}],
)
print(resp.choices[0].message.content)  # -> OK

3. Point notesGPT at it - set these Convex env vars (all optional; unset =
Together AI as before):

LLM_BASE_URL=http://localhost:4000
LLM_API_KEY=sk-litellm-...
LLM_CHAT_MODEL=chat            # or chat-claude, etc.
LLM_EMBEDDING_MODEL=embed
LLM_INSTRUCTOR_MODE=TOOLS      # for non-Together providers

That's it - convex/together.ts picks the client/models up automatically:

// convex/llm.ts (new) - single place all provider config lives
export function createLLMClient() {
  return new OpenAI({ apiKey: llmApiKey, baseURL: llmBaseURL });
}

// convex/together.ts - unchanged call shape, now provider-agnostic
const togetherai = createLLMClient();
const client = Instructor({ client: togetherai, mode: instructorMode });

await client.chat.completions.create({
  messages: [
    { role: 'system', content: 'Extract a title, summary, and action items...' },
    { role: 'user', content: transcript },
  ],
  model: chatModel, // Qwen on Together by default; "chat" via the proxy
  response_model: { schema: NoteSchema, name: 'SummarizeNotes' },
});

[vercel]

@prodmanpd is attempting to deploy a commit to the Together AI Team on Vercel.

A member of the Team first needs to authorize it.