Skip to content

docs(custom-inference): document local model tunneling via ngrok #139

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hongyi-chen merged 3 commits into from
May 27, 2026
Merged

docs(custom-inference): document local model tunneling via ngrok #139

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
9f30309
Document local model tunneling for custom inference endpoints
hongyi-chen May 26, 2026
3f8b3ad
Merge branch 'main' into docs/custom-inference-local-models-ngrok
hongyi-chen May 26, 2026
0874dac
Update src/content/docs/agent-platform/inference/custom-inference-end…
hongyi-chen May 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions src/content/docs/agent-platform/inference/custom-inference-endpoint.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,14 @@ When you explicitly select an endpoint-routed model from the model picker, Warp

The configuration flow mirrors the [Bring Your Own API Key](/agent-platform/inference/bring-your-own-api-key/) setup, so the steps will feel familiar if you've already configured BYOK.

## Using local models

Warp routes inference requests through its servers, so endpoint URLs must be publicly accessible. `localhost`, `127.0.0.1`, and other private or local network URLs are rejected when configuring a custom inference endpoint.

To route through a model running on your own machine (for example, Ollama, LM Studio, vLLM, or llama.cpp), expose it through a tunneling service like [ngrok](https://ngrok.com/) and use the public tunnel URL as the base URL in your endpoint configuration.

@oz-for-oss oz-for-oss Bot May 26, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [IMPORTANT] [SECURITY] This recommends putting a local model server on a public URL without warning users to protect it; add a caution to require tunnel access controls and close the tunnel when finished.

Suggested change
To route through a model running on your own machine (for example, Ollama, LM Studio, vLLM, or llama.cpp), expose it through a tunneling service like [ngrok](https://ngrok.com/) and use the public tunnel URL as the base URL in your endpoint configuration.
To route through a model running on your own machine (for example, Ollama, LM Studio, vLLM, or llama.cpp), expose it through a tunneling service like [ngrok](https://ngrok.com/) and use the public tunnel URL as the base URL in your endpoint configuration.
:::caution
A public tunnel can expose your local model server to anyone with the URL. Require authentication or access controls in the tunneling service, and close the tunnel when you are done.
:::


For example, with a default Ollama install listening on port `11434`, run `ngrok http 11434` and use the resulting `https://*.ngrok-free.app/v1` URL as your endpoint. Other tunneling services that produce a publicly reachable HTTPS URL (Cloudflare Tunnel, Tailscale Funnel, and similar) work the same way.

## Billing behavior

### Warp AI credits
Expand Down
Loading