Skip to content

QUICKSTART.md

Latest commit

 

History

History
165 lines (111 loc) · 5.68 KB

QUICKSTART.md

File metadata and controls

165 lines (111 loc) · 5.68 KB
Raw

Quick Start

This guide gets the proxy running and talking to a backend in minutes. You can run it as a container in Azure Container Apps or directly from source. Point it at a backend with a Host setting, then verify it with the health probe. Everything else (scaling, telemetry, async, APIM) builds on this same starting point.

If you already have .NET 10 installed, then follow the run as code path. If not, the container option is pretty quick as well. You can alternatively run locally as a container if you're good with docker.

TL;DR

  • Clone the repo
  • Deploy to Azure Container Apps or run it locally

Prerequisites

Expand to see the details The proxy needs a backend to talk to. It supports the following: - LLM endpoint(s) - Azure API Management

With the backends in hand, you have two options for running it:

Container

Run as code

Optional Scenarios
Item Notes
Application Insights Tracks telemetry and activity
Azure API Management For Governance and Compliance scenarios
Azure CosmosDB User Profiles
Azure Event Hub If you want to connect to Stream Analytics, DataDog, Splunk, ...
Azure Functions Async mode, LLM Simulator, User Profiles
Azure Service Bus Async mode
Azure Storage Account Async mode

Clone The Repo

git clone https://github.com/microsoft/SimpleL7Proxy.git

Deploy to Azure Container Apps

The deployment to Container Apps is driven by an interactive install script.

  • Deploy Script — You'll specify the details of your installation in a configuration file and then follow the steps.

Run as Code

Pick a backend host:

# LLM endpoint
export Host1="host=https://<endpoint>.openai.azure.com;mode=direct;path=/; processor=MultiLineAllUsage"

# LLM endpoint with API key
export Host1="host=https://<endpoint>.openai.azure.com;mode=direct;path=/; processor=MultiLineAllUsage; api-key=<your-api-key>"

# LLM endpoint with MI
export Host1="host=https://<endpoint>.openai.azure.com;mode=direct;path=/; processor=MultiLineAllUsage; usemi=true;audience=https://cognitiveservices.azure.com;"

# APIM 
export Host1=""

Set the listening port

# Port the proxy listens on
export Port=8000

Run the Proxy

cd SimpleL7Proxy/src/SimpleL7Proxy
dotnet run

If you see this line on the console, it means it started up: alt text

Tip

🎉 You're up and running! The proxy is live and ready to take traffic.

The proxy starts on port 8000. It generates logs in the current folder as eventslog.json.

Check the log file

tail -f eventslog.json

Note

If you are using Managed Identity for tokens, the initial startup can take a few seconds while the tokens are downloaded.

The console is going to be noisy, and you can tune down the logging by setting LogToConsole to something other than *.

Check the health probe

Replace the http://localhost:8000 with your URL if running in container apps:

curl -i http://localhost:8000/health

alt text

Query the proxy

Now that the proxy is setup we can query the LLM using curl. You should also be able to use any SDK you're comfortable with.

Set your hostname:

export PROXYHOST="http://localhost:8000"

Set the URL and Body for the test: ( this is using gpt-4o):

export URL="openai/v1/chat/completions"
export BODY='{"model":"gpt-4o","messages":[{"role":"user","content":"hello"}],"stream":true}'

Run the query:

curl -i -H "Content-Type: application/json" -d "$BODY" "$PROXYHOST/$URL"

You should see a status code of 200 with a response similar to:

alt text

Configure Azure App Configuration

Environment variables are a tedious way to configure the proxy. Alternatively, you can use Azure App Configuration, where the proxy pulls changes every 30 seconds.

Note

Follow the deploy script to create the App Configuration ( Step 7 ).

export AZURE_APPCONFIG_ENDPOINT=https://your-appconfig.azconfig.io
export AZURE_APPCONFIG_LABEL=dev