Deploy a Triton server on Modal serving a text embedding model.
Triton is a high-performance model-serving engine from NVIDIA, designed to serve multiple models concurrently and fully utilize available GPU resources. Triton loads models from a model repository--a directory containing one or more models, as follows:
MODEL_REPO/
├── gte-embedding/ # model name
│ ├── 1/ # model version
│ │ └── model.py # model implementaiton
│ └── config.pbtxt # Triton configuration file
The Triton configuration file (config.pbtxt) contains detail about how this model should be run (docs). In this case, we run the embedding model Alibaba-NLP/gte-large-en-v1.5 using transformers (and torch) as a backend (Triton can also run other backends, like ONNX).
The Triton process is exposed with Modal Servers, Modal's high performance endpoint server. Deploy it using:
modal deploy triton_server.pyThe server endpoint will be available with a public URL.
Run the example script with a prompt as follows:
$ python example.py --prompt "Triton on Modal"
Sending request to triton_url='<YOUR-URL>'
elapsed = 0.209s
embedding dim = 1024
embeddings[0][:8] = [-0.040832020342350006, -0.021930428221821785, 0.010849617421627045, 0.0037354002706706524, -0.023134568706154823, -0.02002531662583351, -0.020769918337464333, -0.015160921961069107]