feat: OpenAI-compatible API Endpoints for Embedding Models (vLLM)

[yinggeh]

What does the PR do?

• Enable /v1/embeddings inference request for OpenAI API frontend in vLLM container

Checklist

• PR title reflects the change and is of format <commit_type>: <Title>
• Changes are described in the pull request.
• Related issues are referenced.
• Populated github labels field
• Added test plan and verified test passes.
• Verified that the PR passes existing CI.
• Verified copyright is correct on all changed files.
• Added succinct git squash message before merging ref.
• All template sections are filled out.
• Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

Check the conventional commit type
box here and add the label to the github PR.

• feat

Related PRs:

triton-inference-server/vllm_backend#104

Where should the reviewer start?

Test plan:

• CI Pipeline ID:

Caveats:

Background

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

• closes GitHub issue: #xxx

[yinggeh]

Rebase to r25.10 to run pipeline.

[pskiran1]

LGTM! Thank you for adding this feature.

[whoisj]

overall, this LGTM.

left one suggestion, but I approve of these changes as they are.

[whoisj]

wouldn't this be safer as below?

if backend == 'trtllm':
  # do something
elif backend == 'vllm':
  # do something else
else:
  raise ValueError(f'Unknown backend "{backend}" provided.')

[yinggeh]

server/python/openai/openai_frontend/engine/triton_engine.py

Lines 475 to 499 in b48425e

	# Explicitly handle ensembles to avoid any runtime validation errors
	if not backend and model.config()["platform"] == "ensemble":
	backend = "ensemble"
	print(f"Found model: {name=}, {backend=}")
	lora_names = None
	if self.backend == "vllm" or backend == "vllm":
	lora_names = _get_vllm_lora_names(
	self.server.options.model_repository, name, model.version
	)
	metadata = TritonModelMetadata(
	name=name,
	backend=backend,
	model=model,
	tokenizer=self.tokenizer,
	lora_names=lora_names,
	create_time=self.create_time,
	inference_request_converter=self._determine_request_converter(
	backend, RequestKind.GENERATION
	),
	embedding_request_converter=self._determine_request_converter(
	backend, RequestKind.EMBEDDING
	),
	)

backend can be ensemble.

[whoisj]

makes sense.

when backend == "ensemble" then we hit this code:

        if request_type == RequestKind.GENERATION:
            return _create_trtllm_generate_request
        else:
            return _create_trtllm_embedding_request

is that desirable?

also, adding the switch-like statement future-proofs the function.

[yinggeh]

    # Use TRT-LLM format as default for everything else. This could be
    # an ensemble, a python or BLS model, a TRT-LLM backend model, etc.