Skip to content

fix: enforce configured model input modalities - #2896

Draft
tlokeshrao wants to merge 2 commits into
devfrom
fix/enforce-model-input-modalities
Draft

fix: enforce configured model input modalities#2896
tlokeshrao wants to merge 2 commits into
devfrom
fix/enforce-model-input-modalities

Conversation

@tlokeshrao

@tlokeshrao tlokeshrao commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Description

Enforces the input modalities configured for a model before building and sending an LLM request. The backend reads the model metadata, evaluates the selected message branch, and rejects requests containing a message part whose modality is not allowed. Models without configured input modalities retain the existing behavior.

Changes Made

  • Load and normalize inputModalities from model metadata when the model engine opens.
  • Validate message parts immediately before the model askCall.
  • Map text/system parts and media parts to the corresponding input modality.
  • Return a clear error identifying the disallowed modality and the model configuration.
  • Add unit coverage for allowed input, rejected input, message-branch validation, and missing modality metadata.

How to Test

  1. Configure a model with TEXT as its only input modality.
  2. Restart or reinitialize the model engine so its metadata is reloaded.
  3. Send a text-only request. It should succeed.
  4. Attach an image and send the request. It should be rejected before the LLM call with an error stating that IMAGE input is not allowed.
  5. Add IMAGE to the model input modalities, reinitialize the engine, and retry. The request should succeed.
  6. Remove/clear the model input modality metadata and retry. Existing unrestricted behavior should remain unchanged.

Notes

This PR enforces configured input modalities. Output modalities continue to be stored and exposed as model metadata but are not validated against generated response parts by this change.

@snyk-io

snyk-io Bot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
Open Source Security 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

@tlokeshrao tlokeshrao self-assigned this Aug 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant