Incorrect endpoint being used for Local LLMs

[Saher-Anwar]

I've setup ollama (llama3) on my pc.

.env file has the following fields:

LLM_URL=http://127.0.0.1:11434
LLM_MODEL=ollama

The error:

For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404
00:17:02 - INFO - [362/365] score=0  Software Developer
00:17:02 - INFO - HTTP Request: POST http://127.0.0.1:11434/chat/completions "HTTP/1.1 404 Not Found"
00:17:02 - ERROR - LLM error scoring job 'Senior Cloud Security Engineer': Client error '404 Not Found' for url 'http://127.0.0.1:11434/chat/completions'

It seems to be defaulting to the /chat/completions endpoint for local LLMs which seems to be causing the issue. On looking at the code, it seems that the assumed response will not work either.

[jayf0x]

same issue. Quick fix could be llm.py ~L168

  chat_endpoint = "api/generate" if self.base_url.startswith("http://localhost:11434") else "chat/completion"
   
resp = self._client.post(
   f"{self.base_url}/{chat_endpoint}",
    json=payload,
     headers=headers,
)

or slightly cleaner:

# L77
_OLLAMA_NATIVE_BASE = "http://localhost:11434"

# in __init__()
self._is_ollama: bool = base_url.startswith(_OLLAMA_NATIVE_BASE)
...
self.chat_endpoint = "api/generate" if self._is_ollama else "chat/completion"
self.chat_url = f"{self.base_url}/{self.chat_endpoint}"

# ~L168
resp = self._client.post(
    self.chat_url,
    json=payload,
    headers=headers,
)

note: untested.

@Saher-Anwar ps. LLM_MODEL should be the actual name of the model, like qwen3:9b.

@claude Could you assist and fact check this? Optionally also add a small sys prompt in the ask method to aid accuracy, especially for local models.

[Sandy2636]

@jayf0x

The API should be called on /api/chat end point not /api/generate , since the payload is in form of messages not a single prompt.

otherwise will get error as below. as the model is loaded but does not know what to do.

i.e.
` {
"model": "qwen3.5:2b ",
"created_at": "2026-04-01T18:20:22.5351195Z",
"response": "",
"done": true,
"done_reason": "load"
}