prompt-rag JSONL parser fails on Claude/non-OpenAI providers (document-rag times out)

[beca-oc]

Version: trustgraph-flow 2.3.21 (Docker image docker.io/trustgraph/trustgraph-flow:2.3.21)

Repro

1. Configure text-completion + text-completion-rag processors with trustgraph.model.text_completion.claude.Processor instead of openai.Processor.
2. Set CLAUDE_KEY env, restart workers, restart flow with --param llm-model=claude-haiku-4-5-20251001 --param llm-rag-model=claude-haiku-4-5-20251001.
3. Verify bare LLM works:

   tg-invoke-llm -f sizzl-ontology "" "Say PONG"
   # -> "PONG" in ~2s, HTTP 200 from api.anthropic.com
   

4. Try tg-invoke-document-rag:

   tg-invoke-document-rag -u http://api-gateway:8088/ -f sizzl-ontology -C sizzl-market -d 2 -q "What does X sell?"
   # -> Exception (empty body) at ~33s
   

Observed logs (docker logs deploy-rag-1)

prompt-rag - WARNING - JSONL parse error on line 1: Expecting value: line 1 column 1 (char 0)
prompt-rag - WARNING - JSONL parse error on line 3: Expecting value: line 1 column 1 (char 0)
prompt-rag - WARNING - JSONL parse error on line 5: Expecting value: line 1 column 1 (char 0)
prompt-rag - WARNING - JSONL parse returned no valid objects
... (repeats)

document-rag - ERROR - Exception processing response:
asyncio.exceptions.CancelledError ...
TimeoutError

text-completion-rag DID get HTTP 200 from Anthropic with token counts logged. The response is plain prose. prompt-rag's parser expects JSONL, gets nothing parseable, returns empty, document-rag times out at 30s.

Root cause

The kg-synthesis / prompt-rag pipeline assumes JSONL output. OpenAI models follow that instruction; Claude (and likely Gemini, Mistral) return prose by default. There's no provider-specific output coercion.

Suggested fix paths

1. Per-provider prompt overlays — register the kg-synthesis template with an Anthropic-friendly variant that uses pre-fill (messages=[..., {"role":"assistant","content":"{"}]) to force JSON.
2. Output coercion in prompt-rag worker — if the response doesn't parse as JSONL, attempt to extract the JSON block from a fenced code block (e.g. json ... ), then re-parse.
3. Document this limitation in the provider docs so operators know Anthropic/Gemini are not drop-in replacements for OpenAI in RAG flows.

I can prepare a PR for (2) if helpful.

Workaround we're using

We bypass the bus on the read path entirely (direct Bolt to Memgraph + direct provider call from the app tier). Works fine; loses TG-mediated tracing.

[beca-oc]

Root cause + proposed patch

Reproduced live in TG 2.3.21 against Claude (claude-3-5-sonnet-20241022) on flow sizzl-everything. prompt-rag (document-rag step) hangs ~60s then times out. Log shows the LLM responded fine — parser drops everything.

The bug is in trustgraph/template/prompt_manager.py::parse_jsonl(). It splits on \n and json.loads() each line. That only works for true JSONL (one object per line). Claude — and any model not specifically prompted with "respond as JSONL" — returns a JSON array instead:

[
  {"question": "..."},
  {"question": "..."}
]

Every line is invalid JSON in isolation ([, {...},, ]), each json.loads raises, and the warning loop swallows them silently. The function returns []. Downstream document-rag gets an empty question list and the chain stalls.

Proposed patch

Add a JSON-array fallback before the line-by-line loop. Keep JSONL behavior as the second-chance path so existing OpenAI-tuned prompts don't regress.

# trustgraph/template/prompt_manager.py
def parse_jsonl(self, text):
    results = []

    # Strip markdown code fences if present
    text = text.strip()
    if text.startswith('```'):
        text = re.sub(r'^```(?:json|jsonl)?\s*\n?', '', text)
    if text.endswith('```'):
        text = text[:-3]

    text = text.strip()

    # NEW: JSON array fallback (Claude, Gemini, most non-OpenAI models)
    if text.startswith('['):
        try:
            parsed = json.loads(text)
            if isinstance(parsed, list):
                return parsed
        except json.JSONDecodeError:
            pass  # fall through to line-by-line

    # Existing JSONL path (one object per line)
    for line_num, line in enumerate(text.split('\n'), 1):
        line = line.strip()
        if not line or line.startswith('```'):
            continue
        try:
            results.append(json.loads(line))
        except json.JSONDecodeError as e:
            logger.warning(f"JSONL parse error on line {line_num}: {e}")

    return results

Why this is the right shape

• Provider-agnostic: no per-model prompt tuning required.
• Truncation-resilient: keeps the JSONL fallback for partial output.
• ~10 LoC, no new dependencies.
• The model name / provider doesn't have to leak into the prompt manager.

Alternative considered

Prompting Claude to "respond as JSONL, one object per line" works but is fragile (Claude routinely ignores it for >5-item outputs and reverts to arrays). The parser is the right place to fix this.

Source pointers (TG 2.3.21)

• trustgraph/template/prompt_manager.py:96-131 — the broken loop
• trustgraph/service/prompt_rag.py (call site) — receives [] and silently no-ops

Happy to open a PR if useful. Filing this here first because it touches the prompt manager core path.

[cybermaggedon]

@beca-oc thanks for the detailed report. I can't reproduce this at the moment. I've tested with claude-haiku-4-5-20251001. You reported that the issue occurs when using tg-invoke-document-rag, which uses two prompts, extract-concepts following by document-synthesis. In default configuration, neither of these prompts uses JSONL output.

Perhaps you could check the prompts you are using, did they get re-configured to use JSONL, output? Both should be text.

document-prompt:
+----------+-----------------------------------------------------------------------+
| prompt   | Study the following context. Use only the information provided in the |
|          | context in your response. Do not speculate if the answer is not found |
|          | in the provided set of knowledge statements.                          |
|          | Here is the context:                                                  |
|          | {{documents}}                                                         |
|          | Use only the provided knowledge statements to respond to the          |
|          | following:                                                            |
|          | {{query}}                                                             |
| response | text                                                                  |
+----------+-----------------------------------------------------------------------+

[cybermaggedon]

Using the document-synthesis prompt with Claude 4.5 Haiku...

[cybermaggedon]

I have also tested, the same model with a prompt which does use JSON, extract-relationships, and this provides expected output:

[cybermaggedon]

Can you check whether you've modified the prompts? Prompts sometimes need domain-specific modifications, the system is E2E tested with some broad knowledge domains, but no prompt can be universally perfect, which is why TrustGraph includes prompt configuration tooling.

[cybermaggedon]

FYI, TrustGraph is test/evaluated against many LLM types. Claude, VertexAI get a lot of testing. Small models: Mistral, various forms of Gemma 4, Nemo 12b. OpenAI models on Azure. We don't do a huge amount of testing with the OpenAI service itself.