feat(sdk): apply chat template in LLM bench text-prompt path#1112
Draft
RemiliaForever (RemiliaForever) wants to merge 1 commit into
Draft
feat(sdk): apply chat template in LLM bench text-prompt path#1112RemiliaForever (RemiliaForever) wants to merge 1 commit into
RemiliaForever (RemiliaForever) wants to merge 1 commit into
Conversation
The VLM bench always templated its prompt while the LLM --prompt-file path fed raw text, so LLM --accuracy runs saw an unformatted prompt unlike real inference. Run --prompt-file text through the bundle's chat template (add_generation_prompt + enable_thinking), matching the VLM path; random-ids prefill still skips the tokenizer and template.
Copilot started reviewing on behalf of
RemiliaForever (RemiliaForever)
June 27, 2026 05:00
View session
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates geniex-bench so the LLM --prompt-file (text-prompt) benchmark path applies the bundle’s chat template (matching real inference behavior and aligning with the existing VLM path), and enables “thinking” in the templating configuration to keep prompt formatting consistent.
Changes:
- Add
build_llm_prompt()to applygeniex_llm_apply_chat_template()for--prompt-fileprompts before callinggeniex_llm_generate(). - Enable thinking in the VLM prompt templating path, and update usage/help text + comments to describe the new behavior.
- Ensure
--prompt-fileformatted prompt memory is freed viageniex_free()on both success and failure paths.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Contributor
|
Mengsheng Wu (@mengshengwu) need to take a look, do not merge till next week after we finish perf scorecard run. |
Mengsheng Wu (mengshengwu)
approved these changes
Jun 28, 2026
Contributor
|
LGTM |
Contributor
Author
|
since qairt need directly pass prompt to generate, consider adding a flag. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The VLM path in
geniex-benchhas always wrapped prompts with a chat template viageniex_vlm_apply_chat_template, but the LLM--prompt-file(text-prompt) path previously fed the raw text directly intoprompt_utf8without applying a template. This made LLM--accuracyoutput inconsistent with real inference (LLM is templated in the CLI'sinfer.go).Changes
build_llm_prompt(mirroringbuild_vlm_prompt): run a single user message throughgeniex_llm_apply_chat_templatewithadd_generation_prompt = trueandenable_thinking = true.run_llmnow applies the template before feedinggin.prompt_utf8in--prompt-filemode, and frees it withgeniex_freeon both the error-exit and normal-completion paths.build_vlm_prompt) also setsenable_thinking = trueto align with LLM.run_vlmcomment, and--prompt-filehelp text accordingly.Behavioral differences remain reasonable: the default random-ids prefill mode (
-p N) still skips the tokenizer and template — it is purely a prefill-speed test by design. Template/think only takes effect on the--prompt-filepath.Notes
geniex.his untouched, so this is not an FFI change.cmake --build build --target geniex-bench.