Skip to content

feat(sdk): support batched prompts in geniex-bench via --- separator#1159

Draft
RemiliaForever (RemiliaForever) wants to merge 1 commit into
mainfrom
feat/bench-batch-prompts
Draft

feat(sdk): support batched prompts in geniex-bench via --- separator#1159
RemiliaForever (RemiliaForever) wants to merge 1 commit into
mainfrom
feat/bench-batch-prompts

Conversation

@RemiliaForever

Copy link
Copy Markdown
Contributor

Summary

geniex-bench's --prompt-file can now hold multiple prompts in one file, separated by a line that is exactly --- (leading/trailing whitespace on that line is ignored).

  • Each ----delimited segment runs as its own prompt, with the KV cache reset between segments (even under --no-reset-between-runs).
  • stdout marks segment boundaries with a [sep ] prompt i/n line (the previous [gen ]-prefixed header is gone, so it no longer collides with generated text).
  • A file with no --- line stays a single prompt, so timing and --accuracy runs share one code path — no branching on mode.
  • Empty / whitespace-only segments (leading, trailing, or back-to-back ---) are skipped; a file that is only separators errors out.

Only sdk/benchmark/benchmark.c changes; benchmark.c is not a public SDK header, so no FFI update is required. Existing QDC entry scripts pass single prose prompt files with no --- line, so their behavior is unchanged.

Test plan

  • gcc -Wall -Wextra compiles and links cleanly against pkg-geniex
  • clang-format --Werror clean
  • End-to-end with Qwen3-0.6B (llama_cpp, cpu): multi-segment split produces distinct, coherent per-segment output
  • Boundary cases: 3 segments, leading ---, back-to-back ---, trailing ---, no --- (single prompt), multi-line segment, whitespace-padded ---, only-separators (errors)
  • On-device QDC bench run (llama_cpp + qairt) still green

A --prompt-file may now hold multiple prompts separated by a line that is
exactly `---`; each segment runs as its own prompt with the KV cache reset
between segments, and stdout marks boundaries with `[sep ] prompt i/n`. A
file without a `---` line stays a single prompt, so timing and --accuracy
runs share one code path.

Signed-off-by: RemiliaForever <remilia@koumakan.cc>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant