feat(qairt): opt-in sliding window CLI flag and env var#1160
Open
David Qian (Davidqian123) wants to merge 3 commits into
Open
feat(qairt): opt-in sliding window CLI flag and env var#1160David Qian (Davidqian123) wants to merge 3 commits into
David Qian (Davidqian123) wants to merge 3 commits into
Conversation
Adds --sliding-window to 'geniex infer' and wires GENIEX_SLIDING_WINDOW
into the qairt plugin's generate(), enabling the ring-buffer context
eviction added in the geniex-qairt-plugin submodule (bumped in this
commit). llama_cpp already has its own always-on context-shift and
ignores the env var.
No FFI/geniex.h struct change: the flag sets GENIEX_SLIDING_WINDOW=1
via os.Setenv() before model creation, read by the plugin on every
generate() call. This keeps the change scoped to sdk/plugins/qairt and
cli/, avoiding a ripple through every binding's FFI surface for what
is currently a QAIRT-only, opt-in feature.
Documents the flag/env var in cli/README.md and
docs/{en,cn}/run/cli/reference.mdx per the issue's acceptance criteria.
Ref: qcom-ai-hub/geniex#1197
Signed-off-by: Davidqian123 <yq2325@nyu.edu>
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
Contributor
|
It looks good, but I have a few questions:
|
Contributor
Author
|
Mengsheng Wu (@mengshengwu) Good catch, already Implemented. |
GENIEX_SLIDING_WINDOW=1 set via os.Setenv() from the Go CLI at runtime is not guaranteed to be visible to a separately-CRT-linked qairt plugin DLL on Windows (each DLL's CRT can hold its own environment-block snapshot) -- the flag was silently non-functional. Moves sliding_window / sliding_window_n_keep onto geniex_GenerationConfig instead, read directly by QairtLlm::generate() on every call. Per CONTRIBUTING.md, updates every binding's FFI surface in this change: Go (bindings/go, cli/cmd/geniex/infer.go), Python (bindings/python), and Android (bindings/android). Drops the GENIEX_SLIDING_WINDOW env var and its documentation entirely. BREAKING CHANGE: geniex_GenerationConfig gains two new trailing fields (sliding_window, sliding_window_n_keep); any code constructing the struct positionally (rather than by field name) needs updating. Signed-off-by: Davidqian123 <yq2325@nyu.edu>
Signed-off-by: Davidqian123 <yq2325@nyu.edu>
David Qian (Davidqian123)
force-pushed
the
feat/qairt-sliding-window
branch
from
51a0783 to
bdb18d4
Compare
Mengsheng Wu (mengshengwu)
approved these changes
Jul 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wires the opt-in sliding-window context eviction (
geniex-qairt-pluginPR #13) into the public SDK API, the CLI, and every language binding, per qcom-ai-hub/geniex#1197.third-party/geniex-qairtsubmodule pointer.sliding_window/sliding_window_n_keepfields togeniex_GenerationConfig(sdk/include/geniex.h), read directly byQairtLlm::generate().llama_cppignores these fields (it always context-shifts).--sliding-windowflag togeniex infer, passed through the typed config on everyGenerate()call.CONTRIBUTING.md): Go (bindings/go), Python (bindings/python), Android (bindings/android).cli/README.mdanddocs/{en,cn}/run/cli/reference.mdx.Why a typed config field instead of an env var
An earlier version of this PR read an env var (
GENIEX_SLIDING_WINDOW=1, set viaos.Setenv()from the Go CLI) directly in the plugin. This had a real bug on Windows:os.Setenv()called at runtime from Go is not guaranteed to be visible to a separately-CRT-linked plugin DLL (each DLL's CRT can hold its own environment-block snapshot) ----sliding-windowwas silently non-functional unless the env var was also exported in the parent shell before launching the process.Moving the flag onto
geniex_GenerationConfigfixes this (same in-process struct pointer already used formax_tokens/sampler_config) and, per review feedback, makes the feature available to the Python and Android bindings as well, not just the Go CLI.Acceptance criteria (qcom-ai-hub/geniex#1197)
--sliding-windowto the CLI enables ring-buffer eviction.Testing
cmake --build) and the Go CLI (bazelisk build //cli --config=windows_arm64) -- clean builds.clang-format,ruff check/ruff format --checkon all touched files -- clean.geniex infer qualcomm/Qwen3-4B-Instruct-2507 --sliding-windowon real hardware withGENIEX_SLIDING_WINDOWunset from the environment (confirming the old env-var dependency is gone): eviction fires (sliding window: discarding N tokens...), generation continues with nocontext length exceedederror.Related