Expand FFI bindings with grammar, LoRA, embeddings, and more#1
Open
nbeerbower wants to merge 1 commit into
Open
Expand FFI bindings with grammar, LoRA, embeddings, and more#1nbeerbower wants to merge 1 commit into
nbeerbower wants to merge 1 commit into
Conversation
New FFI bindings: - Grammar sampling (llama_sampler_init_grammar) for structured output - LoRA adapter support (load, apply, remove adapters) - State/session save and load functions - Additional samplers (XTC, top-n-sigma, infill, DRY) - FIM (Fill-in-Middle) tokens for code completion - Extended model metadata access - Chat template formatting - Embeddings extraction helpers - Control vectors support - Context settings (causal attention, warmup mode) - Additional vocab functions (sep, mask, score, attr) - NUMA and RPC support detection New high-level API functions: - sampler_create_with_grammar() - Grammar-constrained sampling - lora_load/apply/clear() - LoRA adapter management - state_save/load() - Session persistence - chat_format() - Format messages with model template - extract_embeddings() - Get text embeddings as array - fim_prompt() - Create FIM prompts for code models - model_info_extended() - Extended model information - model_meta/model_meta_all() - Access GGUF metadata New examples: - json_output.hml - Generate structured JSON - lora.hml - LoRA adapter usage - embeddings.hml - Text embeddings and similarity - fim.hml - Fill-in-middle code completion - session.hml - Save/restore conversation state Constants added: - Token types and attributes - ROPE_SCALING_LONGROPE - GRAMMAR_JSON (built-in JSON grammar)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New FFI bindings:
New high-level API functions:
New examples:
Constants added: