Add Tool Calling and Reasoning Token Metadata to genai_config.json with Fallback Map

[sayanshaw24]

Add tool-calling and reasoning token IDs to GenAI Tokenizer

Summary

Adds four new token ID fields to the model section of genai_config.json — bot_token_id, eot_token_id, bor_token_id, eor_token_id — and exposes them on the Tokenizer (alongside bos, eos, pad), with a backward-compatible fallback for older model packages.

This establishes a new naming convention for generation-level special tokens:

Abbreviation	Expansion	Purpose
bot	beginning of tool (call)	marks start of tool-call content
eot	end of tool (call)	marks end of tool-call content
bor	beginning of reasoning	marks start of reasoning/thinking content
eor	end of reasoning	marks end of reasoning/thinking content

The naming mirrors bos/eos/pad — short, unambiguous, and follows the same access pattern (GetBotTokenId() alongside GetBosTokenId()).

Changes

Config parsing (config.h / config.cpp)

• Added to Config::Model: bot_token_id, eot_token_id, bor_token_id, eor_token_id (int, default -1)
• Parsed in Model_Element::OnValue() alongside existing token ID fields (bos, eos, pad)

Tokenizer (models/model.h / models/model.cpp)

• New private members: int32_t bot_token_id_, eot_token_id_, bor_token_id_, eor_token_id_
• New public getters: GetBotTokenId(), GetEotTokenId(), GetBorTokenId(), GetEorTokenId()
• Initialized in the Tokenizer constructor from config (identical pattern to bos/eos/pad)
• Fallback logic: if any ID is -1 after reading config, attempts to resolve it by encoding known model-family-specific token strings through the vocabulary. This provides backward compatibility for Foundry Local consuming older model packages that predate these config fields.
  • Fallback map keyed by model.type:
    • qwen2/qwen3/phi3 → <tool_call> / </tool_call>
    • gptoss → <|start|> / <|call|>
    • qwen3 → <think> / </think> (reasoning)

Public C API (ort_genai_c.h / ort_genai_c.cpp)

• OgaTokenizerGetBotTokenId(tokenizer, &token_id) — returns BOT token id or -1
• OgaTokenizerGetEotTokenId(tokenizer, &token_id) — returns EOT token id or -1
• OgaTokenizerGetBorTokenId(tokenizer, &token_id) — returns BOR token id or -1
• OgaTokenizerGetEorTokenId(tokenizer, &token_id) — returns EOR token id or -1

Same pattern as OgaTokenizerGetBosTokenId / OgaTokenizerGetPadTokenId.

C++ wrapper (ort_genai.h)

• int32_t OgaTokenizer::GetBotTokenId()
• int32_t OgaTokenizer::GetEotTokenId()
• int32_t OgaTokenizer::GetBorTokenId()
• int32_t OgaTokenizer::GetEorTokenId()

C# (Tokenizer.cs)

• int Tokenizer.GetBotTokenId()
• int Tokenizer.GetEotTokenId()
• int Tokenizer.GetBorTokenId()
• int Tokenizer.GetEorTokenId()

Java (Tokenizer.java)

• int tokenizer.getBotTokenId()
• int tokenizer.getEotTokenId()
• int tokenizer.getBorTokenId()
• int tokenizer.getEorTokenId()

Objective-C (ort_genai_objc.h)

• - (int32_t)getBotTokenId:(NSError**)error
• - (int32_t)getEotTokenId:(NSError**)error
• - (int32_t)getBorTokenId:(NSError**)error
• - (int32_t)getEorTokenId:(NSError**)error

Python (python.cpp)

• tokenizer.bot_token_id (read-only property)
• tokenizer.eot_token_id
• tokenizer.bor_token_id
• tokenizer.eor_token_id

Tests (test/c_api_tests.cpp)

• TagId_Unknown — verifies gpt2 model (not in fallback map, no config IDs) returns -1 for all four
• TagId_FromConfig — creates a temp model dir with token IDs in the model section, verifies correct parsing via the tokenizer

genai_config.json format

{
  "model": {
    "type": "qwen3",
    "bos_token_id": 151643,
    "eos_token_id": 151645,
    "bot_token_id": 151657,
    "eot_token_id": 151658,
    "bor_token_id": 151659,
    "eor_token_id": 151660,
    "decoder": { ... }
  }
}

All four fields are optional. If absent, the fallback map attempts to encode the known string via the tokenizer vocabulary. If the model type isn't in the fallback map either, -1 is returned (model doesn't support tool calling / reasoning).

Motivation

This is required for the Foundry Local Catalog v2 migration where tool/reasoning metadata is being removed from catalog.

Also, the Foundry Local C++ SDK currently detects tool-call and reasoning tokens by double-decoding every token through both a normal and a special-token tokenizer stream, then doing string.find("tool_call") / string.find("think") per token. This is expensive.

With token IDs exposed by GenAI on the Tokenizer, FL can do a single integer comparison per token in the decode loop — eliminating the special stream entirely for models with configured IDs.

Design Decisions

Why on the Tokenizer (not Model)?

• BOS, EOS, and PAD token IDs are already accessed from the Tokenizer
• Token IDs are a tokenizer-level concept — they describe the vocabulary
• Follows existing pattern: Tokenizer::GetBosTokenId() → Tokenizer::GetBotTokenId()
• No lazy init needed; resolved once in the constructor

Why bot/eot/bor/eor naming?

• Mirrors bos/eos/pad — short, memorable, consistent
• Config field names also follow this pattern: bot_token_id alongside bos_token_id

Fallback map

• Exists specifically for Foundry Local backward compatibility with older model packages
• Only triggered when config fields are not set
• Resolved eagerly in the Tokenizer constructor (no lazy cache needed since tokenizer is already available)

Related PRs

• Foundry-Local: Optimize tool/reasoning detection with token ID comparison in decode loop

[Copilot]

Pull request overview

This PR adds support for exposing model “generation tags” (tool-calling and reasoning start/end tokens) via genai_config.json, with a model-type-based fallback for older model artifacts. It extends the internal config/model layers and surfaces a generic tag getter through the public C API and C++ wrapper, plus adds C API tests to validate both fallback and explicit config parsing.

Changes:

• Extend config schema/parsing with optional tool_calling and reasoning sections (Config::ToolCalling, Config::Reasoning).
• Add Model::GetTag(tag_name) with a static fallback map keyed by model.type.
• Expose tag lookup via OgaModelGetTag (C API) and OgaModel::GetTag (C++ wrapper), and add tests covering fallback and config-driven values.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
test/c_api_tests.cpp	Adds tests for tag fallback behavior and tag parsing from a temp genai_config.json.
src/ort_genai.h	Adds C++ wrapper OgaModel::GetTag(...) calling the new C API.
src/ort_genai_c.h	Adds public C API declaration/docs for OgaModelGetTag(...).
src/ort_genai_c.cpp	Implements OgaModelGetTag(...) by allocating a returned string from Model::GetTag.
src/models/model.h	Declares internal Model::GetTag(...) accessor (config + fallback).
src/models/model.cpp	Implements fallback map + config accessors for tag lookup.
src/config.h	Adds Config::ToolCalling and Config::Reasoning structs to the config model.
src/config.cpp	Wires new SAX elements (ToolCalling_Element, Reasoning_Element) into root parsing.

[rymikula]

The phi3 entry in the fallback map looks wrong, and the catalog metadata this migration replaces confirms it.

The map resolves phi3 to <tool_call> and </tool_call>. But the current Foundry Local catalog entry for Phi-4-mini-instruct has toolCallStart <|tool_call|> and toolCallEnd <|/tool_call|> (with pipes), and those are what the model's added_tokens.json actually defines (ids 200025 and 200026). The string <tool_call> without pipes does not exist anywhere in Phi-4-mini's vocabulary, so TokenToTokenId("<tool_call>") cannot resolve and the IDs stay -1. Phi-4-mini is exactly the model this fallback exists for: its genai_config.json has model.type "phi3" and no bot/eot token IDs.

Consequence once the catalog metadata is removed (microsoft/foundry-local#838): with the IDs unresolved, the enrichment step writes no tool-call marker strings into ModelInfo and no longer marks the model as supporting tool calling. ToolCallStreamAccumulator degrades to passthrough when its markers are empty, so the <|tool_call|>[...] output Phi-4-mini produces under tool_choice "required" would come through as plain text in content with an empty tool_calls array instead of structured tool calls. That path works today because the markers come from the catalog, so this would be a regression.

Suggested fix: map phi3 to <|tool_call|> and <|/tool_call|>, matching the catalog metadata being migrated.

Separate note: under tool_choice "auto", Phi-4-mini emits its tool call as a bare JSON array with no markers at all, because the request's tools are never rendered into its prompt (microsoft/foundry-local#874). So marker detection alone cannot cover the auto case even with corrected IDs.

[sayanshaw24]

The phi3 entry in the fallback map looks wrong, and the catalog metadata this migration replaces confirms it.

The map resolves phi3 to <tool_call> and </tool_call>. But the current Foundry Local catalog entry for Phi-4-mini-instruct has toolCallStart <|tool_call|> and toolCallEnd <|/tool_call|> (with pipes), and those are what the model's added_tokens.json actually defines (ids 200025 and 200026). The string <tool_call> without pipes does not exist anywhere in Phi-4-mini's vocabulary, so TokenToTokenId("<tool_call>") cannot resolve and the IDs stay -1. Phi-4-mini is exactly the model this fallback exists for: its genai_config.json has model.type "phi3" and no bot/eot token IDs.

Consequence once the catalog metadata is removed (microsoft/Foundry-Local#838): with the IDs unresolved, the enrichment step writes no tool-call marker strings into ModelInfo and no longer marks the model as supporting tool calling. ToolCallStreamAccumulator degrades to passthrough when its markers are empty, so the <|tool_call|>[...] output Phi-4-mini produces under tool_choice "required" would come through as plain text in content with an empty tool_calls array instead of structured tool calls. That path works today because the markers come from the catalog, so this would be a regression.

Suggested fix: map phi3 to <|tool_call|> and <|/tool_call|>, matching the catalog metadata being migrated.

Separate note: under tool_choice "auto", Phi-4-mini emits its tool call as a bare JSON array with no markers at all, because the request's tools are never rendered into its prompt (microsoft/Foundry-Local#874). So marker detection alone cannot cover the auto case even with corrected IDs.

Great catch! Phi-4-mini uses <|tool_call|> / <|/tool_call|> (ids 200025/200026), not the pipe-less variants. Fixed in the latest commit: {"phi3", {{"tool_call_start", "<|tool_call|>"}, {"tool_call_end", "<|/tool_call|>"}}}.

Re: the tool_choice: "auto" note - agreed that's a separate issue (tools never rendered into the prompt), tracked in #874.

This issue still exists and I thought it was already fixed.