Qwen 3 1.7B Offline tool calling Android

[vkkhare]

Here's a comprehensive PR description for all the changes:

Description

This PR adds comprehensive support for Qwen 3 1.7B model with tool calling capabilities, uses native ONNX runtime isntead of onnxruntime_genai, and adds the export script for model enhancements.

Key Features Added

• Adds mlc/tokenizer-cpp as third party dependency which brings cpp bindings for rust implementations in HF
• Adds Qwen 3 1.7B support with tool calling in nimblenet_py with module support via zip
• FP16 data type support with proper uint16_t handling in binary operations
• ONNX model export with simplified token_id predictions instead of logits

Cpp bindings

// - dist/tokenizer.json
void HuggingFaceTokenizerExample() {
  // Read blob from file.
  auto blob = LoadBytesFromFile("dist/tokenizer.json");
  // Note: all the current factory APIs takes in-memory blob as input.
  // This gives some flexibility on how these blobs can be read.
  auto tok = Tokenizer::FromBlobJSON(blob);
  std::string prompt = "What is the capital of Canada?";
  // call Encode to turn prompt into token ids
  std::vector<int> ids = tok->Encode(prompt);
  // call Decode to turn ids into string
  std::string decoded_prompt = tok->Decode(ids);
}

void SentencePieceTokenizerExample() {
  // Read blob from file.
  auto blob = LoadBytesFromFile("dist/tokenizer.model");
  // Note: all the current factory APIs takes in-memory blob as input.
  // This gives some flexibility on how these blobs can be read.
  auto tok = Tokenizer::FromBlobSentencePiece(blob);
  std::string prompt = "What is the capital of Canada?";
  // call Encode to turn prompt into token ids
  std::vector<int> ids = tok->Encode(prompt);
  // call Decode to turn ids into string
  std::string decoded_prompt = tok->Decode(ids);
}

Delitepy Bindings

from delitepy import tokenizers

tokenizer = tokenizers.from_json(<tokenizer.json>)
token_ids = tokenizer.encode(text)
input_ids = nm.tensor([token_ids], "int64")
response = tokenizer.decode(input_ids)

FP16 Support

Enhanced binary operations now support FP16 data type through uint16_t:

# FP16 tensors now supported in all binary operations
fp16_tensor = nm.tensor(data, "float16")  # Uses uint16_t internally
result = fp16_tensor + other_tensor  # Works with add, sub, mult, div, pow, mod

Kotlin Interface

Reverse stream of generation from python and subscription in kotlin flows.

    private fun createNimbleNetTensorFromForeignFunction(fn: (String?) -> Unit) : NimbleNetTensor {
        val callbackDelitePy : DelitePyForeignFunction =  fun(input: NimbleNetTensorMap?): NimbleNetTensorMap? {
            val outputStream = input?.get("token_stream")?.data as String?
            fn(outputStream)
            return hashMapOf("result" to NimbleNetTensor(data = true, datatype = DATATYPE.BOOL, shape = intArrayOf()))
        }
        return NimbleNetTensor(data = callbackDelitePy, datatype = DATATYPE.FUNCTION, shape = intArrayOf())
    }

    suspend fun feedInput(input: String, isVoiceInitiated: Boolean, callback: (String?)->Unit) : String? {
        val res = NimbleNet.runMethod(
            "prompt_for_tool_calling",
            inputs = hashMapOf(
                "prompt" to NimbleNetTensor(input, DATATYPE.STRING, null),
                "output_stream_callback" to  createNimbleNetTensorFromForeignFunction(callback)
            ),
        )
        assert(res.status) { "NimbleNet.runMethod('prompt_for_tool_calling') failed with status: ${res.status}" }
        return res.payload?.get("results")?.data as String?
    }

Qwen Demo Setup

The Qwen demo uses a zip-based modules in delitePy

cd nimblenet_py/simulation_assets/qwen_demo
zip -j qwen_modules.zip qwen_modules/*.py
python run_demo.py

Tool Calling Features

• Multi-step conversation support with automatic tool execution
• JSON-based tool calling with <tool_call> XML tags
• Built-in tools: weather, math calculator, time, location
• Error handling and recovery for failed tool calls

Checklist:

• I have added tests that prove my fix is effective or that my feature works
• Has user-facing changes. This may include API or behavior changes and performance improvements, etc

[jpuneet]

1. Is this accidental change?
2. cp -R is the portable form, compared to cp -r.

[vkkhare]

I agree

[jpuneet]

This doesn't look correct.

[vkkhare]

yup will update this