Skip to content

feat(cli): add roll-call command for batch-testing model connectivity#463

Open
cbrunnkvist wants to merge 1 commit intoKilo-Org:devfrom
cbrunnkvist:feat/roll-call-command
Open

feat(cli): add roll-call command for batch-testing model connectivity#463
cbrunnkvist wants to merge 1 commit intoKilo-Org:devfrom
cbrunnkvist:feat/roll-call-command

Conversation

@cbrunnkvist
Copy link

@cbrunnkvist cbrunnkvist commented Feb 19, 2026

Summary

This PR adds a new roll-call subcommand that allows users to batch-test multiple models for connectivity and latency. This helps users discover which provider/model pairs are available and working.

Features

  • Filter models by regex pattern (required for safety - prevents accidental testing of all 300+ models)
  • Configurable prompt, timeout, and parallelization
  • JSON or table output formats
  • Proper provider options handling for reasoning models

Technical Details

The implementation:

  • Applies ProviderTransform options (maxOutputTokens, temperature, topP, topK, providerOptions)
  • Enables reasoning for dedicated thinking models via Kilo Gateway (models with "thinking" in their ID)
  • Uses proper session-based provider option generation

Testing

Tested with various model filters:

% bun run dev -- roll-call "free"                                                                                                                                                   [1]
$ bun run --cwd packages/opencode --conditions=browser src/index.ts roll-call free
Starting roll call for models with prompt: "Hello"
Timeout per model: 25000ms, Parallel calls: 5
Prompting 6 models...
Model                                    | Access | Snippet                                                                                      | Latency
----------------------------------------------------------------------------------------------------------------------------------------------------------
kilo/arcee-ai/trinity-large-preview:free | YES    | Hey there! I'm AFM, your multipurpose autonomous software agent. How can I assist you today? | 1215ms
kilo/openrouter/free                     | YES    | Hello! How can I assist you today?                                                           | 1681ms
kilo/minimax/minimax-m2.5:free           | YES    | Hello! How can I help you today?                                                             | 3171ms
kilo/stepfun/step-3.5-flash:free         | YES    | Hello! 👋 How can I assist you today?                                                        | 6340ms
kilo/z-ai/glm-5:free                     | YES    | Hello! How can I help you today?                                                             | 17428ms
kilo/corethink:free                      | NO     | (Invalid JSON response)                                                                      | 19473ms

5 accessible, 1 failed

Demonstrating truncation, error handling, color coding:

Screenshot 2026-02-20 at 11 22 15

Related Issue

Closes #457

@vercel
Copy link

vercel bot commented Feb 19, 2026

@cbrunnkvist is attempting to deploy a commit to the Kilo Code Team on Vercel.

A member of the Team first needs to authorize it.

@cbrunnkvist cbrunnkvist force-pushed the feat/roll-call-command branch from a8337fd to adb9f5d Compare February 19, 2026 10:08
@cbrunnkvist cbrunnkvist marked this pull request as draft February 19, 2026 10:58
@cbrunnkvist cbrunnkvist force-pushed the feat/roll-call-command branch 2 times, most recently from 26183e0 to 21f6738 Compare February 19, 2026 14:49
@cbrunnkvist cbrunnkvist changed the title feat: add roll-call command for batch-testing model connectivity feat(cli): add roll-call command for batch-testing model connectivity Feb 20, 2026
@cbrunnkvist cbrunnkvist force-pushed the feat/roll-call-command branch from 21f6738 to edafafb Compare February 20, 2026 02:41
@cbrunnkvist cbrunnkvist force-pushed the feat/roll-call-command branch from 733b6c7 to 886c3bb Compare February 20, 2026 04:06
@cbrunnkvist cbrunnkvist marked this pull request as ready for review February 20, 2026 04:27
.option("retries", {
type: "number",
default: 0,
describe: "Number of additional retries for each model call",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: The retries option is defined here but never used in the handler.

On line 129, the destructuring const { prompt, timeout, filter, parallel, output, verbose, quiet } = args omits retries, and no retry logic exists in processModel. Users passing --retries 3 would expect failed model calls to be retried, but the flag silently does nothing.

Either implement retry logic in processModel or remove this option to avoid misleading users.

@@ -0,0 +1,298 @@
import type { Argv } from "yargs"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Missing kilocode_change - new file marker.

Per AGENTS.md, new files outside of kilocode-named directories should have a // kilocode_change - new file comment at the top to help identify Kilo-specific additions during upstream merges. The test file has this marker but the source file does not.

Suggested change
import type { Argv } from "yargs"
// kilocode_change - new file
import type { Argv } from "yargs"

import { UpgradeCommand } from "./cli/cmd/upgrade"
import { UninstallCommand } from "./cli/cmd/uninstall"
import { ModelsCommand } from "./cli/cmd/models"
import { RollCallCommand } from "./cli/cmd/roll-call"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Missing kilocode_change marker on this new import.

Per AGENTS.md, modifications to shared upstream files should be marked with kilocode_change comments so they can be easily identified during merges.

Suggested change
import { RollCallCommand } from "./cli/cmd/roll-call"
import { RollCallCommand } from "./cli/cmd/roll-call" // kilocode_change

.command(ServeCommand)
.command(WebCommand)
.command(ModelsCommand)
.command(RollCallCommand)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Missing kilocode_change marker on this new command registration.

Suggested change
.command(RollCallCommand)
.command(RollCallCommand) // kilocode_change

@kiloconnect
Copy link
Contributor

kiloconnect bot commented Feb 20, 2026

Code Review Summary

Status: 4 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 1
SUGGESTION 3
Issue Details (click to expand)

WARNING

File Line Issue
packages/opencode/src/cli/cmd/roll-call.ts 95 --retries CLI option is defined but never used — users passing --retries N get silently ignored

SUGGESTION

File Line Issue
packages/opencode/src/cli/cmd/roll-call.ts 1 Missing // kilocode_change - new file marker (test file has it, source file does not)
packages/opencode/src/index.ts 11 Missing kilocode_change marker on new import
packages/opencode/src/index.ts 135 Missing kilocode_change marker on new command registration
Files Reviewed (3 files)
  • packages/opencode/src/cli/cmd/roll-call.ts - 2 issues (1 WARNING, 1 SUGGESTION)
  • packages/opencode/src/index.ts - 2 issues (2 SUGGESTION)
  • packages/opencode/test/cli/roll-call.test.ts - 0 issues

Fix these issues in Kilo Cloud

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] Help user discover their provider/model pair availability

1 participant

Comments