Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 32 additions & 2 deletions src/RockBot.Agent/agent/routing-dream.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ directive and analyzer have drifted and you cannot reliably interpret the input.
- **`fallbackExcludedCount`** — fallback-triggered entries are already excluded from
clusters, flagged clusters, and keyword candidates; do not filter them yourself

## Your three jobs
## Your four jobs

### 1. Validate flagged clusters

Expand All @@ -36,6 +36,10 @@ For each entry in `flaggedClusters`:
- **panicEscalation** — Low tier with avg tool calls ≥ 3. The model likely struggled.
Check `samplePrompt` and `count`: if this looks like a real recurring shape (not a fluke),
the cluster's `alternateTier` (Balanced) is the correct routing direction.
When such a cluster is a **short user query naming a tool/topic** (e.g. "what's on my
todo list?", "any new calendar events?") that scores near-zero and routes Low, threshold
nudges can't move it and topic words can't go in `highSignalKeywords`. Use the
**`config.balancedFloorKeywords`** lever instead — see job 4.
- **tokenSurprise** — Low complexity score but high post-injection tokens. **Informational
only** — DO NOT adjust thresholds for this. Post-injection size is an orchestration-layer
concern, not a routing-quality signal. A trivial prompt like "what time is it?" legitimately
Expand Down Expand Up @@ -98,6 +102,31 @@ Trivial guard fields (`trivialGuardCeiling`, `userOriginBias`) are available in
but should only be touched when there is clear evidence of *false Low-tier routing* — never
to widen Balanced.

### 4. Floor tool-intent queries at Balanced with `balancedFloorKeywords`

Some short user queries **need a tool** but score near-zero and route **Low**, where the
small model fails to pick the right MCP tool under heavy injected context. These surface as
recurring **panicEscalation** clusters whose `samplePrompt` is a brief user request naming a
topic/tool (todo, calendar, email, reminders). Thresholds can't move a ~0.0 query, and the
topic word can't go in `highSignalKeywords` (it would over-route to expensive High and be
stripped by the topic blocklist anyway).

The lever is **`config.balancedFloorKeywords`**: when a **user-message** prompt matches a
floor keyword **and** the computed tier is **Low**, the selector escalates it to **Balanced**
(cheap but tool-capable). It **never** escalates to High, and it does **not** affect subagent
traffic. This list is **exempt from the topic blocklist** — it is *meant* to hold topic/tool
words. Additions are merged with compiled defaults (add-only).

Add a floor keyword only when a panicEscalation cluster shows a **recurring** tool-intent
shape (not a one-off). Pick the single recurring topic/tool word from the cluster's
`samplePrompt` (e.g. `todo`, `calendar`). Return your additions in the
`config.balancedFloorKeywords` array — additions only, merged with defaults.

**Keep the distinction sharp:** floor words route **Low→Balanced** (cheap + tool-capable)
and **never** to High. The "reject topic words" rule in job 2 still applies to
`highSignalKeywords` only — topic words belong in `balancedFloorKeywords`, not in the
high-signal list.

## Response format

Return ONLY a JSON object:
Expand All @@ -111,7 +140,8 @@ Return ONLY a JSON object:
"lowCeiling": 0.20,
"balancedCeiling": 0.46,
"highSignalKeywords": ["additions only — merged with compiled defaults"],
"lowSignalKeywords": ["additions only — merged with compiled defaults"]
"lowSignalKeywords": ["additions only — merged with compiled defaults"],
"balancedFloorKeywords": ["tool/topic words that floor Low→Balanced — additions only"]
},
"antiPatterns": [
{
Expand Down
9 changes: 9 additions & 0 deletions src/RockBot.Host.Abstractions/TierSelectorConfig.cs
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,15 @@ public sealed class TierSelectorConfig
/// </summary>
public List<string>? LowSignalKeywords { get; set; }

/// <summary>
/// Keywords that floor a user-message prompt at the Balanced tier when it would
/// otherwise route Low. Intended for tool/topic words (todo, calendar, ...) that
/// signal "this needs a tool" — the small Low model handles those unreliably.
/// Merged with compiled defaults (add-only). Escalates Low→Balanced only; never High.
/// Exempt from the high-signal TopicBlocklist. See issue #486.
/// </summary>
public List<string>? BalancedFloorKeywords { get; set; }

/// <summary>
/// Score threshold below which the trivial guard forces Low tier, regardless of
/// dream-tuned thresholds. Prevents simple prompts from drifting into Balanced
Expand Down
36 changes: 31 additions & 5 deletions src/RockBot.Llm/KeywordTierSelector.cs
Original file line number Diff line number Diff line change
Expand Up @@ -72,10 +72,16 @@ public sealed class KeywordTierSelector : ILlmTierSelector
"tell me about", "show me", "look up",
];

// ── Balanced-floor signals → floor a Low-routed user query at Balanced ───
// Empty by default (add-only caveat): the dream owns this list and no
// un-retractable guesses are baked in. See issue #486.
private static readonly string[] DefaultBalancedFloorKeywords = [];

private static readonly EffectiveConfig Defaults = new(
DefaultLowCeiling, DefaultBalancedCeiling,
DefaultTrivialGuardCeiling, DefaultUserOriginBias,
DefaultHighSignalKeywords, DefaultLowSignalKeywords);
DefaultHighSignalKeywords, DefaultLowSignalKeywords,
DefaultBalancedFloorKeywords);

// ── Code / math / multi-step markers ────────────────────────────────────
private static readonly Regex CodeBlockRegex = new(
Expand Down Expand Up @@ -215,6 +221,19 @@ private TierClassification ClassifyCore(string promptText, TierRoutingContext? c
tier = ModelTier.Balanced;
}

// Balanced-floor override: a first-turn user query naming a known tool/topic
// (e.g. "what's on my todo list?") scores near-zero → Low, where the small model
// fails to select the right MCP tool. When the dream has learned such a floor
// keyword, escalate Low→Balanced (never High) so it lands on a cheap but
// tool-capable tier. Exempt from TopicBlocklist by design. See issue #486.
if (context?.Origin == "user-message"
&& tier == ModelTier.Low
&& config.BalancedFloorKeywords.Length > 0
&& config.BalancedFloorKeywords.Any(k => ContainsWholePhrase(lower, k)))
{
tier = ModelTier.Balanced;
}

return new TierClassification(tier, score, matchedHigh, matchedLow);
}

Expand Down Expand Up @@ -258,6 +277,10 @@ private EffectiveConfig TryLoad()
// Merge dream keywords with compiled defaults (dream adds, never replaces).
var highKeywords = MergeKeywords(DefaultHighSignalKeywords, dto.HighSignalKeywords, "highSignalKeywords");
var lowKeywords = MergeKeywords(DefaultLowSignalKeywords, dto.LowSignalKeywords, "lowSignalKeywords");
// Balanced-floor list is exempt from the TopicBlocklist: SanitizeKeywords only
// applies the blocklist when the list name contains "high", so "balancedFloorKeywords"
// passes topic/tool words (todo, calendar, ...) through unfiltered — by design.
var floorKeywords = MergeKeywords(DefaultBalancedFloorKeywords, dto.BalancedFloorKeywords, "balancedFloorKeywords");

// Clamp dream-tuned thresholds to guardrail ranges.
var lowCeiling = ClampThreshold(dto.LowCeiling, DefaultLowCeiling, MinLowCeiling, MaxLowCeiling, "lowCeiling");
Expand All @@ -271,16 +294,18 @@ private EffectiveConfig TryLoad()
TrivialGuardCeiling: trivialGuard,
UserOriginBias: originBias,
HighSignalKeywords: highKeywords,
LowSignalKeywords: lowKeywords);
LowSignalKeywords: lowKeywords,
BalancedFloorKeywords: floorKeywords);

_logger?.LogInformation(
"KeywordTierSelector: reloaded config from {Path} " +
"(lowCeiling={Low}, balancedCeiling={Balanced}, " +
"trivialGuard={TrivialGuard}, userOriginBias={OriginBias}, " +
"highSignals={HighCount}, lowSignals={LowCount})",
"highSignals={HighCount}, lowSignals={LowCount}, floorKeywords={FloorCount})",
_configPath, result.LowCeiling, result.BalancedCeiling,
result.TrivialGuardCeiling, result.UserOriginBias,
result.HighSignalKeywords.Length, result.LowSignalKeywords.Length);
result.HighSignalKeywords.Length, result.LowSignalKeywords.Length,
result.BalancedFloorKeywords.Length);

return result;
}
Expand Down Expand Up @@ -512,7 +537,8 @@ private sealed record EffectiveConfig(
double TrivialGuardCeiling,
double UserOriginBias,
string[] HighSignalKeywords,
string[] LowSignalKeywords);
string[] LowSignalKeywords,
string[] BalancedFloorKeywords);

private sealed record CachedConfig(EffectiveConfig Config, DateTime LoadedAt);
}
138 changes: 138 additions & 0 deletions tests/RockBot.Llm.Tests/KeywordTierSelectorTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -860,6 +860,144 @@ public void Classify_LongBriefWithComplexityKeyword_StillRoutesHigh()
"A long brief with genuine complexity keywords should still route High");
}

// ── Balanced-floor override (issue #486) ─────────────────────────────────

[TestMethod]
public void BalancedFloor_UserOrigin_MatchEscalatesLowToBalanced()
{
var tempDir = Path.Combine(Path.GetTempPath(), "kts-floor-" + Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(tempDir);
try
{
var config = """{"version":1,"balancedFloorKeywords":["todo"]}""";
File.WriteAllText(Path.Combine(tempDir, "tier-selector.json"), config);

var options = Options.Create(new AgentProfileOptions { BasePath = tempDir });
var selector = new KeywordTierSelector(options, NullLogger<KeywordTierSelector>.Instance);

// A trivial tool-intent query that would otherwise route Low.
var result = selector.Classify(
"what's on my todo list?",
new TierRoutingContext(Origin: "user-message"));

Assert.AreEqual(ModelTier.Balanced, result.Tier,
"A user-origin floor-keyword match should escalate Low to Balanced");
}
finally
{
Directory.Delete(tempDir, recursive: true);
}
}

[TestMethod]
public void BalancedFloor_SubagentOrigin_NoEffect()
{
var tempDir = Path.Combine(Path.GetTempPath(), "kts-floor-sub-" + Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(tempDir);
try
{
var config = """{"version":1,"balancedFloorKeywords":["todo"]}""";
File.WriteAllText(Path.Combine(tempDir, "tier-selector.json"), config);

var options = Options.Create(new AgentProfileOptions { BasePath = tempDir });
var selector = new KeywordTierSelector(options, NullLogger<KeywordTierSelector>.Instance);

// Same prompt/floor, subagent origin — the floor override must not fire.
var result = selector.Classify(
"what's on my todo list?",
new TierRoutingContext(Origin: "subagent"));

Assert.AreEqual(ModelTier.Low, result.Tier,
"The balanced-floor override must not affect subagent traffic");
}
finally
{
Directory.Delete(tempDir, recursive: true);
}
}

[TestMethod]
public void BalancedFloor_NeverEscalatesToHigh()
{
var tempDir = Path.Combine(Path.GetTempPath(), "kts-floor-nohigh-" + Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(tempDir);
try
{
var config = """{"version":1,"balancedFloorKeywords":["todo"]}""";
File.WriteAllText(Path.Combine(tempDir, "tier-selector.json"), config);

var options = Options.Create(new AgentProfileOptions { BasePath = tempDir });
var selector = new KeywordTierSelector(options, NullLogger<KeywordTierSelector>.Instance);

var result = selector.Classify(
"what's on my todo list?",
new TierRoutingContext(Origin: "user-message"));

Assert.AreNotEqual(ModelTier.High, result.Tier,
"The floor override only fires on Low and only sets Balanced — never High");
}
finally
{
Directory.Delete(tempDir, recursive: true);
}
}

[TestMethod]
public void BalancedFloor_NonMatchingTrivialChat_StaysLow()
{
var tempDir = Path.Combine(Path.GetTempPath(), "kts-floor-nomatch-" + Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(tempDir);
try
{
var config = """{"version":1,"balancedFloorKeywords":["todo"]}""";
File.WriteAllText(Path.Combine(tempDir, "tier-selector.json"), config);

var options = Options.Create(new AgentProfileOptions { BasePath = tempDir });
var selector = new KeywordTierSelector(options, NullLogger<KeywordTierSelector>.Instance);

// Trivial user chat that does NOT name the floor keyword — must stay Low.
var result = selector.Classify(
"what's the weather?",
new TierRoutingContext(Origin: "user-message"));

Assert.AreEqual(ModelTier.Low, result.Tier,
"A trivial user prompt not matching any floor keyword must stay Low");
}
finally
{
Directory.Delete(tempDir, recursive: true);
}
}

[TestMethod]
public void BalancedFloor_TopicBlocklistExemption_FloorKeywordStillFires()
{
var tempDir = Path.Combine(Path.GetTempPath(), "kts-floor-exempt-" + Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(tempDir);
try
{
// "todo" is a blocklisted topic word (stripped from high-signal). It must
// survive in the balanced-floor list and still fire the override, proving
// the floor list is exempt from the TopicBlocklist.
var config = """{"version":1,"balancedFloorKeywords":["todo"]}""";
File.WriteAllText(Path.Combine(tempDir, "tier-selector.json"), config);

var options = Options.Create(new AgentProfileOptions { BasePath = tempDir });
var selector = new KeywordTierSelector(options, NullLogger<KeywordTierSelector>.Instance);

var result = selector.Classify(
"what's on my todo list?",
new TierRoutingContext(Origin: "user-message"));

Assert.AreEqual(ModelTier.Balanced, result.Tier,
"Blocklisted topic word 'todo' must not be stripped from the floor list");
}
finally
{
Directory.Delete(tempDir, recursive: true);
}
}

[TestMethod]
public void ActiveThreadOverride_MatchedHighKeyword_GateBlocksPromotion()
{
Expand Down
Loading