fix: voice STT full transcript and German device tools#2
Merged
Conversation
Accumulate STT until listening ends so early finalResult no longer sends a partial message. Add open-app/alarm intent shortcuts and restore tool capability text when native FC is disabled on Android Gemma 4. Co-authored-by: Cursor <cursoragent@cursor.com>
Reviewer's GuideImplements more robust voice input handling (accumulated STT transcript + live preview), deterministic Android open-app and alarm shortcuts with German support, and restores on-device tool capabilities in compact prompts, plus corresponding tests and beginner-mode guards. Sequence diagram for updated voice STT accumulation and send behaviorsequenceDiagram
actor User
participant VoiceInputButton
participant SpeechToText as SpeechToText
participant AssistantScreen
participant ModelOrchestrator
User->>VoiceInputButton: press and hold
VoiceInputButton->>SpeechToText: listen(SpeechListenOptions)
SpeechToText-->>VoiceInputButton: onResult(result)
VoiceInputButton->>VoiceInputButton: _lastWords = result.recognizedWords
VoiceInputButton-->>AssistantScreen: onPartial(_lastWords)
AssistantScreen->>AssistantScreen: _inputController.text = partial
User->>VoiceInputButton: release / engine pauseFor
SpeechToText-->>VoiceInputButton: onStatus(done | notListening)
VoiceInputButton->>VoiceInputButton: _finishListening()
VoiceInputButton->>SpeechToText: stop()
VoiceInputButton->>VoiceInputButton: transcript = _lastWords.trim()
VoiceInputButton-->>AssistantScreen: onTranscription(transcript)
AssistantScreen->>AssistantScreen: _inputController.text = transcript
AssistantScreen->>AssistantScreen: [if _isGenerating or ModelOrchestrator.isBusy] keep text only
AssistantScreen->>ModelOrchestrator: _sendMessage() [else]
ModelOrchestrator-->>AssistantScreen: response
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
|
Normalize umlauts so geöffnet is not treated as an open intent, gate device-tool capability text to Android, and align beginner voice busy checks with the main chat screen. Co-authored-by: Cursor <cursoragent@cursor.com>
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 3 issues, and left some high level feedback:
- In
OpenAppIntentParser.tryParsePackage, matching aliases vialower.contains(alias)can produce false positives for substrings inside longer words (e.g.chromeinchromebook); consider enforcing word boundaries or token-based matching for aliases to reduce accidental app resolutions. - In
_startListeningyouawait _speech.listenbut ignore its boolean result; consider handling the failure case (e.g. returning early and resetting_isListening) so the UI state doesn’t get stuck if listening cannot actually start.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- In `OpenAppIntentParser.tryParsePackage`, matching aliases via `lower.contains(alias)` can produce false positives for substrings inside longer words (e.g. `chrome` in `chromebook`); consider enforcing word boundaries or token-based matching for aliases to reduce accidental app resolutions.
- In `_startListening` you `await _speech.listen` but ignore its boolean result; consider handling the failure case (e.g. returning early and resetting `_isListening`) so the UI state doesn’t get stuck if listening cannot actually start.
## Individual Comments
### Comment 1
<location path="lib/services/model_orchestrator.dart" line_range="1583-1579" />
<code_context>
}
+ // Deterministic open-app shortcut (YouTube, Settings, etc.).
+ final openPackage = OpenAppIntentParser.tryParsePackage(query);
+ if (openPackage != null) {
+ _statusController.add('Opening app...');
+ final result = await ToolExecutorService.instance.openApp(openPackage);
+ final ok = result['success'] == true;
+ yield InferenceResult(
+ text: ok
+ ? 'Opened $openPackage.'
+ : 'Could not open the app: ${result['error'] ?? 'unknown error'}',
+ model: selector.primaryHeavy,
+ isStreaming: false,
+ toolCalls: [
+ {
+ 'name': 'open_app',
+ 'args': {'package': openPackage},
+ 'result': result,
+ },
+ ],
+ );
+ return;
+ }
+
</code_context>
<issue_to_address>
**issue (bug_risk):** Open-app shortcut currently bypasses any confirmation or disambiguation, which may be surprising on ambiguous queries.
Because this shortcut fires on any query that parses as both an open intent and a known alias, it will also trigger on informational prompts like “Explain how to open YouTube settings”, issuing an `open_app` call instead of letting the model respond. Consider tightening the heuristic (e.g., require the intent and alias to be close/within the same clause, or skip obvious “how to”/explanatory phrasing) so purely informational queries don’t invoke the tool.
</issue_to_address>
### Comment 2
<location path="lib/utils/open_app_intent_parser.dart" line_range="17-26" />
<code_context>
+ );
+
+ /// Common app aliases → package names.
+ static const Map<String, String> knownApps = {
+ 'youtube': 'com.google.android.youtube',
+ 'yt': 'com.google.android.youtube',
+ 'settings': 'com.android.settings',
+ 'einstellungen': 'com.android.settings',
+ 'chrome': 'com.android.chrome',
+ 'maps': 'com.google.android.apps.maps',
+ 'google maps': 'com.google.android.apps.maps',
+ 'kamera': 'com.android.camera',
+ 'camera': 'com.android.camera',
+ 'photos': 'com.google.android.apps.photos',
+ 'galeria': 'com.google.android.apps.photos',
+ 'gallery': 'com.google.android.apps.photos',
+ 'whatsapp': 'com.whatsapp',
+ 'spotify': 'com.spotify.music',
+ 'clock': 'com.android.deskclock',
+ 'uhr': 'com.android.deskclock',
+ 'wecker': 'com.android.deskclock',
+ };
+
</code_context>
<issue_to_address>
**suggestion:** Review alias list and matching strategy for coverage and accidental matches.
Two small follow‑ups:
- Confirm whether `'galeria'` is intentional; if so, consider also adding `'galerie'`.
- `lower.contains(alias)` may match aliases as substrings inside longer words. For some entries (e.g. WhatsApp), consider using word‑boundary‑style matching (like `\bwhatsapp\b`) to reduce accidental matches without much added complexity.
Suggested implementation:
```
/// Common app aliases → package names.
///
/// Note:
/// - `"galeria"` is intentionally kept as a Spanish/Portuguese variant.
/// - `"galerie"` is added as the German variant.
static const Map<String, String> knownApps = {
'youtube': 'com.google.android.youtube',
'yt': 'com.google.android.youtube',
'settings': 'com.android.settings',
'einstellungen': 'com.android.settings',
'chrome': 'com.android.chrome',
'maps': 'com.google.android.apps.maps',
'google maps': 'com.google.android.apps.maps',
'kamera': 'com.android.camera',
'camera': 'com.android.camera',
'photos': 'com.google.android.apps.photos',
'galeria': 'com.google.android.apps.photos',
'galerie': 'com.google.android.apps.photos',
'gallery': 'com.google.android.apps.photos',
'whatsapp': 'com.whatsapp',
'spotify': 'com.spotify.music',
'clock': 'com.android.deskclock',
'uhr': 'com.android.deskclock',
'wecker': 'com.android.deskclock',
};
```
I can’t see the call site that uses `knownApps`, but your comment mentions a `lower.contains(alias)` check. To implement “word‑boundary‑style” matching (especially for entries like `"whatsapp"`), you should:
1. Locate the loop where `knownApps` is used, something like:
```dart
for (final entry in knownApps.entries) {
if (lower.contains(entry.key)) {
return entry.value;
}
}
```
2. Replace the `contains` check with a small helper that uses a `RegExp` with word boundaries for alphabetic aliases, falling back to `contains` when needed. For example:
```dart
bool _matchesAlias(String lowerUtterance, String alias) {
// Use word boundaries for simple alphabetic aliases (e.g. "whatsapp", "spotify").
final isAlphabetic = RegExp(r'^[a-z]+(?:\s+[a-z]+)*$').hasMatch(alias);
if (isAlphabetic) {
final pattern = RegExp('\\b${RegExp.escape(alias)}\\b', caseSensitive: false);
return pattern.hasMatch(lowerUtterance);
}
// Fallback: plain substring match for aliases with non‑letters / special cases.
return lowerUtterance.contains(alias.toLowerCase());
}
```
3. Then update the loop to use the helper:
```dart
for (final entry in knownApps.entries) {
if (_matchesAlias(lower, entry.key)) {
return entry.value;
}
}
```
This preserves the current behavior for most aliases while reducing accidental matches for names like `"whatsapp"` and `"spotify"`. You may want to place `_matchesAlias` near the other regex helpers in `open_app_intent_parser.dart`.
</issue_to_address>
### Comment 3
<location path="lib/utils/open_app_intent_parser.dart" line_range="11-8" />
<code_context>
+ caseSensitive: false,
+ );
+
+ static final RegExp _machAufIntent = RegExp(
+ r'\bmach(?:en)?\s+\S+\s+auf\b',
+ caseSensitive: false,
+ );
+
</code_context>
<issue_to_address>
**suggestion:** The `mach ... auf` pattern misses some natural German phrasings.
This regex won’t match phrases with extra words between `mach` and `auf`, e.g. "mach mal YouTube auf" or "mach bitte die Kamera auf". To support these, consider allowing a limited span between them, e.g. `\bmach(?:en)?\b[\s\w]{0,20}\bauf\b`, or a similar bounded pattern to avoid over‑matching.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
| @@ -1578,6 +1579,29 @@ class ModelOrchestrator { | |||
| return; | |||
There was a problem hiding this comment.
issue (bug_risk): Open-app shortcut currently bypasses any confirmation or disambiguation, which may be surprising on ambiguous queries.
Because this shortcut fires on any query that parses as both an open intent and a known alias, it will also trigger on informational prompts like “Explain how to open YouTube settings”, issuing an open_app call instead of letting the model respond. Consider tightening the heuristic (e.g., require the intent and alias to be close/within the same clause, or skip obvious “how to”/explanatory phrasing) so purely informational queries don’t invoke the tool.
Comment on lines
+17
to
+26
| static const Map<String, String> knownApps = { | ||
| 'youtube': 'com.google.android.youtube', | ||
| 'yt': 'com.google.android.youtube', | ||
| 'settings': 'com.android.settings', | ||
| 'einstellungen': 'com.android.settings', | ||
| 'chrome': 'com.android.chrome', | ||
| 'maps': 'com.google.android.apps.maps', | ||
| 'google maps': 'com.google.android.apps.maps', | ||
| 'kamera': 'com.android.camera', | ||
| 'camera': 'com.android.camera', |
There was a problem hiding this comment.
suggestion: Review alias list and matching strategy for coverage and accidental matches.
Two small follow‑ups:
- Confirm whether
'galeria'is intentional; if so, consider also adding'galerie'. lower.contains(alias)may match aliases as substrings inside longer words. For some entries (e.g. WhatsApp), consider using word‑boundary‑style matching (like\bwhatsapp\b) to reduce accidental matches without much added complexity.
Suggested implementation:
/// Common app aliases → package names.
///
/// Note:
/// - `"galeria"` is intentionally kept as a Spanish/Portuguese variant.
/// - `"galerie"` is added as the German variant.
static const Map<String, String> knownApps = {
'youtube': 'com.google.android.youtube',
'yt': 'com.google.android.youtube',
'settings': 'com.android.settings',
'einstellungen': 'com.android.settings',
'chrome': 'com.android.chrome',
'maps': 'com.google.android.apps.maps',
'google maps': 'com.google.android.apps.maps',
'kamera': 'com.android.camera',
'camera': 'com.android.camera',
'photos': 'com.google.android.apps.photos',
'galeria': 'com.google.android.apps.photos',
'galerie': 'com.google.android.apps.photos',
'gallery': 'com.google.android.apps.photos',
'whatsapp': 'com.whatsapp',
'spotify': 'com.spotify.music',
'clock': 'com.android.deskclock',
'uhr': 'com.android.deskclock',
'wecker': 'com.android.deskclock',
};
I can’t see the call site that uses knownApps, but your comment mentions a lower.contains(alias) check. To implement “word‑boundary‑style” matching (especially for entries like "whatsapp"), you should:
- Locate the loop where
knownAppsis used, something like:for (final entry in knownApps.entries) { if (lower.contains(entry.key)) { return entry.value; } }
- Replace the
containscheck with a small helper that uses aRegExpwith word boundaries for alphabetic aliases, falling back tocontainswhen needed. For example:bool _matchesAlias(String lowerUtterance, String alias) { // Use word boundaries for simple alphabetic aliases (e.g. "whatsapp", "spotify"). final isAlphabetic = RegExp(r'^[a-z]+(?:\s+[a-z]+)*$').hasMatch(alias); if (isAlphabetic) { final pattern = RegExp('\\b${RegExp.escape(alias)}\\b', caseSensitive: false); return pattern.hasMatch(lowerUtterance); } // Fallback: plain substring match for aliases with non‑letters / special cases. return lowerUtterance.contains(alias.toLowerCase()); }
- Then update the loop to use the helper:
for (final entry in knownApps.entries) { if (_matchesAlias(lower, entry.key)) { return entry.value; } }
This preserves the current behavior for most aliases while reducing accidental matches for names like "whatsapp" and "spotify". You may want to place _matchesAlias near the other regex helpers in open_app_intent_parser.dart.
| /// ASCII open verbs (safe with `\b`). Umlaut forms use normalized matching. | ||
| static final RegExp _asciiOpenIntent = RegExp( | ||
| r'\b(open|launch|start|starte|oeffne|oeffnen|aufmachen)\b', | ||
| caseSensitive: false, |
There was a problem hiding this comment.
suggestion: The mach ... auf pattern misses some natural German phrasings.
This regex won’t match phrases with extra words between mach and auf, e.g. "mach mal YouTube auf" or "mach bitte die Kamera auf". To support these, consider allowing a limited span between them, e.g. \bmach(?:en)?\b[\s\w]{0,20}\bauf\b, or a similar bounded pattern to avoid over‑matching.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a deterministic open-app shortcut parser supporting English and German, extends the alarm parser to support German queries, and improves voice input handling to support partial transcriptions and prevent premature message sending. The review feedback suggests refining the open-app parser by using word boundaries to avoid false substring matches, handling negated queries, and adding the German 'galerie' alias. Additionally, it recommends dynamically building the device tools capability prompt in the orchestrator to prevent the model from hallucinating unavailable tools.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
Comment on lines
+64
to
+68
| for (final alias in aliases) { | ||
| if (lower.contains(alias)) { | ||
| return knownApps[alias]; | ||
| } | ||
| } |
There was a problem hiding this comment.
Using lower.contains(alias) performs a substring match rather than a whole-word match. This leads to critical false positives where common words containing the alias as a substring trigger the app opening (e.g., "Python" matching "yt" and opening YouTube, or "Ruhr" matching "uhr" and opening the Clock app). We should use a regular expression with word boundaries (\b) to ensure we only match whole words.
Suggested change
| for (final alias in aliases) { | |
| if (lower.contains(alias)) { | |
| return knownApps[alias]; | |
| } | |
| } | |
| for (final alias in aliases) { | |
| final pattern = RegExp(r'\b' + RegExp.escape(alias) + r'\b'); | |
| if (pattern.hasMatch(lower)) { | |
| return knownApps[alias]; | |
| } | |
| } |
Comment on lines
+57
to
+58
| static String? tryParsePackage(String query) { | ||
| if (!_hasOpenIntent(query)) return null; |
There was a problem hiding this comment.
The deterministic open-app shortcut currently matches queries even if they contain negations (e.g., "Don't open YouTube" or "öffne YouTube nicht"). This results in the app being opened when the user explicitly requested the opposite. We should add a negation check to return null immediately if negation keywords are detected.
Suggested change
| static String? tryParsePackage(String query) { | |
| if (!_hasOpenIntent(query)) return null; | |
| static String? tryParsePackage(String query) { | |
| final negationPattern = RegExp( | |
| r"\b(don\s*'?\s*t|not|no|nicht|kein|keine|nie|niemals|verhindern|stoppen|stop)\b", | |
| caseSensitive: false, | |
| ); | |
| if (negationPattern.hasMatch(query)) return null; | |
| if (!_hasOpenIntent(query)) return null; |
Comment on lines
+28
to
+29
| 'galeria': 'com.google.android.apps.photos', | ||
| 'gallery': 'com.google.android.apps.photos', |
There was a problem hiding this comment.
The German word for gallery is "Galerie", but it is currently missing from knownApps (which only has "galeria" and "gallery"). Adding "galerie" ensures German users can open the photos app using their native term.
Suggested change
| 'galeria': 'com.google.android.apps.photos', | |
| 'gallery': 'com.google.android.apps.photos', | |
| 'galeria': 'com.google.android.apps.photos', | |
| 'galerie': 'com.google.android.apps.photos', | |
| 'gallery': 'com.google.android.apps.photos', |
Comment on lines
+2383
to
+2400
| String _deviceToolsCapabilitySuffix(List<Tool>? textTools) { | ||
| final names = (textTools ?? const <Tool>[]) | ||
| .map((t) => t.name) | ||
| .where((n) => n.isNotEmpty) | ||
| .toList(); | ||
| final listed = names.isEmpty | ||
| ? 'get_time, set_alarm, cancel_alarm, open_app, open_settings, search_web' | ||
| : names.join(', '); | ||
|
|
||
| return ' You control this Android device via tools: $listed. ' | ||
| 'When the user asks to open an app, set/cancel an alarm, open settings, ' | ||
| 'or search the web, call the tool immediately — do not say you lack ' | ||
| 'device access. ' | ||
| 'open_app packages: com.google.android.youtube, com.android.settings, ' | ||
| 'com.android.chrome. ' | ||
| 'If native function calling is unavailable, reply with only JSON like ' | ||
| '{"name":"open_app","arguments":{"package":"com.google.android.youtube"}}.'; | ||
| } |
There was a problem hiding this comment.
The capability suffix prompt contains hardcoded instructions telling the model to call specific tools (like open_app, set_alarm, etc.) even if those tools are not actually present in the textTools list. This can cause the model to hallucinate or attempt to call unavailable tools. We should dynamically build the instructions based on the tools actually present in textTools.
String _deviceToolsCapabilitySuffix(List<Tool>? textTools) {
final names = (textTools ?? const <Tool>[])
.map((t) => t.name)
.where((n) => n.isNotEmpty)
.toList();
final hasTools = names.isNotEmpty;
final listed = hasTools
? names.join(', ')
: 'get_time, set_alarm, cancel_alarm, open_app, open_settings, search_web';
final buffer = StringBuffer(' You control this Android device via tools: $listed. ');
final actions = <String>[];
if (!hasTools || names.contains('open_app')) actions.add('open an app');
if (!hasTools || names.contains('set_alarm') || names.contains('cancel_alarm')) actions.add('set/cancel an alarm');
if (!hasTools || names.contains('open_settings')) actions.add('open settings');
if (!hasTools || names.contains('search_web')) actions.add('search the web');
if (actions.isNotEmpty) {
final actionStr = actions.join(', ');
buffer.write('When the user asks to $actionStr, call the tool immediately — do not say you lack device access. ');
}
if (!hasTools || names.contains('open_app')) {
buffer.write('open_app packages: com.google.android.youtube, com.android.settings, com.android.chrome. ');
}
buffer.write('If native function calling is unavailable, reply with only JSON like '
'\'{"name":"open_app","arguments":{"package":"com.google.android.youtube"}}\'.');
return buffer.toString();
}![kilo-code-bot[bot]](https://avatars.githubusercontent.com/in/2193792?s=60&v=4){kind=small}
| // Engine may auto-stop after pauseFor — flush full transcript once. | ||
| final ended = status == 'done' || status == 'notListening'; | ||
| if (ended && _isListening) { | ||
| unawaited(_finishListening()); |
There was a problem hiding this comment.
WARNING: unawaited(_finishListening()) swallows exceptions from _speech.stop()
If _speech.stop() throws, the exception is silently dropped because of unawaited, making speech failures impossible to debug from logs.
Reply with @kilocode-bot fix it to have Kilo Code address this issue.
Code Review SummaryStatus: 1 Issue Found | Recommendation: Address before merge Overview
Issue Details (click to expand)WARNING
Files Reviewed (9 files)
Fix these issues in Kilo Cloud Reviewed by step-3.7-flash · Input: 172K · Output: 47.3K · Cached: 4.8M |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan
öffne YouTube/stell einen Wecker um 19:00should act without model refusalIst YouTube schon geöffnet?should NOT open YouTubeflutter test test/utils/open_app_intent_parser_test.dart test/utils/alarm_time_parser_test.dartSummary by Sourcery
Improve voice input transcription reliability and add deterministic Android device tool handling for German and English open-app and alarm requests.
New Features:
Bug Fixes:
Enhancements:
Tests: