Skip to content

fix: voice STT full transcript and German device tools #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
involvex merged 2 commits into from
Jul 18, 2026
Merged

fix: voice STT full transcript and German device tools #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5fb8a7f
fix: send full voice transcripts and honor German device tools
involvex Jul 18, 2026
fbde0cf
fix: harden open-app parsing and Android-only tool suffix
involvex Jul 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 12 additions & 3 deletions lib/screens/assistant_screen.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2109,11 +2109,20 @@ class _AssistantScreenState extends State<AssistantScreen>
),
const SizedBox(width: 8),
VoiceInputButton(
onPartial: (partial) {
_inputController.text = partial;
_inputController.selection = TextSelection.collapsed(
offset: partial.length,
);
},
onTranscription: (transcript) {
if (transcript.isNotEmpty) {
_inputController.text = transcript;
_sendMessage();
if (transcript.isEmpty) return;
_inputController.text = transcript;
if (_isGenerating || ModelOrchestrator.instance.isBusy) {
// Leave full text in the field for the user to send.
return;
}
_sendMessage();
},
),
const SizedBox(width: 8),
Expand Down
14 changes: 11 additions & 3 deletions lib/screens/assistant_screen_beginner.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -489,11 +489,19 @@ class _AssistantScreenBeginnerState extends State<AssistantScreenBeginner> {
),
const SizedBox(width: 12),
VoiceInputButton(
onPartial: (partial) {
_inputController.text = partial;
_inputController.selection = TextSelection.collapsed(
offset: partial.length,
);
},
onTranscription: (text) {
if (text.isNotEmpty) {
_inputController.text = text;
_sendMessage();
if (text.isEmpty) return;
_inputController.text = text;
if (_isGenerating || ModelOrchestrator.instance.isBusy) {
return;
}
_sendMessage();
},
),
const SizedBox(width: 8),
Expand Down
57 changes: 57 additions & 0 deletions lib/services/model_orchestrator.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ import 'package:nova_assistant/services/platform_adaptation_service.dart';
import 'package:nova_assistant/utils/alarm_time_parser.dart';
import 'package:nova_assistant/utils/agent_debug_log.dart';
import 'package:nova_assistant/utils/message_limits.dart';
import 'package:nova_assistant/utils/open_app_intent_parser.dart';
import 'package:nova_assistant/services/memory_diagnostics_service.dart';
import 'package:nova_assistant/services/session_history_reinjection.dart';

Expand Down Expand Up @@ -1578,6 +1579,29 @@ class ModelOrchestrator {
return;

@sourcery-ai sourcery-ai Bot Jul 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (bug_risk): Open-app shortcut currently bypasses any confirmation or disambiguation, which may be surprising on ambiguous queries.

Because this shortcut fires on any query that parses as both an open intent and a known alias, it will also trigger on informational prompts like “Explain how to open YouTube settings”, issuing an open_app call instead of letting the model respond. Consider tightening the heuristic (e.g., require the intent and alias to be close/within the same clause, or skip obvious “how to”/explanatory phrasing) so purely informational queries don’t invoke the tool.

}

// Deterministic open-app shortcut (YouTube, Settings, etc.).
final openPackage = OpenAppIntentParser.tryParsePackage(query);
if (openPackage != null) {
_statusController.add('Opening app...');
final result = await ToolExecutorService.instance.openApp(openPackage);
final ok = result['success'] == true;
yield InferenceResult(
text: ok
? 'Opened $openPackage.'
: 'Could not open the app: ${result['error'] ?? 'unknown error'}',
model: selector.primaryHeavy,
isStreaming: false,
toolCalls: [
{
'name': 'open_app',
'args': {'package': openPackage},
'result': result,
},
],
);
return;
}

// Check if there are image attachments that need vision processing
final hasImageAttachments = attachments.any((att) {
if (att.type != AttachedDataType.file) return false;
Expand Down Expand Up @@ -1702,6 +1726,7 @@ class ModelOrchestrator {
}

final chatTools = _toolsForCreateChat(model, tools);
final textToolPrompt = chatTools.isEmpty && tools.isNotEmpty;
final wasNull = _activeChat == null;
if (wasNull) {
_isLoadingModel = true;
Expand All @@ -1713,6 +1738,7 @@ class ModelOrchestrator {
model,
ragContext,
_capContextInjection(attachmentContext),
textToolPrompt ? tools : null,
),
tools: chatTools,
supportImage: model.hasVision && _activeModelSupportsImage,
Expand Down Expand Up @@ -2283,6 +2309,7 @@ class ModelOrchestrator {
NovaModel model, [
String? ragContext,
String? attachmentContext,
List<Tool>? textTools,
]) async {
final compact =
!kIsWeb &&
Expand Down Expand Up @@ -2312,6 +2339,15 @@ class ModelOrchestrator {
buffer.write('\n\n${await _languageInstruction()}');
}

// Compact Android prompts drop the role's tool list — restore capability
// so the model does not claim it cannot open apps / set alarms.
// Gate to Android only: web FC-off must not get "Android device" wording.
final isAndroid =
!kIsWeb && defaultTargetPlatform == TargetPlatform.android;
if (isAndroid && (compact || (textTools != null && textTools.isNotEmpty))) {
buffer.write(_deviceToolsCapabilitySuffix(textTools));
}

if (_adultModeEnabled) {
buffer.write(AdultModePolicy.systemPromptSuffix(compact: compact));
}
Expand Down Expand Up @@ -2342,6 +2378,27 @@ class ModelOrchestrator {
return buffer.toString();
}

/// Short capability block for on-device tools (and text-JSON FC when native FC
/// is unavailable on Android Gemma 4).
String _deviceToolsCapabilitySuffix(List<Tool>? textTools) {
final names = (textTools ?? const <Tool>[])
.map((t) => t.name)
.where((n) => n.isNotEmpty)
.toList();
final listed = names.isEmpty
? 'get_time, set_alarm, cancel_alarm, open_app, open_settings, search_web'
: names.join(', ');

return ' You control this Android device via tools: $listed. '
'When the user asks to open an app, set/cancel an alarm, open settings, '
'or search the web, call the tool immediately — do not say you lack '
'device access. '
'open_app packages: com.google.android.youtube, com.android.settings, '
'com.android.chrome. '
'If native function calling is unavailable, reply with only JSON like '
'{"name":"open_app","arguments":{"package":"com.google.android.youtube"}}.';
}
Comment on lines +2383 to +2400

@gemini-code-assist gemini-code-assist Bot Jul 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The capability suffix prompt contains hardcoded instructions telling the model to call specific tools (like open_app, set_alarm, etc.) even if those tools are not actually present in the textTools list. This can cause the model to hallucinate or attempt to call unavailable tools. We should dynamically build the instructions based on the tools actually present in textTools.

  String _deviceToolsCapabilitySuffix(List<Tool>? textTools) {
    final names = (textTools ?? const <Tool>[])
        .map((t) => t.name)
        .where((n) => n.isNotEmpty)
        .toList();
    final hasTools = names.isNotEmpty;
    final listed = hasTools
        ? names.join(', ')
        : 'get_time, set_alarm, cancel_alarm, open_app, open_settings, search_web';

    final buffer = StringBuffer(' You control this Android device via tools: $listed. ');
    
    final actions = <String>[];
    if (!hasTools || names.contains('open_app')) actions.add('open an app');
    if (!hasTools || names.contains('set_alarm') || names.contains('cancel_alarm')) actions.add('set/cancel an alarm');
    if (!hasTools || names.contains('open_settings')) actions.add('open settings');
    if (!hasTools || names.contains('search_web')) actions.add('search the web');

    if (actions.isNotEmpty) {
      final actionStr = actions.join(', ');
      buffer.write('When the user asks to $actionStr, call the tool immediately — do not say you lack device access. ');
    }

    if (!hasTools || names.contains('open_app')) {
      buffer.write('open_app packages: com.google.android.youtube, com.android.settings, com.android.chrome. ');
    }

    buffer.write('If native function calling is unavailable, reply with only JSON like '
        '\'{"name":"open_app","arguments":{"package":"com.google.android.youtube"}}\'.');

    return buffer.toString();
  }


static AgentIdentity? _cachedIdentity;

static AgentIdentity? _getCachedIdentity() => _cachedIdentity;
Expand Down
10 changes: 7 additions & 3 deletions lib/tools/tool_definitions.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,14 +88,18 @@ class NovaTools {
name: 'open_app',
description:
'Open an application on the device. '
'Use this ONLY when the user explicitly asks to open '
'or launch a specific application.',
'Use when the user asks to open/launch an app (English or German: '
'open, launch, öffne, starte). '
'Common packages: YouTube=com.google.android.youtube, '
'Settings=com.android.settings, Chrome=com.android.chrome, '
'Spotify=com.spotify.music, Maps=com.google.android.apps.maps.',
parameters: {
'type': 'object',
'properties': {
'package': {
'type': 'string',
'description': 'App package name (e.g. com.twitter.android)',
'description':
'Android package name, e.g. com.google.android.youtube',
},
},
'required': ['package'],
Expand Down
27 changes: 23 additions & 4 deletions lib/utils/alarm_time_parser.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,18 @@
/// Parses natural-language alarm times from user queries.
///
/// Examples: "set an alarm for 7 AM", "alarm at 7pm", "wake me at 19:30".
/// Examples: "set an alarm for 7 AM", "alarm at 7pm", "wake me at 19:30",
/// "stell einen Wecker um 19:00", "Wecker auf 7 Uhr".
class AlarmTimeParser {
AlarmTimeParser._();

static final RegExp _alarmIntent = RegExp(
r'\b(set\s+(an?\s+)?alarm|alarm\s+(for|at)|wake\s+me)\b',
r'\b('
r'set\s+(an?\s+)?alarm|alarm\s+(for|at)|wake\s+me|'
r'wecker|alarm\s+stellen|'
r'stell(e|en)?\s+(einen?\s+)?(wecker|alarm)|'
r'weck\s+mich|'
r'(wecker|alarm)\s+(auf|um|für|fuer)'
r')\b',
caseSensitive: false,
);

Expand All @@ -16,6 +23,11 @@ class AlarmTimeParser {

static final RegExp _time24h = RegExp(r'\b([01]?\d|2[0-3]):([0-5]\d)\b');

static final RegExp _timeGermanUhr = RegExp(
r'\b(?:um\s+|auf\s+|für\s+|fuer\s+)?(\d{1,2})(?:[:.](\d{2}))?\s*uhr\b',
caseSensitive: false,
);

/// Returns `(hour, minute)` in 24-hour format, or null if not an alarm request
/// with a parseable time.
static ({int hour, int minute})? tryParse(String query) {
Expand Down Expand Up @@ -47,8 +59,15 @@ class AlarmTimeParser {
return (hour: hour, minute: minute);
}

// Bare hour with AM/PM already handled; try "at 7" / "for 7" without period
// only when clearly morning/evening context is absent — skip ambiguous cases.
final matchUhr = _timeGermanUhr.firstMatch(query);
if (matchUhr != null) {
final hour = int.parse(matchUhr.group(1)!);
final minute = int.parse(matchUhr.group(2) ?? '0');
if (hour > 23 || minute > 59) return null;

return (hour: hour, minute: minute);
}

return null;
}
}
72 changes: 72 additions & 0 deletions lib/utils/open_app_intent_parser.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
/// Resolves natural-language "open app" requests to Android package names.
class OpenAppIntentParser {
OpenAppIntentParser._();

/// ASCII open verbs (safe with `\b`). Umlaut forms use normalized matching.
static final RegExp _asciiOpenIntent = RegExp(
r'\b(open|launch|start|starte|oeffne|oeffnen|aufmachen)\b',
caseSensitive: false,

@sourcery-ai sourcery-ai Bot Jul 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: The mach ... auf pattern misses some natural German phrasings.

This regex won’t match phrases with extra words between mach and auf, e.g. "mach mal YouTube auf" or "mach bitte die Kamera auf". To support these, consider allowing a limited span between them, e.g. \bmach(?:en)?\b[\s\w]{0,20}\bauf\b, or a similar bounded pattern to avoid over‑matching.

);

static final RegExp _machAufIntent = RegExp(
r'\bmach(?:en)?\s+\S+\s+auf\b',
caseSensitive: false,
);

/// Common app aliases → package names.
static const Map<String, String> knownApps = {
'youtube': 'com.google.android.youtube',
'yt': 'com.google.android.youtube',
'settings': 'com.android.settings',
'einstellungen': 'com.android.settings',
'chrome': 'com.android.chrome',
'maps': 'com.google.android.apps.maps',
'google maps': 'com.google.android.apps.maps',
'kamera': 'com.android.camera',
'camera': 'com.android.camera',
Comment on lines +17 to +26

@sourcery-ai sourcery-ai Bot Jul 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: Review alias list and matching strategy for coverage and accidental matches.

Two small follow‑ups:

  • Confirm whether 'galeria' is intentional; if so, consider also adding 'galerie'.
  • lower.contains(alias) may match aliases as substrings inside longer words. For some entries (e.g. WhatsApp), consider using word‑boundary‑style matching (like \bwhatsapp\b) to reduce accidental matches without much added complexity.

Suggested implementation:

  /// Common app aliases → package names.
  ///
  /// Note:
  /// - `"galeria"` is intentionally kept as a Spanish/Portuguese variant.
  /// - `"galerie"` is added as the German variant.
  static const Map<String, String> knownApps = {
    'youtube': 'com.google.android.youtube',
    'yt': 'com.google.android.youtube',
    'settings': 'com.android.settings',
    'einstellungen': 'com.android.settings',
    'chrome': 'com.android.chrome',
    'maps': 'com.google.android.apps.maps',
    'google maps': 'com.google.android.apps.maps',
    'kamera': 'com.android.camera',
    'camera': 'com.android.camera',
    'photos': 'com.google.android.apps.photos',
    'galeria': 'com.google.android.apps.photos',
    'galerie': 'com.google.android.apps.photos',
    'gallery': 'com.google.android.apps.photos',
    'whatsapp': 'com.whatsapp',
    'spotify': 'com.spotify.music',
    'clock': 'com.android.deskclock',
    'uhr': 'com.android.deskclock',
    'wecker': 'com.android.deskclock',
  };

I can’t see the call site that uses knownApps, but your comment mentions a lower.contains(alias) check. To implement “word‑boundary‑style” matching (especially for entries like "whatsapp"), you should:

  1. Locate the loop where knownApps is used, something like: 
    for (final entry in knownApps.entries) {
  if (lower.contains(entry.key)) {
    return entry.value;
  }
}
  2. Replace the contains check with a small helper that uses a RegExp with word boundaries for alphabetic aliases, falling back to contains when needed. For example: 
    bool _matchesAlias(String lowerUtterance, String alias) {
  // Use word boundaries for simple alphabetic aliases (e.g. "whatsapp", "spotify").
  final isAlphabetic = RegExp(r'^[a-z]+(?:\s+[a-z]+)*$').hasMatch(alias);
  if (isAlphabetic) {
    final pattern = RegExp('\\b${RegExp.escape(alias)}\\b', caseSensitive: false);
    return pattern.hasMatch(lowerUtterance);
  }
  // Fallback: plain substring match for aliases with non‑letters / special cases.
  return lowerUtterance.contains(alias.toLowerCase());
}
  3. Then update the loop to use the helper: 
    for (final entry in knownApps.entries) {
  if (_matchesAlias(lower, entry.key)) {
    return entry.value;
  }
}

This preserves the current behavior for most aliases while reducing accidental matches for names like "whatsapp" and "spotify". You may want to place _matchesAlias near the other regex helpers in open_app_intent_parser.dart.

'photos': 'com.google.android.apps.photos',
'galeria': 'com.google.android.apps.photos',
'gallery': 'com.google.android.apps.photos',
Comment on lines +28 to +29

@gemini-code-assist gemini-code-assist Bot Jul 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The German word for gallery is "Galerie", but it is currently missing from knownApps (which only has "galeria" and "gallery"). Adding "galerie" ensures German users can open the photos app using their native term.

Suggested change
'galeria': 'com.google.android.apps.photos',
'gallery': 'com.google.android.apps.photos',
'galeria': 'com.google.android.apps.photos',
'galerie': 'com.google.android.apps.photos',
'gallery': 'com.google.android.apps.photos',

'whatsapp': 'com.whatsapp',
'spotify': 'com.spotify.music',
'clock': 'com.android.deskclock',
'uhr': 'com.android.deskclock',
'wecker': 'com.android.deskclock',
};

/// Normalize German umlauts so ASCII `\b` works ("öffne" → "oeffne").
/// Also prevents "geöffnet" → "geoeffnet" from matching `\boeffne\b`.
static String _normalize(String query) {
return query
.toLowerCase()
.replaceAll('ö', 'oe')
.replaceAll('ä', 'ae')
.replaceAll('ü', 'ue')
.replaceAll('ß', 'ss');
}

static bool _hasOpenIntent(String query) {
final normalized = _normalize(query);
if (_asciiOpenIntent.hasMatch(normalized)) return true;
if (_machAufIntent.hasMatch(normalized)) return true;

return false;
}

/// Returns a package name when [query] clearly asks to open a known app.
static String? tryParsePackage(String query) {
if (!_hasOpenIntent(query)) return null;
Comment on lines +57 to +58

@gemini-code-assist gemini-code-assist Bot Jul 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The deterministic open-app shortcut currently matches queries even if they contain negations (e.g., "Don't open YouTube" or "öffne YouTube nicht"). This results in the app being opened when the user explicitly requested the opposite. We should add a negation check to return null immediately if negation keywords are detected.

Suggested change
static String? tryParsePackage(String query) {
if (!_hasOpenIntent(query)) return null;
static String? tryParsePackage(String query) {
final negationPattern = RegExp(
r"\b(don\s*'?\s*t|not|no|nicht|kein|keine|nie|niemals|verhindern|stoppen|stop)\b",
caseSensitive: false,
);
if (negationPattern.hasMatch(query)) return null;
if (!_hasOpenIntent(query)) return null;


final lower = query.toLowerCase();
// Prefer longer aliases first (e.g. "google maps" before "maps").
final aliases = knownApps.keys.toList()
..sort((a, b) => b.length.compareTo(a.length));
for (final alias in aliases) {
if (lower.contains(alias)) {
return knownApps[alias];
}
}
Comment on lines +64 to +68

@gemini-code-assist gemini-code-assist Bot Jul 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Using lower.contains(alias) performs a substring match rather than a whole-word match. This leads to critical false positives where common words containing the alias as a substring trigger the app opening (e.g., "Python" matching "yt" and opening YouTube, or "Ruhr" matching "uhr" and opening the Clock app). We should use a regular expression with word boundaries (\b) to ensure we only match whole words.

Suggested change
for (final alias in aliases) {
if (lower.contains(alias)) {
return knownApps[alias];
}
}
for (final alias in aliases) {
final pattern = RegExp(r'\b' + RegExp.escape(alias) + r'\b');
if (pattern.hasMatch(lower)) {
return knownApps[alias];
}
}


return null;
}
}
Loading
Loading