Skip to content

feat: new voices and wakewords#88

Open
mikejgray wants to merge 21 commits into
NeonGeckoCom:devfrom
mikejgray:FEAT_NewVoice
Open

feat: new voices and wakewords#88
mikejgray wants to merge 21 commits into
NeonGeckoCom:devfrom
mikejgray:FEAT_NewVoice

Conversation

@mikejgray

@mikejgray mikejgray commented May 17, 2024

Copy link
Copy Markdown

Description

Allows users to use one of three presets for voice/wakeword combinations:

  1. Neon ("Hey Neon", custom Neon Coqui voice model)
  2. Classic Mycroft ("Hey Mycroft", Piper Alan Pope voice model)
  3. The Butler ("Hey Jarvis", Piper Alan Pope voice model)

Issues

#87

Other Notes

N/A

Comment thread locale/en-us/dialog/ask_enable.dialog Outdated
Comment thread locale/en-us/dialog/neon_confirmation.dialog Outdated
Comment thread locale/en-us/intent/classic_mycroft.intent Outdated
Comment thread __init__.py
Comment thread __init__.py Outdated
Comment thread __init__.py Outdated
Comment thread __init__.py

def initialize(self):
self.bus.on("mycroft.ready", self._speak_restart_dialog)
if self.dialog_to_speak and self.pending_audio_restart:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this happens before the audio module is ready, I believe the dialog may be lost.. The on("mycroft.ready") should handle it in any case so this appears to be redundant anyways

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See above, audio should always be ready before skills load in a Mark 2, so this is the primary and mycroft.ready is the fallback. mycroft.ready hasn't always been consistent for me so that's why I made this primary

Comment thread __init__.py Outdated
@NeonDaniel

Copy link
Copy Markdown
Member

Regarding the test failure, can you log wake_word_config when this assertion fails? I suspect something with the test listeners is causing the config to not update properly in this test case, though I don't see an obvious cause in your changes

@NeonDaniel

NeonDaniel commented Jul 2, 2024

Copy link
Copy Markdown
Member

I got a "sorry, something has gone wrong. I will continue using-" followed by services restarting.

audio.log

2024-07-02 15:42:10.046 - audio - neon_audio.tts.neon:execute:373 - DEBUG - responses={'en-us': {'sentence': "Okay, I'll be back shortly.", 'translated': False, 'phonemes': None, 'genders': ['female'], 'female': '/home/neon/.cache/neon/tts/CoquiRemoteTTS/en-us/female/8f18e5d69575daf28d21830251e1a820.wav'}}
2024-07-02 15:42:10.249 - audio - ovos_bus_client.session:get:605 - DEBUG - No message, use default session
2024-07-02 15:42:12.954 - audio - neon_audio.tts.neon:_play:185 - INFO - Played 1719960128.2668295
2024-07-02 15:42:12.957 - audio - neon_audio.tts.neon:_sort_timing_metrics:149 - DEBUG - Parsed timing context: {'timestamps': {'handle_utterance': 1719960107.7861273, 'speech_start': 1719960128.266869, 'audio_begin': 1719960130.2339814, 'audio_end': 1719960132.8814085}, 'durations': {'transform_audio': 3.1948089599609375e-05, 'get_stt': 5.132228374481201, 'save_transcript': 1.049041748046875e-05, 'text_parsers': 0.01896810531616211, 'transform_utterance': 0.01896810531616211, 'get_tts': 1.5334551334381104}}
2024-07-02 15:42:12.986 - audio - neon_audio.service:handle_speak:148 - DEBUG - Playback completed for: 1719960128.2668295
2024-07-02 15:42:38.631 - audio - ovos_audio.service:execute_tts:381 - INFO - Speak: Sorry, something went wrong. I will continue using the old wake word.
2024-07-02 15:42:38.726 - audio - neon_audio.tts.neon:get_requested_tts_languages:79 - DEBUG - Got profiles: [{'user': {'first_name': '', 'middle_name': '', 'last_name': '', 'preferred_name': '', 'full_name': '', 'dob': 'YYYY/MM/DD', 'age': '', 'email': 'daniel@neon.ai', 'username': 'local', 'password': '', 'picture': '', 'about': '', 'phone': '', 'phone_verified': False, 'email_verified': False}, 'brands': {'ignored_brands': {}, 'favorite_brands': {}, 'specially_requested': {}}, 'speech': {'stt_language': 'en-us', 'alt_languages': ['en-us', 'uk-ua'], 'tts_language': 'en-us', 'tts_gender': 'female', 'neon_voice': '', 'secondary_tts_language': '', 'secondary_tts_gender': 'male', 'secondary_neon_voice': '', 'speed_multiplier': 1.0}, 'units': {'time': 12, 'date': 'MDY', 'measure': 'imperial'}, 'location': {'lat': 47.6765382, 'lng': -122.2070775, 'city': 'Kirkland', 'state': 'Washington', 'country': 'United States', 'tz': 'America/Los_Angeles', 'utc': -8.0, 'country_code': 'us'}, 'response_mode': {'speed_mode': 'quick', 'hesitation': False, 'limit_dialog': False}, 'privacy': {'save_audio': True, 'save_text': True}}]
2024-07-02 15:42:38.729 - audio - neon_audio.tts.neon:get_requested_tts_languages:87 - DEBUG - local requesting female en-us
2024-07-02 15:42:38.731 - audio - neon_audio.tts.neon:get_requested_tts_languages:132 - DEBUG - Got 1 TTS Voice Requests
2024-07-02 15:42:38.734 - audio - neon_audio.tts.neon:get_multiple_tts:300 - DEBUG - tts_requested=[{'speaker': 'Neon', 'language': 'en-us', 'gender': 'female', 'voice': None}]
2024-07-02 15:42:38.739 - audio - neon_audio.tts.neon:get_multiple_tts:304 - DEBUG - utterance_lang=en-us
2024-07-02 15:42:38.799 - audio - ovos_config.models:load_local:112 - DEBUG - Configuration /home/neon/.config/neon/neon.yaml loaded
2024-07-02 15:42:38.801 - audio - ovos_config.config:_on_file_change:315 - INFO - /home/neon/.config/neon/neon.yaml changed on disk
2024-07-02 15:42:38.804 - audio - neon_audio.tts.neon:_get_tts:268 - INFO - Legacy Neon TTS signature found (CoquiRemoteTTS)
2024-07-02 15:42:38.825 - audio - ovos_config.config:_on_file_change:316 - DEBUG - Calling 1 callbacks
2024-07-02 15:42:38.834 - audio - ovos_audio.service:_maybe_reload_tts:360 - DEBUG - Skipping fallback TTS init
2024-07-02 15:42:39.037 - audio - ovos_audio.service:execute_tts:381 - INFO - Speak: Wake word hey neon is still enabled. Check the logs.
2024-07-02 15:42:39.352 - audio - ovos_config.models:load_local:112 - DEBUG - Configuration /home/neon/.config/neon/neon.yaml loaded
2024-07-02 15:42:39.412 - audio - ovos_config.config:_on_file_change:315 - INFO - /home/neon/.config/neon/neon.yaml changed on disk
2024-07-02 15:42:39.419 - audio - ovos_config.config:_on_file_change:316 - DEBUG - Calling 1 callbacks
2024-07-02 15:42:39.504 - audio - neon_audio.tts.neon:execute:373 - DEBUG - responses={'en-us': {'sentence': 'Sorry, something went wrong. I will continue using the old wake word.', 'translated': False, 'phonemes': None, 'genders': ['female'], 'female': '/home/neon/.cache/neon/tts/CoquiRemoteTTS/en-us/female/ea369b8d5507d03365d8a4c806a801bb.wav'}}
2024-07-02 15:42:39.617 - audio - neon_audio.tts.neon:get_requested_tts_languages:79 - DEBUG - Got profiles: [{'user': {'first_name': '', 'middle_name': '', 'last_name': '', 'preferred_name': '', 'full_name': '', 'dob': 'YYYY/MM/DD', 'age': '', 'email': 'daniel@neon.ai', 'username': 'local', 'password': '', 'picture': '', 'about': '', 'phone': '', 'phone_verified': False, 'email_verified': False}, 'brands': {'ignored_brands': {}, 'favorite_brands': {}, 'specially_requested': {}}, 'speech': {'stt_language': 'en-us', 'alt_languages': ['en-us', 'uk-ua'], 'tts_language': 'en-us', 'tts_gender': 'female', 'neon_voice': '', 'secondary_tts_language': '', 'secondary_tts_gender': 'male', 'secondary_neon_voice': '', 'speed_multiplier': 1.0}, 'units': {'time': 12, 'date': 'MDY', 'measure': 'imperial'}, 'location': {'lat': 47.6765382, 'lng': -122.2070775, 'city': 'Kirkland', 'state': 'Washington', 'country': 'United States', 'tz': 'America/Los_Angeles', 'utc': -8.0, 'country_code': 'us'}, 'response_mode': {'speed_mode': 'quick', 'hesitation': False, 'limit_dialog': False}, 'privacy': {'save_audio': True, 'save_text': True}}]
2024-07-02 15:42:39.621 - audio - neon_audio.tts.neon:get_requested_tts_languages:87 - DEBUG - local requesting female en-us
2024-07-02 15:42:39.628 - audio - neon_audio.tts.neon:get_requested_tts_languages:132 - DEBUG - Got 1 TTS Voice Requests
2024-07-02 15:42:39.658 - audio - neon_audio.tts.neon:get_multiple_tts:300 - DEBUG - tts_requested=[{'speaker': 'Neon', 'language': 'en-us', 'gender': 'female', 'voice': None}]
2024-07-02 15:42:39.662 - audio - neon_audio.tts.neon:get_multiple_tts:304 - DEBUG - utterance_lang=en-us
2024-07-02 15:42:39.668 - audio - neon_audio.tts.neon:_get_tts:268 - INFO - Legacy Neon TTS signature found (CoquiRemoteTTS)
2024-07-02 15:42:39.737 - audio - ovos_bus_client.session:get:605 - DEBUG - No message, use default session
2024-07-02 15:42:41.984 - audio - neon_audio.tts.neon:execute:373 - DEBUG - responses={'en-us': {'sentence': 'Wake word hey neon is still enabled. Check the logs.', 'translated': False, 'phonemes': None, 'genders': ['female'], 'female': '/home/neon/.cache/neon/tts/CoquiRemoteTTS/en-us/female/7f492da5f1189046714154697d04426e.wav'}}
2024-07-02 15:42:41.988 - audio - ovos_audio.service:_maybe_reload_tts:349 - INFO - (re)loading TTS engine
2024-07-02 15:42:41.991 - audio - ovos_plugin_manager.utils.config:get_plugin_config:40 - DEBUG - Loaded configuration: {'lang': 'en-us', 'voice': 'alan-low', 'module': 'ovos-tts-plugin-piper', 'pulse_duck': False, 'preload_fallback': False, 'fallback_module': 'coqui'}
2024-07-02 15:42:42.150 - audio - ovos_plugin_manager.utils:load_plugin:161 - WARNING - Could not find the plugin mycroft.plugin.tts.ovos-tts-plugin-piper
2024-07-02 15:42:42.155 - audio - neon_audio.tts:create:59 - ERROR - Could not find plugin: ovos-tts-plugin-piper
2024-07-02 15:42:42.159 - audio - ovos_config.config:_on_file_change:321 - ERROR - Error in config update callback handler
Traceback (most recent call last):
  File "/home/neon/venv/lib/python3.10/site-packages/ovos_config/config.py", line 319, in _on_file_change
    handler()
  File "/home/neon/venv/lib/python3.10/site-packages/ovos_audio/service.py", line 351, in _maybe_reload_tts
    self.tts.init(self.bus, self.playback_thread)
AttributeError: 'NoneType' object has no attribute 'init'
2024-07-02 15:42:43.525 - audio - neon_audio.tts.neon:_play:185 - INFO - Played 1719960158.5661793
2024-07-02 15:42:43.528 - audio - neon_audio.tts.neon:_sort_timing_metrics:149 - DEBUG - Parsed timing context: {'timestamps': {'handle_utterance': 1719960107.7861273, 'speech_start': 1719960158.5661988, 'audio_begin': 1719960159.7325459, 'audio_end': 1719960163.4334536}, 'durations': {'transform_audio': 3.1948089599609375e-05, 'get_stt': 5.132228374481201, 'save_transcript': 1.049041748046875e-05, 'text_parsers': 0.01896810531616211, 'transform_utterance': 0.01896810531616211, 'get_tts': 0.7698671817779541}}
2024-07-02 15:42:43.566 - audio - neon_audio.service:handle_speak:148 - DEBUG - Playback completed for: 1719960158.5661793
2024-07-02 15:42:49.745 - audio - ovos_utils:wait_for_exit_signal:183 - DEBUG - Exiting on KeyboardInterrupt
2024-07-02 15:42:50.137 - audio - ovos_config.config:_on_file_change:312 - DEBUG - Ignoring non-config file change: /home/neon/.config/neon/schedule.json
2024-07-02 15:42:51.317 - audio - ovos_bus_client.client.client:on_error:122 - ERROR - === RuntimeError('cannot schedule new futures after shutdown') ===
Traceback (most recent call last):
  File "/home/neon/venv/lib/python3.10/site-packages/websocket/_app.py", line 672, in _callback
    callback(self, *args)
  File "/home/neon/venv/lib/python3.10/site-packages/ovos_bus_client/client/client.py", line 160, in on_message
    self.emitter.emit(parsed_message.msg_type, parsed_message)
  File "/home/neon/venv/lib/python3.10/site-packages/pyee/_base.py", line 113, in emit
    handled = self._call_handlers(event, args, kwargs)
  File "/home/neon/venv/lib/python3.10/site-packages/pyee/_base.py", line 96, in _call_handlers
    self._emit_run(f, args, kwargs)
  File "/home/neon/venv/lib/python3.10/site-packages/pyee/_executor.py", line 50, in _emit_run
    future = self._executor.submit(f, *args, **kwargs)
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 167, in submit
    raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2024-07-02 15:42:51.375 - audio - ovos_bus_client.client.client:on_error:126 - ERROR - Failed to emit error event: cannot schedule new futures after shutdown
Traceback (most recent call last):
  File "/home/neon/venv/lib/python3.10/site-packages/ovos_bus_client/client/client.py", line 124, in on_error
    self.emitter.emit('error', error)
  File "/home/neon/venv/lib/python3.10/site-packages/pyee/_base.py", line 116, in emit
    self._emit_handle_potential_error(event, args[0] if args else None)
  File "/home/neon/venv/lib/python3.10/site-packages/pyee/_base.py", line 86, in _emit_handle_potential_error
    raise error
  File "/home/neon/venv/lib/python3.10/site-packages/websocket/_app.py", line 672, in _callback
    callback(self, *args)
  File "/home/neon/venv/lib/python3.10/site-packages/ovos_bus_client/client/client.py", line 160, in on_message
    self.emitter.emit(parsed_message.msg_type, parsed_message)
  File "/home/neon/venv/lib/python3.10/site-packages/pyee/_base.py", line 113, in emit
    handled = self._call_handlers(event, args, kwargs)
  File "/home/neon/venv/lib/python3.10/site-packages/pyee/_base.py", line 96, in _call_handlers
    self._emit_run(f, args, kwargs)
  File "/home/neon/venv/lib/python3.10/site-packages/pyee/_executor.py", line 50, in _emit_run
    future = self._executor.submit(f, *args, **kwargs)
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 167, in submit
    raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2024-07-02 15:42:51.835 - audio - ovos_bus_client.client.client:on_error:134 - WARNING - Message Bus Client will reconnect in 5.0 seconds.
2024-07-02 15:42:56.850 - audio - ovos_bus_client.client.client:on_open:93 - DEBUG - Connected

skills.log

2024-07-02 15:42:08.292 - skills - neon_utils.skills.neon_skill:speak:646 - DEBUG - Skill speak! {'utterance': "Okay, I'll be back shortly.", 'lang': 'en-us', 'expect_response': False, 'meta': {'dialog': 'confirm_change', 'data': {}, 'skill': 'DeviceControlCenterSkill'}, 'speaker': None, 'speak_ident': '1719960128.2668295'}
2024-07-02 15:42:08.295 - skills - neon_utils.skills.neon_skill:speak:647 - DEBUG - speak
2024-07-02 15:42:08.303 - skills - skill_device_controls:_set_user_tts_settings:565 - DEBUG - Patching user ngi config for persona jarvis
2024-07-02 15:42:08.308 - skills - skill_device_controls:_retrieve_ngi_config:583 - DEBUG - Patching user ngi config for Neon TTS
2024-07-02 15:42:08.339 - skills - ovos_bus_client.client.client:on_default_session_update:166 - DEBUG - synced default_session
2024-07-02 15:42:08.538 - skills - neon_utils.configuration_utils:_write_yaml_file:344 - DEBUG - YAML updated ngi_user_info
2024-07-02 15:42:08.540 - skill-device_controls.neongeckocom - DEBUG - Attempting to enable WW: hey_jarvis
2024-07-02 15:42:10.389 - skills - ovos_bus_client.util.scheduler:check_state:186 - DEBUG - Call scheduled event: skill-ovos-homescreen.openvoiceos:skill-ovos-homescreen.openvoiceosupdate_dt
2024-07-02 15:42:20.408 - skills - ovos_bus_client.util.scheduler:check_state:186 - DEBUG - Call scheduled event: skill-ovos-homescreen.openvoiceos:skill-ovos-homescreen.openvoiceosupdate_dt
2024-07-02 15:42:30.429 - skills - ovos_bus_client.util.scheduler:check_state:186 - DEBUG - Call scheduled event: skill-ovos-homescreen.openvoiceos:skill-ovos-homescreen.openvoiceosupdate_dt
2024-07-02 15:42:38.548 - skills - skill_device_controls:_emit_enable_ww_message:594 - ERROR - No response to WW enable request for hey_jarvis!
2024-07-02 15:42:38.551 - skills - skill_device_controls:_enable_wake_word:435 - ERROR - No response to WW enable request for hey_jarvis!
2024-07-02 15:42:38.555 - skills - neon_utils.skills.neon_skill:speak_dialog:680 - DEBUG - data={}
2024-07-02 15:42:38.568 - skills - neon_utils.skills.neon_skill:speak:646 - DEBUG - Skill speak! {'utterance': 'Sorry, something went wrong. I will continue using the old wake word.', 'lang': 'en-us', 'expect_response': False, 'meta': {'dialog': 'error_ww_change_failed', 'data': {}, 'skill': 'DeviceControlCenterSkill'}, 'speaker': None, 'speak_ident': '1719960158.5661793'}
2024-07-02 15:42:38.571 - skills - neon_utils.skills.neon_skill:speak:647 - DEBUG - speak
2024-07-02 15:42:38.752 - skill-device_controls.neongeckocom - DEBUG - Found available WW: {'hey_mycroft': {'active': False, 'module': 'ovos-ww-plugin-precise-lite', 'listen': True, 'sound': 'snd/start_listening.wav', 'version': '0.3', 'model': 'https://github.com/OpenVoiceOS/precise-lite-models/raw/master/wakewords/en/hey_mycroft.tflite', 'expected_duration': 3, 'trigger_level': 3, 'sensitivity': 0.5}, 'hey_neon': {'module': 'ovos-ww-plugin-vosk', 'listen': True, 'sound': 'snd/start_listening.wav', 'debug': False, 'rule': 'fuzzy', 'lang': 'en', 'samples': ['hey neon', '[unk]'], 'model_folder': '/home/neon/.local/share/neon/vosk-model-small-en-us-0.15', 'active': True}, 'hey_jarvis': {'module': 'ovos-ww-plugin-openwakeword', 'active': True, 'listen': True}}
2024-07-02 15:42:38.754 - skill-device_controls.neongeckocom - DEBUG - Found enabled WWs: ['hey_neon', 'hey_jarvis']
2024-07-02 15:42:38.755 - skill-device_controls.neongeckocom - DEBUG - Disabling WW: hey_neon
2024-07-02 15:42:38.927 - skills - skill_device_controls:_disable_wake_word:466 - WARNING - WW disable failed with response: {'error': 'only one active ww', 'active': True, 'wake_word': 'hey_neon'}
2024-07-02 15:42:38.933 - skills - neon_utils.skills.neon_skill:speak_dialog:680 - DEBUG - data={'ww': 'hey neon'}

It seems there's something going on with disabling the old WW before enabling the new one which causes an error to avoid a state where there is no active WW.

UPDATE:
After restarting, it looks like the Audio module is failing to load due to a missing plugin:

2024-07-02 15:46:18.874 - audio - neon_audio.tts:create:59 - ERROR - Could not find plugin: ovos-tts-plugin-piper
2024-07-02 15:46:18.880 - audio - ovos_audio.service:__init__:95 - ERROR - 'NoneType' object has no attribute 'init'
AttributeError: 'NoneType' object has no attribute 'init'
2024-07-02 15:47:41.265 - audio - ovos_utils.process_utils:exists:353 - ERROR - Failed to kill PID <_io.TextIOWrapper name='/tmp/neon/audio.pid' mode='r' encoding='UTF-8'>: [Errno 3] No such process
2024-07-02 15:47:57.173 - audio - neon_audio.tts:create:59 - ERROR - Could not find plugin: ovos-tts-plugin-piper
2024-07-02 15:47:57.176 - audio - ovos_audio.service:__init__:95 - ERROR - 'NoneType' object has no attribute 'init'
AttributeError: 'NoneType' object has no attribute 'init'

I'll install that and try again

mikejgray added a commit to mikejgray/NeonCore that referenced this pull request Jul 11, 2024
Allows NeonGeckoCom/skill-device_controls#88 to succeed, among other things
@mikejgray

Copy link
Copy Markdown
Author

@JarbasAl JarbasAl removed their request for review January 25, 2025 04:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants