Skip to content

new device API#274

Merged
marmarek merged 43 commits into
QubesOS:mainfrom
piotrbartman:prb/new_device_api
Jun 15, 2024
Merged

new device API#274
marmarek merged 43 commits into
QubesOS:mainfrom
piotrbartman:prb/new_device_api

Conversation

@piotrbartman

@piotrbartman piotrbartman commented Feb 6, 2024

Copy link
Copy Markdown
Member

after QubesOS/qubes-core-admin/pull/579

Proposed device events:
sometimes event can be fired twice, we can ignore it
e.g. if device is plugged out and in in short time we can have doubled
detaching and attaching...

  • device-list-changed: device-added -> device-added{devclass}

  • device-list-changed: device-remove -> device-removed{devclass}

  • device-property-changed: property_name: TODO?

  • device-assignment-changed: created -> device-assign:{devclass}

  • device-assignment-changed: removed -> device-unassign:{devclass}

  • device-assignment-changed: attached -> device-attach:{devclass}

  • device-assignment-changed: detached -> device-detach:{devclass}

  • device-assignment-changed: property-set -> device-assignment-changed:{devclass}

QubesOS/qubes-issues#4626

@piotrbartman piotrbartman marked this pull request as draft February 6, 2024 17:36
Comment thread qubesadmin/devices.py Outdated
Comment thread qubesadmin/tools/qvm_device.py Outdated
assignment.options = options
vm.devices[args.devclass].assign(assignment)
if vm.is_running() and not assignment.attached:
print("Assigned. Now you can manually attach device or restart domain.")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This (and other below) should respect -q (same below).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment thread qubesadmin/tools/qvm_device.py Outdated
vm.devices[args.devclass].get_assigned_devices()):
vm.devices[args.devclass].unassign(device_assignment)
if device_assignment.attached:
print("Unassigned. Now you can manually detach device "

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This (besides -q) should also print which device.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment on lines -306 to -311
# Special treatment for '--list-device-classes' (alias --list-classes)
curr_action = sys.argv[1:]
if set(curr_action).intersection(
{'--list-device-classes', '--list-classes'}):
sys.exit(main(args=['', '--list-device-classes']))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happened to this one (besides it being already slightly broken due to requiring a device class anyway)?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I slightly fix it and now device class is not required :D (see lines 408-411).

@qubesos-bot

qubesos-bot commented Mar 29, 2024

Copy link
Copy Markdown

OpenQA test summary

Complete test suite and dependencies: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2024061301-4.3&flavor=pull-requests

Test run included the following:

New failures, excluding unstable

Compared to: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2024052808-4.3&flavor=update

  • system_tests_extra

    • TC_00_QVCTest_whonix-gateway-17: test_010_screenshare (failure)
      self.assertNotEqual(vm.run('test -e /dev/vid... AssertionError: 0 == 0
  • system_tests_network_updates

    • TC_10_QvmTemplate_fedora-40-xfce: test_000_template_list (failure)
      AssertionError: libvirt event impl drain timeout
  • system_tests_basic_vm_qrexec_gui_zfs

    • TC_06_AppVM_whonix-gateway-17-pool: test_010_os_metadata (failure)
      self.assertEqual(tpl.features.get(... AssertionError: None != 'debian'

    • TC_06_AppVM_whonix-workstation-17-pool: test_010_os_metadata (failure)
      self.assertEqual(tpl.features.get(... AssertionError: None != 'debian'

  • system_tests_guivm_vnc_gui_interactive

    • guivm_startup: Failed (test died)
      # Test died: command 'qvm-run --nogui -pu root sys-gui-vnc env XAUT...
  • system_tests_basic_vm_qrexec_gui_btrfs

    • TC_06_AppVM_whonix-gateway-17-pool: test_010_os_metadata (failure)
      self.assertEqual(tpl.features.get(... AssertionError: None != 'debian'

    • TC_06_AppVM_whonix-workstation-17-pool: test_010_os_metadata (failure)
      self.assertEqual(tpl.features.get(... AssertionError: None != 'debian'

  • system_tests_basic_vm_qrexec_gui_ext4

    • TC_06_AppVM_whonix-gateway-17-pool: test_010_os_metadata (failure)
      self.assertEqual(tpl.features.get(... AssertionError: None != 'debian'

    • TC_06_AppVM_whonix-workstation-17-pool: test_010_os_metadata (failure)
      self.assertEqual(tpl.features.get(... AssertionError: None != 'debian'

  • system_tests_basic_vm_qrexec_gui_xfs

    • TC_06_AppVM_whonix-gateway-17-pool: test_010_os_metadata (failure)
      self.assertEqual(tpl.features.get(... AssertionError: None != 'debian'

    • TC_06_AppVM_whonix-workstation-17-pool: test_010_os_metadata (failure)
      self.assertEqual(tpl.features.get(... AssertionError: None != 'debian'

  • system_tests_basic_vm_qrexec_gui

    • TC_06_AppVM_whonix-gateway-17: test_010_os_metadata (failure)
      self.assertEqual(tpl.features.get(... AssertionError: None != 'debian'

    • TC_06_AppVM_whonix-workstation-17: test_010_os_metadata (failure)
      self.assertEqual(tpl.features.get(... AssertionError: None != 'debian'

    • TC_20_AudioVM_PipeWire_fedora-40-xfce: test_226_audio_playback_pipewire (failure)
      AssertionError: Command 'timeout 20s pw-play --format=f32 --rate=44...

  • system_tests_kde_gui_interactive

    • gui_filecopy: unnamed test (unknown)
    • gui_filecopy: Failed (test died)
      # Test died: no candidate needle with tag(s) 'files-test-file' matc...

Failed tests

23 failures
  • system_tests_pvgrub_salt_storage

    • TC_41_HVMGrub_fedora-40-xfce: test_000_standalone_vm (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

    • TC_41_HVMGrub_fedora-40-xfce: test_010_template_based_vm (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

  • system_tests_extra

    • [unstable] TC_00_QVCTest_debian-12-xfce: test_020_webcam (failure + cleanup)
      AssertionError: 'qubes-video-companion webcam' exited early (0): b'...

    • [unstable] TC_00_QVCTest_fedora-40-xfce: test_020_webcam (failure + cleanup)
      AssertionError: 'qubes-video-companion webcam' exited early (0): b'...

    • TC_00_QVCTest_whonix-gateway-17: test_010_screenshare (failure)
      self.assertNotEqual(vm.run('test -e /dev/vid... AssertionError: 0 == 0

    • [unstable] TC_00_QVCTest_whonix-workstation-17: test_010_screenshare (failure)
      self.assertNotEqual(vm.run('test -e /dev/vid... AssertionError: 0 == 0

    • [unstable] TC_00_QVCTest_whonix-workstation-17: test_020_webcam (failure + cleanup)
      AssertionError: 'qubes-video-companion webcam' exited early (0): b'...

  • system_tests_network_updates

    • TC_10_QvmTemplate_fedora-40-xfce: test_000_template_list (failure)
      AssertionError: libvirt event impl drain timeout
  • system_tests_basic_vm_qrexec_gui_zfs

    • TC_06_AppVM_whonix-gateway-17-pool: test_010_os_metadata (failure)
      self.assertEqual(tpl.features.get(... AssertionError: None != 'debian'

    • TC_06_AppVM_whonix-workstation-17-pool: test_010_os_metadata (failure)
      self.assertEqual(tpl.features.get(... AssertionError: None != 'debian'

  • system_tests_guivm_vnc_gui_interactive

    • guivm_startup: Failed (test died)
      # Test died: command 'qvm-run --nogui -pu root sys-gui-vnc env XAUT...
  • system_tests_basic_vm_qrexec_gui_btrfs

    • TC_06_AppVM_whonix-gateway-17-pool: test_010_os_metadata (failure)
      self.assertEqual(tpl.features.get(... AssertionError: None != 'debian'

    • TC_06_AppVM_whonix-workstation-17-pool: test_010_os_metadata (failure)
      self.assertEqual(tpl.features.get(... AssertionError: None != 'debian'

  • system_tests_basic_vm_qrexec_gui_ext4

    • TC_06_AppVM_whonix-gateway-17-pool: test_010_os_metadata (failure)
      self.assertEqual(tpl.features.get(... AssertionError: None != 'debian'

    • TC_06_AppVM_whonix-workstation-17-pool: test_010_os_metadata (failure)
      self.assertEqual(tpl.features.get(... AssertionError: None != 'debian'

  • system_tests_basic_vm_qrexec_gui_xfs

    • TC_06_AppVM_whonix-gateway-17-pool: test_010_os_metadata (failure)
      self.assertEqual(tpl.features.get(... AssertionError: None != 'debian'

    • TC_06_AppVM_whonix-workstation-17-pool: test_010_os_metadata (failure)
      self.assertEqual(tpl.features.get(... AssertionError: None != 'debian'

  • system_tests_splitgpg

  • system_tests_basic_vm_qrexec_gui

    • TC_06_AppVM_whonix-gateway-17: test_010_os_metadata (failure)
      self.assertEqual(tpl.features.get(... AssertionError: None != 'debian'

    • TC_06_AppVM_whonix-workstation-17: test_010_os_metadata (failure)
      self.assertEqual(tpl.features.get(... AssertionError: None != 'debian'

    • TC_20_AudioVM_PipeWire_fedora-40-xfce: test_226_audio_playback_pipewire (failure)
      AssertionError: Command 'timeout 20s pw-play --format=f32 --rate=44...

  • system_tests_kde_gui_interactive

    • gui_filecopy: unnamed test (unknown)
    • gui_filecopy: Failed (test died)
      # Test died: no candidate needle with tag(s) 'files-test-file' matc...

Fixed failures

Compared to: https://openqa.qubes-os.org/tests/101100#dependencies

38 fixed
  • system_tests_pvgrub_salt_storage

    • StorageFile: test_001_non_volatile (error)
      subprocess.CalledProcessError: Command '/usr/lib/qubes/destroy-snap...
  • system_tests_network_updates

    • TC_00_Dom0Upgrade_whonix-gateway-17: test_006_update_flag_clear (failure)
      Error: Failed to download metadata for repo 'test': Cannot download...

    • TC_10_QvmTemplate_debian-12-xfce: test_010_template_install (failure)
      AssertionError: qvm-template failed: Downloading 'qubes-template-de...

    • TC_10_QvmTemplate_fedora-40-xfce: test_010_template_install (failure)
      AssertionError: qvm-template failed: Downloading 'qubes-template-de...

    • TC_10_QvmTemplate_whonix-gateway-17: test_010_template_install (failure)
      AssertionError: qvm-template failed: Downloading 'qubes-template-de...

    • TC_11_QvmTemplateMgmtVM_debian-12-xfce: test_010_template_install (failure)
      AssertionError: qvm-template failed: Downloading 'qubes-template-de...

    • TC_11_QvmTemplateMgmtVM_fedora-40-xfce: test_010_template_install (failure)
      AssertionError: qvm-template failed: Downloading 'qubes-template-de...

    • TC_11_QvmTemplateMgmtVM_whonix-gateway-17: test_010_template_install (failure)
      AssertionError: qvm-template failed: Downloading 'qubes-template-de...

  • system_tests_basic_vm_qrexec_gui_zfs

  • system_tests_basic_vm_qrexec_gui_btrfs

  • system_tests_basic_vm_qrexec_gui_ext4

  • system_tests_basic_vm_qrexec_gui_xfs

  • system_tests_basic_vm_qrexec_gui@hw1

  • system_tests_usbproxy

  • system_tests_basic_vm_qrexec_gui

  • system_tests_guivm_gui_interactive

    • update_guivm: Failed (test died)
      # Test died: command '(set -o pipefail; qubesctl --all --show-outpu...

Unstable tests

Details
  • system_tests_suspend

    suspend/ (2/4 times with errors)
    suspend/Failed (2/4 times with errors)
    • job 102431 # Test died: no candidate needle with tag(s) 'xscreensaver-prompt' ...
    • job 102443 # Test died: no candidate needle with tag(s) 'xscreensaver-prompt' ...
    suspend/wait_serial (2/4 times with errors)
    • job 102431 # wait_serial expected: qr/2E8vz-\d+-/...
    • job 102443 # wait_serial expected: qr/2E8vz-\d+-/...
  • system_tests_pvgrub_salt_storage

    TC_41_HVMGrub_debian-12-xfce/test_000_standalone_vm (1/2 times with errors)
    • job 101773 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    StorageFile/test_001_non_volatile (1/2 times with errors)
    • job 101124 subprocess.CalledProcessError: Command '/usr/lib/qubes/destroy-snap...
    TC_41_HVMGrub_debian-12-xfce/test_010_template_based_vm (1/2 times with errors)
    • job 101773 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
  • system_tests_extra

    TC_00_QVCTest_whonix-workstation-17/test_010_screenshare (1/2 times with errors)
    • job 101116 self.assertNotEqual(vm.run('test -e /dev/vid... AssertionError: 0 == 0
    TC_00_QVCTest_debian-12-xfce/test_020_webcam (1/2 times with errors)
    • job 101116 AssertionError: 'qubes-video-companion webcam' exited early (0): b'...
    TC_00_QVCTest_fedora-40-xfce/test_020_webcam (1/2 times with errors)
    • job 101765 AssertionError: 'qubes-video-companion webcam' exited early (0): b'...
    TC_00_QVCTest_whonix-workstation-17/test_020_webcam (1/2 times with errors)
    • job 101116 AssertionError: 'qubes-video-companion webcam' exited early (0): b'...
  • system_tests_network_updates

    TC_00_Dom0Upgrade_whonix-gateway-17/test_006_update_flag_clear (1/2 times with errors)
    • job 101123 Error: Failed to download metadata for repo 'test': Cannot download...
    TC_10_QvmTemplate_debian-12-xfce/test_010_template_install (1/2 times with errors)
    • job 101123 AssertionError: qvm-template failed: Downloading 'qubes-template-de...
    TC_10_QvmTemplate_fedora-40-xfce/test_010_template_install (1/2 times with errors)
    • job 101123 AssertionError: qvm-template failed: Downloading 'qubes-template-de...
    TC_10_QvmTemplate_whonix-gateway-17/test_010_template_install (1/2 times with errors)
    • job 101123 AssertionError: qvm-template failed: Downloading 'qubes-template-de...
    TC_11_QvmTemplateMgmtVM_debian-12-xfce/test_010_template_install (1/2 times with errors)
    • job 101123 AssertionError: qvm-template failed: Downloading 'qubes-template-de...
    TC_11_QvmTemplateMgmtVM_fedora-40-xfce/test_010_template_install (1/2 times with errors)
    • job 101123 AssertionError: qvm-template failed: Downloading 'qubes-template-de...
    TC_11_QvmTemplateMgmtVM_whonix-gateway-17/test_010_template_install (1/2 times with errors)
    • job 101123 AssertionError: qvm-template failed: Downloading 'qubes-template-de...
  • system_tests_dispvm

    TC_20_DispVM_fedora-40-xfce/test_100_open_in_dispvm (1/2 times with errors)
    • job 101764 AssertionError: './open-file test.txt' failed with ./open-file test...
  • system_tests_basic_vm_qrexec_gui_btrfs

    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_220_audio_play_pulseaudio (1/2 times with errors)
    • job 101110 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_222_audio_rec_unmuted_pulseaudio (1/2 times with errors)
    • job 101110 AssertionError: only silence detected, no useful audio data
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_223_audio_play_hvm (1/2 times with errors)
    • job 101110 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    TC_20_AudioVM_Pulse_fedora-40-xfce-pool/test_225_audio_rec_unmuted_hvm (1/2 times with errors)
    • job 101782 AssertionError: too short audio, expected 10s, got 6.28875283446712...
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_252_audio_playback_audiovm_switch_hvm (2/2 times with errors)
    • job 101110 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    • job 101782 AssertionError: pacat for test-inst-vm1 (xid 90) running(False) in ...
  • system_tests_basic_vm_qrexec_gui_ext4

    TC_00_Basic/test_141_libvirt_objects_reconnect (1/2 times with errors)
    • job 101111 AssertionError: libvirt event impl drain timeout
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_220_audio_play_pulseaudio (1/2 times with errors)
    • job 101111 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_222_audio_rec_unmuted_pulseaudio (1/2 times with errors)
    • job 101111 AssertionError: only silence detected, no useful audio data
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_223_audio_play_hvm (1/2 times with errors)
    • job 101111 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_252_audio_playback_audiovm_switch_hvm (1/2 times with errors)
    • job 101111 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
  • system_tests_basic_vm_qrexec_gui_xfs

    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_220_audio_play_pulseaudio (2/2 times with errors)
    • job 101112 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    • job 101781 AssertionError: too short audio, expected 10s, got 8.33947845804988...
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_222_audio_rec_unmuted_pulseaudio (2/2 times with errors)
    • job 101112 AssertionError: only silence detected, no useful audio data
    • job 101781 AssertionError: too short audio, expected 10s, got 8.17784580498866...
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_223_audio_play_hvm (2/2 times with errors)
    • job 101112 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    • job 101781 AssertionError: too short audio, expected 10s, got 7.71551020408163...
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_252_audio_playback_audiovm_switch_hvm (1/2 times with errors)
    • job 101112 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
  • system_tests_basic_vm_qrexec_gui@hw1

    TC_20_AudioVM_Pulse_whonix-workstation-17/test_220_audio_play_pulseaudio (2/2 times with errors)
    • job 101109 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    • job 101758 AssertionError: too short audio, expected 10s, got 8.73532879818594...
    TC_20_AudioVM_Pulse_whonix-workstation-17/test_222_audio_rec_unmuted_pulseaudio (2/2 times with errors)
    • job 101109 AssertionError: only silence detected, no useful audio data
    • job 101758 AssertionError: too short audio, expected 10s, got 9.36446712018140...
    TC_20_AudioVM_Pulse_whonix-workstation-17/test_223_audio_play_hvm (1/2 times with errors)
    • job 101109 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    TC_20_AudioVM_Pulse_whonix-workstation-17/test_252_audio_playback_audiovm_switch_hvm (1/2 times with errors)
    • job 101109 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
  • system_tests_suspend@hw1

    suspend/ (2/4 times with errors)
    suspend/Failed (2/4 times with errors)
    • job 102431 # Test died: no candidate needle with tag(s) 'xscreensaver-prompt' ...
    • job 102443 # Test died: no candidate needle with tag(s) 'xscreensaver-prompt' ...
    suspend/wait_serial (2/4 times with errors)
    • job 102431 # wait_serial expected: qr/2E8vz-\d+-/...
    • job 102443 # wait_serial expected: qr/2E8vz-\d+-/...
  • system_tests_usbproxy

    TC_20_USBProxy_core3_whonix-gateway-17/test_070_attach_not_installed_front (1/2 times with errors)
    • job 101101 qubesusbproxy.core3ext.QubesUSBException: Device attach failed: 202...
  • system_tests_basic_vm_qrexec_gui

    TC_20_AudioVM_Pulse_whonix-workstation-17/test_220_audio_play_pulseaudio (2/2 times with errors)
    • job 101109 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    • job 101758 AssertionError: too short audio, expected 10s, got 8.73532879818594...
    TC_20_AudioVM_Pulse_whonix-workstation-17/test_222_audio_rec_unmuted_pulseaudio (2/2 times with errors)
    • job 101109 AssertionError: only silence detected, no useful audio data
    • job 101758 AssertionError: too short audio, expected 10s, got 9.36446712018140...
    TC_20_AudioVM_Pulse_whonix-workstation-17/test_223_audio_play_hvm (1/2 times with errors)
    • job 101109 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    TC_20_AudioVM_Pulse_whonix-workstation-17/test_252_audio_playback_audiovm_switch_hvm (1/2 times with errors)
    • job 101109 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...

@piotrbartman piotrbartman marked this pull request as ready for review April 10, 2024 03:53
@marmarek

Copy link
Copy Markdown
Member

Starting a qube with CDROM seems to be broken: https://openqa.qubes-os.org/tests/96799#step/TC_00_Basic/4

@marmarek

Copy link
Copy Markdown
Member

And also, unit tests need an update (and also pylint is unhappy)

@marmarek

Copy link
Copy Markdown
Member

I keep device_protocol.py review in QubesOS/qubes-core-admin#579, other changes here looks okay (but I haven't tested the most recent version here yet).

@marmarek

marmarek commented May 1, 2024

Copy link
Copy Markdown
Member

qvm-start --cdrom still is not happy:

Traceback (most recent call last):
  File "/usr/lib64/python3.11/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/qubes/tests/integ/basic.py", line 105, in test_120_start_standalone_with_cdrom_dom0
    self.assertEqual(p.returncode, 0, stdout)
AssertionError: 1 != 0 : b'Can not remove an required assignment from a non halted qube.\n'

See also updated comment from the today's test run, most failures looks to be related to the devices API rework (but some are probably "just" tests to be updated).

@piotrbartman piotrbartman force-pushed the prb/new_device_api branch from bc0d9bb to a1ba96e Compare May 8, 2024 13:03
@marmarek marmarek merged commit 769e7e1 into QubesOS:main Jun 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants