Cache libvirt domain state by ben-grande · Pull Request #819 · QubesOS/qubes-core-admin

ben-grande · 2026-06-01T17:48:09Z

The state changes rarely, but querying it can take a considerable time, that is blocking, when looping through the state of multiple domains.

For: QubesOS/qubes-issues#10569
For: QubesOS/qubes-issues#9902

Didn't run openqa locally, only experiment some things:

start
shutdown
kill
pause
unpause
checking Domains Widget
restart qubesd
qubed-query -e dom0 admin.vm.CurrentState QUBE

TODO:

test with Fix devices and other behavior for remotevms #833

ben-grande · 2026-06-01T17:54:05Z

PipelineRetryFailed

ben-grande · 2026-06-01T17:58:23Z

Didn't run openqa locally, only experiment some things, start, shutdown, kill, checking Domains Widget.

ben-grande · 2026-06-01T19:39:47Z

PipelineRetryFailed

codecov · 2026-06-01T19:45:19Z

Codecov Report

❌ Patch coverage is 43.11927% with 62 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.12%. Comparing base (9196c9a) to head (367ae38).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
qubes/vm/qubesvm.py	31.42%	48 Missing ⚠️
qubes/device_protocol.py	0.00%	5 Missing ⚠️
qubes/vm/adminvm.py	20.00%	4 Missing ⚠️
qubes/ext/pci.py	87.50%	3 Missing ⚠️
qubes/app.py	33.33%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #819      +/-   ##
==========================================
- Coverage   70.42%   70.12%   -0.30%     
==========================================
  Files          61       61              
  Lines       14143    14174      +31     
==========================================
- Hits         9960     9940      -20     
- Misses       4183     4234      +51

Flag	Coverage Δ
unittests	`70.12% <43.11%> (-0.30%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ben-grande · 2026-06-01T21:55:44Z

PipelineRetryFailed

ben-grande · 2026-06-02T07:42:48Z

PipelineRetryFailed

ben-grande · 2026-06-02T09:05:37Z

Marked as draft as there still is some caching issues for the _power_state.

marmarek · 2026-06-02T09:35:58Z

If necessary (or simpler), it's IMO okay to use cache only for is_running() (that is used frequently), but still do active check for get_power_state(). It could be also a safety valve for cases when cache are not reliable (like during startup or shutdown).

ben-grande · 2026-06-02T11:33:48Z

If necessary (or simpler), it's IMO okay to use cache only for is_running() (that is used frequently), but still do active check for get_power_state(). It could be also a safety valve for cases when cache are not reliable (like during startup or shutdown).

Experimenting with cache a bit more and it's getting better, but if I don't get it to a hundred percent, will not cache the power state.

ben-grande · 2026-06-02T12:08:04Z

From my tests, it seems to be working. My nemesis, OpenQA, should try me. Do you know a subset of tests that are interesting before joining this PR in the full run?

Looking at this recent test:

system_tests_suspend (soft-failed)
system_tests_usbproxy
system_tests_devices
system_tests_gui_tools
system_tests_guivm_gui_interactive

Line to use after this PR is not draft anymore:

openQArun TEST=system_tests_suspend,system_tests_usbproxy,system_tests_devices,system_tests_gui_tools,system_tests_guivm_gui_interactive

marmarek · 2026-06-02T13:27:44Z

I'd include also at least one of the guivm tests

ben-grande · 2026-06-02T22:03:45Z

PipelineRetryFailed

ben-grande · 2026-06-02T22:09:18Z

Last CI failed on Fedora 43 to due to:

======================================================================
ERROR: qubes.tests.api_admin/TC_00_VMs/test_150_pool_info
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.14/contextlib.py", line 85, in inner
    return func(*args, **kwds)
  File "/home/gitlab-runner/builds/QubesOS/qubes-core-admin/qubes/tests/api_admin.py", line 76, in setUp
    super().setUp()
    ~~~~~~~~~~~~~^^
  File "/home/gitlab-runner/builds/QubesOS/qubes-core-admin/qubes/tests/__init__.py", line 516, in setUp
    self.loop = asyncio.get_event_loop()
                ~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib64/python3.14/asyncio/events.py", line 715, in get_event_loop
    raise RuntimeError('There is no current event loop in thread %r.'
                       % threading.current_thread().name)
RuntimeError: There is no current event loop in thread 'MainThread'.

So it finally reached this repo.

ben-grande · 2026-06-02T22:15:22Z

PipelineRetryFailed

ben-grande · 2026-06-03T09:05:59Z

openQArun TEST=system_tests_suspend,system_tests_usbproxy,system_tests_devices,system_tests_gui_tools,system_tests_guivm_gui_interactive

Since there is "resume", and there is a moment when resuming that domain might still be suspended, or event suspending a domain on purposed, allow clients to know the domain state.

Useful when debugging to know what's happening. I thought of logging the pretty name of the detail, but that got really big and I think it's out of scope from the Qubes OS project and in scope of the python-libvirt package.

- Keep events related to domain state in the same place - Shorter exception messages, as it's already prefixed with qube name - Much easier to read with less body on the if statements - Will help on a future commit to access private attributes without having to disable pylint checks - The events are related to a domain, not to the larger app instance

The state changes rarely, but querying it can take a considerable blocking time, worse when looping through the state of multiple domains. For: QubesOS/qubes-issues#10569 For: QubesOS/qubes-issues#9902

These data only changes once when domain is running.

This function reimports the same information over and over and is time consuming when looping through multiple domains. Reduce listing USB devices while there is no cache from 2.8s to 0.5s.

Method 'startwith' adds considerable overhead.

Libvirt API can be quite slow on the methods for "listAllDevices" and "XMLDesc", aggravated in loops.

Assuming that hotplug is unsupported.

Each call was recording 1ms, now the calls are below that.

Although it doesn't help reduce the time to get the XML, as XMLDesc() is super slow, this helps cleanup the code a bit.

Skip PVH when listing attached PCI devices, as they can't possibly be there.

DemiMarie

XPath variables are super nice. They are the XPath equivalent of SQL parameterized queries.

DemiMarie · 2026-06-22T22:33:40Z

-            if source.get("dev") == "/dev/%s" % volume.vid:
-                return True
+        disks = parsed_xml.xpath(
+            "//domain/devices/disk/source[@dev='/dev/{}']".format(volume.vid)


Instead of injecting the variable with .format(), use XPath variables. Something like this (not tested):

disks = parsed_xml.xpath( "//domain/devices/disk/source[@dev = $dev]", dev = "/dev/" + volume.vid)

This avoids an XPath injection vulnerability if volume.vid winds up containing an apostraphe. You can use a regular expression to validate volume.vid, but even then it is much better to use XPath variables.

.find() is even faster if it’s an option.

DemiMarie · 2026-06-22T22:33:52Z

+        disks = parsed_xml.xpath(
+            "//domain/devices/disk/source[@dev='/dev/{}']".format(volume.vid)
+        )


Use an XPath variable.

ben-grande force-pushed the cache-running branch from 16f1a70 to fd2a0d4 Compare June 1, 2026 18:03

ben-grande force-pushed the cache-running branch 2 times, most recently from 5eff83f to aa2e823 Compare June 1, 2026 21:26

ben-grande marked this pull request as draft June 2, 2026 07:55

ben-grande force-pushed the cache-running branch from aa2e823 to d8a2614 Compare June 2, 2026 11:19

ben-grande force-pushed the cache-running branch 4 times, most recently from 9538d21 to 6011ca3 Compare June 2, 2026 21:30

ben-grande marked this pull request as ready for review June 3, 2026 08:41

ben-grande mentioned this pull request Jun 3, 2026

qubes-core-admin tests are broken when running in a Fedora 43 AppVM QubesOS/qubes-issues#10820

Closed

qubesos-qa Bot temporarily deployed to qa June 3, 2026 09:06 Inactive

ben-grande force-pushed the cache-running branch 4 times, most recently from 7bda37a to a0a7e60 Compare June 12, 2026 16:12

ben-grande added 3 commits June 13, 2026 00:35

Inform clients about suspend

08e7f0c

Since there is "resume", and there is a moment when resuming that domain might still be suspended, or event suspending a domain on purposed, allow clients to know the domain state.

Log libvirt events

9492f3c

Useful when debugging to know what's happening. I thought of logging the pretty name of the detail, but that got really big and I think it's out of scope from the Qubes OS project and in scope of the python-libvirt package.

ben-grande force-pushed the cache-running branch 2 times, most recently from f140ed3 to 29678d5 Compare June 13, 2026 00:14

ben-grande added 2 commits June 17, 2026 11:17

Cache libvirt domain state

e401382

The state changes rarely, but querying it can take a considerable blocking time, worse when looping through the state of multiple domains. For: QubesOS/qubes-issues#10569 For: QubesOS/qubes-issues#9902

Remove outdated power state graph

8a88b26

ben-grande force-pushed the cache-running branch 2 times, most recently from 2d4a274 to fd6ef64 Compare June 17, 2026 12:03

ben-grande mentioned this pull request Jun 17, 2026

Report more statistics #827

Draft

ben-grande added 12 commits June 18, 2026 14:52

Cache static libvirt information

e2d2092

These data only changes once when domain is running.

Cache pure function to get entry points

92144a7

This function reimports the same information over and over and is time consuming when looping through multiple domains. Reduce listing USB devices while there is no cache from 2.8s to 0.5s.

Avoid use of startwith in a large loop

47dce9b

Method 'startwith' adds considerable overhead.

Query only PCI devices from libvirt API

5258c21

Libvirt API can be quite slow on the methods for "listAllDevices" and "XMLDesc", aggravated in loops.

Cache libvirt device nodes

f239554

Assuming that hotplug is unsupported.

Avoid looping host devices that aren't PCI

f3b2104

Skip case normalization for speed by matching case

1c10380

Each call was recording 1ms, now the calls are below that.

EXPERIMENTAL Filter block devices to avoid XMLDesc

ccfcc35

Avoid code duplication

88dcffd

Fine-grained XML query for cleanliness

d1af6e9

Although it doesn't help reduce the time to get the XML, as XMLDesc() is super slow, this helps cleanup the code a bit.

Cache PCI devices

cb1bfa8

Skip PVH when listing attached PCI devices, as they can't possibly be there.

Cache block devices

3e21f97

ben-grande force-pushed the cache-running branch from fd6ef64 to 3e21f97 Compare June 18, 2026 12:52

ben-grande mentioned this pull request Jun 22, 2026

Menu is noticably slow to open QubesOS/qubes-issues#10777

Open

DemiMarie reviewed Jun 22, 2026

View reviewed changes

ben-grande marked this pull request as draft June 24, 2026 12:23

Uh oh!

Uh oh!

Conversation

ben-grande commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ben-grande commented Jun 1, 2026

Uh oh!

ben-grande commented Jun 1, 2026

Uh oh!

ben-grande commented Jun 1, 2026

Uh oh!

codecov Bot commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ben-grande commented Jun 1, 2026

Uh oh!

ben-grande commented Jun 2, 2026

Uh oh!

ben-grande commented Jun 2, 2026

Uh oh!

marmarek commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ben-grande commented Jun 2, 2026

Uh oh!

ben-grande commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

marmarek commented Jun 2, 2026

Uh oh!

ben-grande commented Jun 2, 2026

Uh oh!

ben-grande commented Jun 2, 2026

Uh oh!

ben-grande commented Jun 2, 2026

Uh oh!

ben-grande commented Jun 3, 2026

Uh oh!

DemiMarie left a comment

Choose a reason for hiding this comment

Uh oh!

DemiMarie Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

DemiMarie Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ben-grande commented Jun 1, 2026 •

edited

Loading

codecov Bot commented Jun 1, 2026 •

edited

Loading

marmarek commented Jun 2, 2026 •

edited

Loading

ben-grande commented Jun 2, 2026 •

edited

Loading