Add top-like monitoring tool

[ben-grande]

Add top-like monitoring tool

Inspiration and code from qui-domains and visuals from xentop.

• Colors entries according to state
• Footer with clock to show last time the screen was refreshed
• Allows scrolling domain list when it is longer than the screen height,
  with a less-type hint indicating how many lines to scroll
• To scroll, readline style (Emacs) navigation is supported, with some
  Vi keys also supported
• Inexpensive calls using "admin.vm.Stats"

Requires: QubesOS/qubes-core-admin#826
Requires: QubesOS/qubes-core-admin#827

Breaking change on a commit, which requires new client versions:

• Update CPU usage to float type qubes-desktop-linux-manager#315

[marmarek]

is a blocking operation as well as proc.stdout.read(2)

Do you mean read() hangs for you? Maybe the service waits for something before starting to output any data (doesn't look like it based on the other PR...).

[ben-grande]

Not the service by default, I prepended sleep to awk to test if a DomU could hang operation on Dom0 and if this is a valid concern.

…

On Tue, Jan 28, 2025, 12:25 PM Marek Marczykowski-Górecki < ***@***.***> wrote: is a blocking operation as well as proc.stdout.read(2) Do you mean read() hangs for you? Maybe the service waits for something before starting to output any data (doesn't look like it based on the other PR...). — Reply to this email directly, view it on GitHub <#333 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BCE2O4LDOXXM2LA43ENCLQ32M5SKPAVCNFSM6AAAAABWAFNTCOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMJYG4ZDGNBUG4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[marmarek]

Based on discussion in the other PR, this should be converted to use stat events (and events extended to include necessary info). Converting to draft until then.

[ben-grande]

LLM helped on draw_table(). It's almost stable. It works nicely on R4.2, but not on R4.3. I don't know yet why builds are failing on manpages.

[codecov]

Codecov Report

❌ Patch coverage is 1.38889% with 1065 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.04%. Comparing base (0c7b921) to head (b9be4aa).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
qubesadmin/tools/qvm_top.py	0.00%	1058 Missing ⚠️
qubesadmin/app.py	54.54%	5 Missing ⚠️
qubesadmin/tools/__init__.py	83.33%	1 Missing ⚠️
qubesadmin/vm/__init__.py	66.66%	1 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (0c7b921) and HEAD (b9be4aa). Click for more details.

HEAD has 1 upload less than BASE

Flag	BASE (0c7b921)	HEAD (b9be4aa)
	2	1

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #333      +/-   ##
==========================================
- Coverage   76.79%   69.04%   -7.75%     
==========================================
  Files          53       54       +1     
  Lines        9402    10473    +1071     
==========================================
+ Hits         7220     7231      +11     
- Misses       2182     3242    +1060     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ben-grande]

One thing I don't plan to fix is the memory reporting being wrong, as only qmemman has the knowledge about memory of each qube, and getting information from it is not currently nice with it's current API (synchonous), and I don't want to delay reporting statistics because qmemman is busy for seconds at a time.

• Asynchronous qmemman API qubes-issues#10857

[ben-grande]

Maybe it's not necessary, reading past conversations, something that I didn't understand one year ago (qmemman, admin.vm.Stats), I think I will settle for meminfo: QubesOS/qubes-core-agent-linux#550 (comment).

[ben-grande]

xentop has cpu_time for each vcpu, but that may be too much. Xentop has actually a lot of things that can be show about virtual blocks also, but I don't like that much info. Customizing delay is not important to us, the screen refreshed in a loop and the delay is pretty insignificant and is not causing issues to navigate up and down. One think I like is that xentop has sort order, but I can't really see which column is currently being sorted, it's supposed to be bold. Anyway, I like how qvm-top is right now and don't plan to improve further (unless I think about this again, tomorrow...).

[ben-grande]

And one more bug, for dom0, it should use cpu_usage_raw instead of cpu_usage. It seems that the value gets much closer to what xentop reports about Domain-0.

[ben-grande]

And yes, vm-stats from my core-admin PR is reporting stubdomains, but as the client VMCollection doesn't know about stubdomains, it's never considered. Maybe I could workaround this limitation.

[ben-grande]

• stubdom reporting, can't show memory usage for it, but can show CPU stats
• NA is better than lying how much memory the qube is using
• I don't know why Xen adds one mebibyte to maxmem_kb (domain_getinfo) but not on static-max xenstore key.

[marmarek]

I don't know why Xen adds one mebibyte to maxmem_kb (domain_getinfo) but not on static-max xenstore key.

I'm not sure either, but I guess domain_getinfo includes some internal overhead in Xen (EPT? maybe something else too?), but static-max is just what OS can really see.

Anyway, I don't like exposing stubdomains in a high level tool, it should be counted as part of the target qube; if anything, maybe some separate column for some internal overhead, but not separate row.
If somebody really want to look under the hood, they have xentop and other low level tools.

[ben-grande]

I have added a ton of things recently. You can do everything from the keyboard or mouse. You can filter the view, sort by column (or invert), select one or more domains. For the latter, it does nothing currently besides highlighting, the goal is to have an htop like kill column, where you select the kill signal, in our case, it would be "shutdown, kill, (re)start". If there is anything you'd like to see, please let me know. There will be a video tomorrow, if QVC doesn't burn my computer.

In the far future, this might become the qube manager for headless connections, but for now, I will restrict just for statistics due to time.

[ben-grande]

• Tested with RemoteVM
• Reduced default view to just a few columns
• Improved no color view
• Actions available, from niceties such as terminal or console to controlling qube state
• Reordered columns as total, used, internal
• New swap column
• Qube names are colored according to their label if the terminal supports, else it will fallback to something close enough

I think I am finished on the client, but will have to retest it after writing and fixing tests of QubesOS/qubes-core-admin#827.

---

For the description of each column, see the manual page or the help= of the Column.

Minimal view (55 columns):

Default view (79 columns):

Maximum view (203 columns):

The min view is what qui-domains should show.

The default view is what you get without specifying any arguments.

If you noticed, I didn't include maxmem property in the default or min, as I think is MAU -> "memory_assigned_usable", which is what is available in the domain, is more relevant to show available to the domain and MU/MAU to show memory usage (excluding swap, separate column MUS).

The max view is all you can get. Great for debugging the application and see if values add up when they should.

---

Actions:

Notice that when selecting both domains, you can only get the common actions.