When hitting a crash in e.g. QEMU it would be helpful to have debug symbols so that backtraces could be better understood. For Debian this could mean providing an incus-dbgsym package.
E.g. during a stateful migration from/to QEMU 10.1.2 (6.19.1-debian12-202511291632) hit an assertion in a bitmap setter.
# coredumpctl
TIME PID UID GID SIG COREFILE EXE SIZE
Mon 2025-12-01 11:01:33 CET 1554941 0 0 SIGABRT present /opt/incus/bin/qemu-system-x86_64 911.9M
Dec 01 11:01:33 incus2.example.com incusd[1022819]: time="2025-12-01T11:01:33+01:00" level=error msg="Failed cancelling block job" err="Monitor is disconnected" instance=auth2 instanceType=virtual-machine project=default
Dec 01 11:01:33 incus2.example.com incusd[1022819]: time="2025-12-01T11:01:33+01:00" level=warning msg="Instance stopped" instance=auth2 instanceType=virtual-machine project=default reason=disconnect target=stop
Dec 01 11:01:34 incus2.example.com incusd[1022819]: time="2025-12-01T11:01:34+01:00" level=warning msg="Failed removing NBD storage target device" err="Failed removing block device: Monitor is disconnected" instance=auth2 instanceType=virtual-machine project=default
Dec 01 11:01:34 incus2.example.com incusd[1022819]: time="2025-12-01T11:01:34+01:00" level=warning msg="Failed resuming instance" err="Monitor is disconnected" instance=auth2 instanceType=virtual-machine project=default
Dec 01 11:01:34 incus2.example.com incusd[1022819]: time="2025-12-01T11:01:34+01:00" level=error msg="Failed merging migration storage snapshot" err="Monitor is disconnected" instance=auth2 instanceType=virtual-machine project=default
Dec 01 11:01:34 incus2.example.com incusd[1022819]: time="2025-12-01T11:01:34+01:00" level=error msg="Failed migration on source" clusterMoveSourceName=auth2 err="Failed waiting for state transfer to reach pre-switchover stage: Monitor is disconnected" instance=auth2 live=true project=default push=false
# incus config show auth2 -e
architecture: x86_64
config:
boot.autostart: "true"
limits.cpu: "2"
limits.memory: 2GiB
migration.stateful: "true"
security.csm: "true"
security.secureboot: "false"
user.ha_group: auth1-2
volatile.cloud-init.instance-id: 692edf6a-ede9-4420-8f87-22850e012c54
volatile.eth0.host_name: tap6b7c465c
volatile.eth0.hwaddr: 10:66:6a:c6:ab:9e
volatile.last_state.power: RUNNING
volatile.uuid: b1c6ba88-7161-428d-9ebd-61c56e1e9c9e
volatile.uuid.generation: b1c6ba88-7161-428d-9ebd-61c56e1e9c9e
volatile.vm.definition: pc-q35-10.1
volatile.vm.rtc_adjustment: "2"
volatile.vm.rtc_offset: "-1"
volatile.vsock_id: "274146095"
devices:
eth0:
network: vlan2015
type: nic
root:
path: /
pool: tank
size: 32GiB
type: disk
ephemeral: false
profiles:
- vm
- cpu2
- ram2
- disk32
stateful: false
description: ""
and the backtrace looks like this:
(gdb) bt full
#0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
tid = <optimized out>
ret = 0
pd = <optimized out>
old_mask = {__val = {94081552270688}}
ret = <optimized out>
#1 0x00007f4896b18f4f in __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
No locals.
#2 0x00007f4896ac9fb2 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
ret = <optimized out>
#3 0x00007f4896ab4472 in __GI_abort () at ./stdlib/abort.c:79
save_stage = 1
act = {__sigaction_handler = {sa_handler = 0x20, sa_sigaction = 0x20}, sa_mask = {__val = {167, 94081649149760, 81, 139949743938656, 110, 47244640256, 0, 139946871458296,
3592576350872602368, 5, 18446744073709542696, 11, 94081325734995, 167, 94081325735012, 94081552270688}}, sa_flags = -1766690993, sa_restorer = 0x7f4896c2647c <_nl_C_name>}
#4 0x00007f4896ab4395 in __assert_fail_base (fmt=0x7f4896c29a90 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x559104001c64 "start >= 0 && nr >= 0",
file=file@entry=0x559104001c53 "../util/bitmap.c", line=line@entry=167, function=function@entry=0x559104001ce8 "bitmap_set") at ./assert/assert.c:94
str = 0x559117470740 "P\354\231H\224U"
total = 4096
#5 0x00007f4896ac2ec2 in __GI___assert_fail (assertion=0x559104001c64 "start >= 0 && nr >= 0", file=0x559104001c53 "../util/bitmap.c", line=167, function=0x559104001ce8 "bitmap_set")
at ./assert/assert.c:103
No locals.
#6 0x0000559103e0a4a8 in ?? ()
No symbol table info available.
#7 0x0000559103cfc5db in ?? ()
No symbol table info available.
#8 0x0000559103cfc728 in ?? ()
No symbol table info available.
#9 0x0000559103cf650e in ?? ()
No symbol table info available.
#10 0x0000559103cf6cea in ?? ()
No symbol table info available.
#11 0x0000559103cf834a in ?? ()
No symbol table info available.
#12 0x0000559103ce8024 in ?? ()
No symbol table info available.
#13 0x0000559103ce810b in ?? ()
No symbol table info available.
#14 0x0000559103e1d82b in ?? ()
No symbol table info available.
#15 0x00007f4896adf9c0 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#16 0x00007ffe671b0b90 in ?? ()
No symbol table info available.
#17 0x0000000000000000 in ?? ()
No symbol table info available.
It would be helpful if we could track this down to the consumer that caused this, but without debug symbols that feels next to impossible.
When hitting a crash in e.g. QEMU it would be helpful to have debug symbols so that backtraces could be better understood. For Debian this could mean providing an
incus-dbgsympackage.E.g. during a stateful migration from/to QEMU 10.1.2 (6.19.1-debian12-202511291632) hit an assertion in a bitmap setter.
and the backtrace looks like this:
It would be helpful if we could track this down to the consumer that caused this, but without debug symbols that feels next to impossible.