AEM not booting on Intel legacy boot

[miczyg1]

I was hitting a bug in this function: https://github.com/TrenchBoot/xen/blob/aem-4.17.4/xen/arch/x86/intel_txt.c#L2
on the E820 checks/modifications:

(XEN) Xen version 4.17.4 (user@[unknown]) (gcc (GCC) 12.3.1 20230508 (Red Hat 12.3.1-1)) debug=y Sun Sep 22 09:49:27 UTC 2024
(XEN) Latest ChangeSet: 
(XEN) build-id: 0f9c9aafe32a3dd85ea590f5dd139d085afbd9d3
(XEN) Bootloader: GRUB 2.06
(XEN) Command line: placeholder loglvl=all console=com1 com1=115200,8n1,pci dom0_mem=min:1024M dom0_mem=max:4096M smt=off ucode=scan gnttab_max_frames=2048 gnttab_max_maptrack_frames=4096
(XEN) Xen image load base address: 0x75000000
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN) Disc information:
(XEN)  Found 1 MBR signatures
(XEN)  Found 1 EDD information structures
(XEN) CPU Vendor: Intel, Family 6 (0x6), Model 154 (0x9a), Stepping 3 (raw 000906a3)
(XEN) Enabling Supervisor Shadow Stacks
(XEN) Enabling Indirect Branch Tracking
(XEN) SLAUNCH: reserving event log (0x7582c000 - 0x75834000)
(XEN) SLAUNCH: reserving TXT heap (0x7b710000 - 0x7b800000)
(XEN) Xen BUG at arch/x86/intel_txt.c:44
(XEN) ----[ Xen-4.17.4  x86_64  debug=y  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e008:[<ffff82d040403efa>] protect_txt_mem_regions+0x115/0x11f
(XEN) RFLAGS: 0000000000010046   CONTEXT: hypervisor
(XEN) rax: 0000000000000000   rbx: 0000000075834000   rcx: 0000000000000001
(XEN) rdx: 0000000000000012   rsi: 000000000000001e   rdi: 000000007b800000
(XEN) rbp: ffff82d04041fd98   rsp: ffff82d04041fd88   r8:  0000000000000002
(XEN) r9:  0000000000000000   r10: 0000000000000001   r11: 0000000000000010
(XEN) r12: 000000007582c000   r13: 0000000000000000   r14: ffff82d0403ebb30
(XEN) r15: 0000000000000000   cr0: 0000000080050033   cr4: 00000000008000a0
(XEN) cr3: 0000000075481000   cr2: 0000000000000000
(XEN) fsb: 0000000000000000   gsb: 0000000000000000   gss: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) Xen code around <ffff82d040403efa> (protect_txt_mem_regions+0x115/0x11f):
(XEN)  0f 5b 41 5c 5d c3 0f 0b <0f> 0b 0f 0b 0f 0b 0f 0b 0f 0b 48 8b 15 ed 2b 05
(XEN) Xen stack trace from rsp=ffff82d04041fd88:
(XEN)    0000000075834000 000000007582c000 ffff82d04041fdb8 ffff82d040407c40
(XEN)    ffff82d0403a3070 00007d2fbfca34d0 ffff82d04041fee8 ffff82d040405c35
(XEN)    ffffffffffffffff 0000000000000000 753ebb2c753eb0f0 7541fe5c7541fea0
(XEN)    ffff82d04041fef8 7541fe1800000012 0000000000000020 7582c4d000000000
(XEN)    7541fe5c753ebb2c 0000000000000000 0020000b00000001 000000007582c4d2
(XEN)    ffff83000009dec0 0000000000000000 ffff83000009df80 ffff83000009dfb0
(XEN)    0000000000000000 0000000000000000 0000000100000000 7582c4d20020000b
(XEN)    0000000000000000 0000000000000000 0000000000000000 2d00000000000000
(XEN)    0000000800000000 000000010000006e 0000000000000003 00000000000002f8
(XEN)    753ea64801000000 753ebe3a0009ded0 0000000000000002 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 ffff82d04027961b 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000e01000000000
(XEN)    0000000000000000 0000000000000000 00000000008000a0 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000
(XEN) Xen call trace:
(XEN)    [<ffff82d040403efa>] R protect_txt_mem_regions+0x115/0x11f
(XEN)    [<ffff82d040407c40>] F protect_slaunch_mem_regions+0x73/0xf8
(XEN)    [<ffff82d040405c35>] F __start_xen+0xaa6/0x26cc
(XEN)    [<ffff82d04027961b>] F __high_start+0xdb/0xdd
(XEN) 
(XEN) 
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Xen BUG at arch/x86/intel_txt.c:44
(XEN) ****************************************
(XEN) 
(XEN) Reboot in five seconds...

This code version worked well: https://github.com/TrenchBoot/xen/blob/v0.4.0/xen/arch/x86/intel_txt.c#L37

Hit the bug on protectli VP4670 running coreboot. coreboot already marks the TXT memory as reserved, so this could be a problem possibly?

[krystian-hebel]

Xen BUG at arch/x86/intel_txt.c:44

There is no BUG_ON in that line, and neighboring ones suggest that this may happen because either TXTCR_SINIT_BASE or TXTCR_SINIT_SIZE is 0. However, this would be caught by GRUB, so I'm assuming this used to be this BUG_ON. reserve_e820_ram is just a wrapper over e820_change_range_type so I don't expect it was the cause. The BUG_ON itself is new, perhaps it silently failed earlier.

e820_change_range_type can fail in one of two ways: either the entry that covers the whole to-be-reserved region can't be found in E820, or reserving the region would cause it to be split into smaller ones, and there is no more space for additional mapping in the table. If it were the latter, a warning would be printed, which doesn't happen, so it must be the first problem. This in turn would mean that either the range is not described in E820 produced by coreboot/SeaBIOS (instead of being reserved), or it is split across two or more ranges.

v0.5.0-rc3 uses the old way and no BUG_ON, so it may be once again failing silently. Nevertheless, it fails to boot few calls later:

Enabling slaunch ...
Loading Xen 4.17.5 ...
Loading Linux 6.6.48-1.qubes.fc37.x86_64 ...
Loading initial ramdisk ...
Loading SINIT module CFL_SINIT_20221220_PRODUCTION_REL_NT_O1_1.10.1_signed.bin
... Xen 4.17.5
(XEN) Xen version 4.17.5 (user@[unknown]) (gcc (GCC) 12.3.1 20230508 (Red Hat 12.3.1-1)) debug=n Wed Mar 26 16:41:34 UTC 2025
(XEN) Latest ChangeSet: 
(XEN) build-id: eacbdda794f1a1941db53fa7e7dad27857e241b1
(XEN) Bootloader: GRUB 2.13
(XEN) Command line: placeholder console=vga,com1 loglvl=all dom0_mem=min:1024M dom0_mem=max:4096M ucode=no-scan smt=off gnttab_max_frames=2048 gnttab_max_maptrack_frames=4096 no-real-mode edd=off
(XEN) Xen image load base address: 0x98a00000
(XEN) Video information:
(XEN)  VGA is graphics mode 640x480, 32 bpp
(XEN) Disc information:
(XEN)  Found 0 MBR signatures
(XEN)  Found 0 EDD information structures
(XEN) CPU Vendor: Intel, Family 6 (0x6), Model 166 (0xa6), Stepping 0 (raw 000a0660)
(XEN) SLAUNCH: reserving event log (0x99413000 - 0x9941b000)
(XEN) SLAUNCH: reserving TXT heap (0x99f10000 - 0x9a000000)
(XEN) SLAUNCH: reserving SINIT memory (0x99ec0000 - 0x99f10000)
(XEN) Early fatal page fault at e008:ffff82d0403d9ac1 (cr2=0000000099f100d8, ec=0000)
(XEN) ----[ Xen-4.17.5  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e008:[<ffff82d0403d9ac1>] txt_restore_mtrrs+0x31/0x3b0
(XEN) RFLAGS: 0000000000010046   CONTEXT: hypervisor
(XEN) rax: 0000000099f100c8   rbx: 0000000000100800   rcx: 00000000000002ff
(XEN) rdx: 0000000000000000   rsi: ffff82d04037ed1a   rdi: 0000000000000000
(XEN) rbp: 0000000000000000   rsp: ffff82d0403f7d30   r8:  0000000000000000
(XEN) r9:  0000000000000002   r10: ffff8300000ffff1   r11: ffff8300000fffe8
(XEN) r12: ffff82d04037f699   r13: 0000000000001d0a   r14: 0000000000000027
(XEN) r15: ffff82d040429ab0   cr0: 0000000080050033   cr4: 00000000000000a0
(XEN) cr3: 0000000098e58000   cr2: 0000000099f100d8
(XEN) fsb: 0000000000000000   gsb: 0000000000000000   gss: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) Xen code around <ffff82d0403d9ac1> (txt_restore_mtrrs+0x31/0x3b0):
(XEN)  75 45 8b 05 87 a8 05 00 <8b> 48 10 48 8d 58 10 0f 1f 84 00 00 00 00 00 83
(XEN) Xen stack trace from rsp=ffff82d0403f7d30:
(XEN)    0000000099466040 ffff82d0403fc1f0 0000000000100800 ffff82d0405c7b40
(XEN)    ffff82d04037f699 ffff82d0403f7d88 0000000000000027 ffff82d0403d39e3
(XEN)    ffff82d0404299bc ffff82d040429980 000003ed99f10000 49656e69756e6547
(XEN)    000000006c65746e ffff82d040429980 ffff82d04037f699 ffff830000000000
(XEN)    0000000000000138 ffff83000009dfb0 ffff83000009dec0 ffff82d0403db179
(XEN)    ffff83000009dfb0 ffffffffffffffff 98df7e5c00000000 0000000000000000
(XEN)    0000000000000000 000000000009df80 9941360800000000 ffff82d0403f7ef8
(XEN)    0000000000000000 0014000400000002 0020000b994135f4 000000009941360a
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000002
(XEN)    0000000000000000 0000000200000000 0000000800000000 000000010000006e
(XEN)    0000000000000003 00000000000002f8 2d00000000000000 0009df7898dbf11e
(XEN)    98dbf11998dbf169 98dbec9d00000065 0000000498dbf178 98dbf17801dbf14e
(XEN)    98dc09fa0009ded0 98dbee3e00000000 0000000000000000 0000000099ec0000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    ffff82d04027589b 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) Xen call trace:
(XEN)    [<ffff82d0403d9ac1>] R txt_restore_mtrrs+0x31/0x3b0
(XEN)    [<ffff82d0403d39e3>] S init_e820+0x513/0x540
(XEN)    [<ffff82d0403db179>] S __start_xen+0x689/0x26f5
(XEN)    [<ffff82d04027589b>] S __high_start+0xdb/0xe0
(XEN) 
(XEN) Pagetable walk from 0000000099f100d8:
(XEN)  L4[0x000] = 0000000098e14063 ffffffffffffffff
(XEN) 
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) FATAL TRAP: vec 14, #PF[0000] IN INTERRUPT CONTEXT
(XEN) ****************************************
(XEN) 
(XEN) Reboot in five seconds...

This #PF most likely means that SLRT isn't mapped. It didn't happen on AMD because SLTR is part of SLB there, while for Intel it can either be part of TXT heap (in which case it should be mapped) or normal RAM (in which case it isn't, unless by pure chance it lands on the same 2MiB page as something that is mapped). Interestingly, for legacy boot it should be in TXT heap, and for UEFI - normal RAM.

It doesn't explain lack of mapping either way, since txt_restore_mtrrs is called from init_e820, which is executed after map_slaunch_mem_regions and tpm_measure_slrt. First one of those maps (among other things) TXT heap, and the second one calls slr_get_table, which maps SLTR again. I don't really understand why it isn't enough.

@SergiiDmytruk CC, but perhaps this is something for v0.6.0.

[SergiiDmytruk]

Given that slaunch_slrt holds a physical address, seems like

xen/xen/arch/x86/intel_txt.c

Line 146 in 1a33e90

slrt = (struct slr_table *)(uintptr_t)slaunch_slrt;

lacks __va().

[krystian-hebel]

Seems like it. I suggest hiding slaunch_slrt and exposing slr_get_table() instead, but this may be difficult for code in tpm.c because it needs to work both in early and normal Xen code.

[miczyg1]

There is no BUG_ON in that line, and neighboring ones suggest that this may happen because either TXTCR_SINIT_BASE or TXTCR_SINIT_SIZE is 0. However, this would be caught by GRUB, so I'm assuming this used to be this BUG_ON. reserve_e820_ram is just a wrapper over e820_change_range_type so I don't expect it was the cause. The BUG_ON itself is new, perhaps it silently failed earlier.

I have been trying to fix this and locate the root cause, so the lines in the log are not exactly matching the code on repo.

[SergiiDmytruk]

Adding __va() solves an issue with txt_restore_mtrrs().

I suggest hiding slaunch_slrt and exposing slr_get_table() instead, but this may be difficult for code in tpm.c because it needs to work both in early and normal Xen code.

It can't be hidden completely because head.S and efi-boot.h set it, but can likely be removed from the header. However, I don't think it's worth doing for v0.5.0-RC4 and risk breaking something. I'll do that for TrenchBoot/trenchboot-issues#46.

e820_change_range_type can fail in one of two ways: either the entry that covers the whole to-be-reserved region can't be found in E820, or reserving the region would cause it to be split into smaller ones, and there is no more space for additional mapping in the table.

There is a third case when that function fails: the region is already marked as reserved. And this is what happens for TXT heap:

(XEN)  [0000000000000000, 000000000009fbff] (usable)
(XEN)  [000000000009fc00, 000000000009ffff] (reserved)
(XEN)  [00000000000f0000, 00000000000fffff] (reserved)
(XEN)  [0000000000100000, 0000000099413fff] (usable)
(XEN)  [0000000099414000, 000000009941bfff] (reserved)
(XEN)  [000000009941c000, 0000000099439fff] (usable)
(XEN)  [000000009943a000, 000000009f7fffff] (reserved)
       [0000000099f10000, 0000000099ffffff] <<< TXT HEAP, included in the above entry
(XEN)  [00000000e0000000, 00000000efffffff] (reserved)
(XEN)  [00000000fc000000, 00000000fc000fff] (reserved)
(XEN)  [00000000fe000000, 00000000fe00ffff] (reserved)
(XEN)  [00000000fed10000, 00000000fed17fff] (reserved)
(XEN)  [00000000fed20000, 00000000fed91fff] (reserved)
(XEN)  [00000000feda0000, 00000000feda1fff] (reserved)
(XEN)  [00000000ff000000, 0000000100000fff] (reserved)
(XEN)  [0000000100001000, 000000045e7fffff] (usable)

In 3402356 I wrap the function to not fail in this case. I don't know if there is any value in having that region separately in e820 and whether this wrapper can be useful elsewhere, so limiting it to that one file.

Already checked manually that AEM works with these changes.

[krystian-hebel]

Already checked manually that AEM works with these changes.

On which platform and firmware type?

[SergiiDmytruk]

On which platform and firmware type?

VP4670 with Dasharo+SeaBIOS where it didn't work. openQA on vp4670-legacy branch failed with unrelated

# Test died: command 'grep "MB module$" /tmp/aem_event.log' failed at /usr/lib/os-autoinst/autotest.pm line 414.

because GRUB uses a different string (Measured MB2 module here) and I didn't bother to check that with the sources during review. That's at the end of the second boot, so the rest must work fine.

[krystian-hebel]

Tests on Optiplex also fail late because of Measured MB2 module (related issue, results in test number 3777 on our internal openQA server), but otherwise work. It took few attempts because CMOS IRQ handler messes up serial output by using ANSI escape codes to move the cursor, but that isn't related to AEM.

The problem from this issue has been resolved.

[SergiiDmytruk]

I think we need to merge the branch to close this: #24