ninja install fails with install_megadrivers.py error

[emulti]

I'm trying to update my Toshiba AC100 with the latest grate-driver (kernel 5.7.8, Arch Linux Arm) using the instructions on https://github.com/grate-driver/grate/wiki/Grate-driver:

Building natively on the AC100.

1.  git clone https://github.com/grate-driver/mesa.git
2.     cd mesa
3.     meson -Dprefix=/usr -Dgallium-drivers=grate -Ddri-drivers=swrast -Dplatforms=x11,drm -Dshared-glapi=true -Dgbm=true -Dglx=dri -Dosmesa=none -Dgles1=false -Dgles2=true -Degl=true -Dgallium-xa=false -Dgallium-vdpau=false -Dgallium-va=false -Dgallium-xvmc=false -Duse-elf-tls=false -Dgallium-nine=false -Db_ndebug=true -Dvulkan-drivers= -Dlibunwind=false -Dllvm=false build
4.     cd build/
5.     ninja && ninja install

Step 3 failed at first with 'no choice for 'grate'' error. I added 'grate' to the array under gallium-drivers in meson_option.txt and was then able to build mesa master branch with ninja in the first part of step 5.

However, the 'sudo ninja install' command fails:

[1/15] Generating git_sha1.h with a custom command
[1/2] Installing files.
installing /home/chris/Downloads/build/mesa/build/src/mesa/drivers/dri/libmesa_dri_drivers.so to /usr/lib/dri/swrast_dri.so
Installing src/mapi/shared-glapi/libglapi.so.0.0.0 to /usr/lib
Installing src/mapi/es2api/libGLESv2.so.2.0.0 to /usr/lib
Installing src/mesa/drivers/dri/libmesa_dri_drivers.so to /usr/lib/dri
Installing src/glx/libGL.so.1.2.0 to /usr/lib
Installing src/gbm/libgbm.so.1.0.0 to /usr/lib
Installing src/egl/libEGL.so.1.0.0 to /usr/lib
Installing src/gallium/targets/dri/libgallium_dri.so to /usr/lib/dri
Installing /home/chris/Downloads/build/mesa/include/KHR/khrplatform.h to /usr/include/KHR
Installing /home/chris/Downloads/build/mesa/include/GLES2/gl2.h to /usr/include/GLES2
Installing /home/chris/Downloads/build/mesa/include/GLES2/gl2ext.h to /usr/include/GLES2
Installing /home/chris/Downloads/build/mesa/include/GLES2/gl2platform.h to /usr/include/GLES2
Installing /home/chris/Downloads/build/mesa/include/GLES3/gl3.h to /usr/include/GLES3
Installing /home/chris/Downloads/build/mesa/include/GLES3/gl31.h to /usr/include/GLES3
Installing /home/chris/Downloads/build/mesa/include/GLES3/gl32.h to /usr/include/GLES3
Installing /home/chris/Downloads/build/mesa/include/GLES3/gl3ext.h to /usr/include/GLES3
Installing /home/chris/Downloads/build/mesa/include/GLES3/gl3platform.h to /usr/include/GLES3
Installing /home/chris/Downloads/build/mesa/include/GL/gl.h to /usr/include/GL
Installing /home/chris/Downloads/build/mesa/include/GL/glcorearb.h to /usr/include/GL
Installing /home/chris/Downloads/build/mesa/include/GL/glext.h to /usr/include/GL
Installing /home/chris/Downloads/build/mesa/include/GL/glx.h to /usr/include/GL
Installing /home/chris/Downloads/build/mesa/include/GL/glxext.h to /usr/include/GL
Installing /home/chris/Downloads/build/mesa/include/EGL/egl.h to /usr/include/EGL
Installing /home/chris/Downloads/build/mesa/include/EGL/eglext.h to /usr/include/EGL
Installing /home/chris/Downloads/build/mesa/include/EGL/eglplatform.h to /usr/include/EGL
Installing /home/chris/Downloads/build/mesa/include/EGL/eglmesaext.h to /usr/include/EGL
Installing /home/chris/Downloads/build/mesa/include/EGL/eglextchromium.h to /usr/include/EGL
Installing /home/chris/Downloads/build/mesa/include/GL/internal/dri_interface.h to /usr/include/GL/internal
Installing /home/chris/Downloads/build/mesa/src/gbm/main/gbm.h to /usr/include
Installing /home/chris/Downloads/build/mesa/src/util/00-mesa-defaults.conf to /usr/share/drirc.d
Installing /home/chris/Downloads/build/mesa/build/meson-private/glesv2.pc to /usr/lib/pkgconfig
Installing /home/chris/Downloads/build/mesa/build/meson-private/dri.pc to /usr/lib/pkgconfig
Installing /home/chris/Downloads/build/mesa/build/meson-private/gbm.pc to /usr/lib/pkgconfig
Installing /home/chris/Downloads/build/mesa/build/meson-private/egl.pc to /usr/lib/pkgconfig
Installing /home/chris/Downloads/build/mesa/build/meson-private/gl.pc to /usr/lib/pkgconfig
Running custom install script '/usr/bin/python /home/chris/Downloads/build/mesa/bin/install_megadrivers.py /home/chris/Downloads/build/mesa/build/src/mesa/drivers/dri/libmesa_dri_drivers.so /usr/lib/dri swrast_dri.so'
Running custom install script '/usr/bin/python /home/chris/Downloads/build/mesa/bin/install_megadrivers.py /home/chris/Downloads/build/mesa/build/src/gallium/targets/dri/libgallium_dri.so /usr/lib/dri'
FAILED: meson-install
/usr/bin/meson install --no-rebuild
ninja: build stopped: subcommand failed.

It looks like there is an argument missing to the install_megadrivers.py script for the libgallium_dri.so file.
Can you offer advice on how to fix this error please? I am new to the Meson build system, so don't know where the install script is taking values from.

[kusma]

Sounds like you're trying to use the wrong Mesa branch. We don't have a grate driver upstream, only our own fork. And that driver isn't useful for much more than running glxgears.

[emulti]

Thanks, seems it's the Mesa 19.3 branch that contains the grate driver. I will try and build that one.
The last build I did was 13 May 2019 when Autotools were still used.
I am happy to help with testing if there is interest. The AC100 is a nice little device though constrained with RAM and storage.

[digetx]

The 19.3 is the most actual branch and indeed you could only run glxgears using the current Mesa driver. Besides 3D, Mesa is also useful for the libvdpau-tegra because libvdpau core uses DRI for retrieving the VDPAU driver name, otherwise you'll need to manually specify the driver name in environment variables.

Help with the testing is very appreciated! And you could do quite a lot things on AC100 without a 3D driver!

All mobile devices are very resource-constrained and this is a big part of the development fun to optimize everything :)

[emulti]

Thanks for the info. I can confirm the 19.3 branch of Mesa builds fine, along with the other components including libvdpau-tegra. Maybe put a note on the wiki page to use 193. branch rather than Master?
I also built the libraries in 'Grate' which get installed in /usr/local/lib by default. Is there an associated test application?
The AC100 hardware is very nice, well balanced with a good keyboard. It runs nicely with i3 and lightweight apps like Sylpheed, Gnumeric, Abiword and company, and I got rid of a lot of bloat to make space on the EMMC.
There are some kernel oops during boot associated with clk.c and the nvec keyboard/touchpad doesn't always work on every boot. Nothing to do with Xorg/Mesa of course, but do you know where bug reports can be submitted?

[digetx]

I guess it should be better to replace the master branch with 19.3. Actually, I was going to update the master sometime ago, but haven't got to it yet. It will be fixed sometime soon, thank you for getting attention to it!

The 'Grate' test applications aren't installed, but you could run them manually by executing tests/grate/* tests/host1x/*. Also, there is no need to install the libraries because they are not used anywhere. There was intention to utilize the libgrate in the past, but then plans changed.

The vanilla upstream 5.7 kernel is known to produce the clk warnings, they are harmless and eventually will be fixed once patches will be backported from 5.8.

The NVEC driver isn't actively maintained, so should be more productive if you could submit patches instead of the bug reports :)

[emulti]

Thanks again. I built the grate-driver kernel (5.8.0-rc4...) and that doesn't have the problem with clk.c oops or the nvec failing to reset/touchpad freezing. I also rebuilt the libdrm, xf86-opentegra, mesa and libvdpau-grate packages.
While I would be happy to submit patches sadly it's beyond my technical capability.

However, I did find an issue with the 5.8.0-rc4 kernel on AC100.
It is not present with mainline 5.7.8.

After a cold start from power off, Xorg fails to start with repeated messages:
Jul 18 09:22:23 alarm kernel: [drm] tegra_drm_sched_timedout_job: 3d channel: pipes 0x2 (process:Xorg pid:616)
Jul 18 09:22:23 alarm kernel: [drm:tegra_drm_sched_timedout_job] ERROR 3d channel: pipes 0x2 (process:Xorg pid:616)
Jul 18 09:22:23 alarm kernel: tegra-gr3d 54180000.gr3d: [drm:tegra_drm_sched_timedout_job] resetting hardware
Jul 18 09:22:23 alarm kernel: [drm] tegra_drm_sched_timedout_job: 3d channel: pipes 0x2 (process:Xorg pid:616)
Jul 18 09:22:23 alarm kernel: [drm:tegra_drm_sched_timedout_job] ERROR 3d channel: pipes 0x2 (process:Xorg pid:616)
Jul 18 09:22:23 alarm kernel: tegra-gr3d 54180000.gr3d: [drm:tegra_drm_sched_timedout_job] resetting hardware
Jul 18 09:22:24 alarm kernel: [drm] tegra_drm_sched_timedout_job: 2d channel: pipes 0x1 (process:Xorg pid:616)
Jul 18 09:22:24 alarm kernel: [drm:tegra_drm_sched_timedout_job] ERROR 2d channel: pipes 0x1 (process:Xorg pid:616)
Jul 18 09:22:24 alarm kernel: tegra-gr2d 54140000.gr2d: [drm:tegra_drm_sched_timedout_job] resetting hardware
Jul 18 09:22:24 alarm kernel: [drm] tegra_drm_sched_timedout_job: 2d channel: pipes 0x1 (process:Xorg pid:616)
Jul 18 09:22:24 alarm kernel: [drm:tegra_drm_sched_timedout_job] ERROR 2d channel: pipes 0x1 (process:Xorg pid:616)
Jul 18 09:22:24 alarm kernel: tegra-gr2d 54140000.gr2d: [drm:tegra_drm_sched_timedout_job] resetting hardware
Jul 18 09:22:24 alarm kernel: [drm] tegra_drm_sched_timedout_job: 2d channel: pipes 0x1 (process:Xorg pid:616)

In the dmesg in this state there is this message:
tegra-mc 7000f000.memory-controller: host1xdmar: DMA blocked
tegra-mc 7000f000.memory-controller: host1xdmar: read @0xb565fa30: EMEM address decode error (EMEM decode error)

After a warm reboot, Xorg starts fine. The issue only happens when starting from power off. I am wondering if it is related to how u-boot (2013-07) initialises hardware, and if it is related to the inability to power off completely with poweroff command in recent kernels.

On a separate question, I tried running the tests in grate and host1x.
Most seem (gr3d) seem to fail with
INFO: x11_overlay_create:39 overlay unsupported
ERROR: grate_overlay_create: host1x_overlay_create() failed: -1

dmesg-cold.txt
dmesg-warm.txt

Xorg.0.cold.log
Xorg.0.warm.log

[digetx]

Thank you very much for the report!

Could you please try the recent grate-kernel update? I added this change grate-driver/linux@15996b0. It should be a kernel driver problem which pops up only if hardware is in a certain state during boot. This problem was already reported for AC100 not so long time ago and I couldn't reproduce it.

Host1x driver doesn't support recovering from a blocked DMA. So hardware is getting a reset, but DMA stays blocked until the warm reboot, hence that's why it works after warm reboot.

The overlay unsupported message and further error are supposed to happen when running tests under Xorg, you may ignore it. Could you please clarify what do you mean by seem to fail, do you get any other errors?

[emulti]

I'll update the kernel on AC100 and test again tonight if possible.
I should not have said 'seem', the following are results, running from the grate folder with tests/xxx/yyy so the files in asm are found:
host1x:
gr2d-blit: INFO: main:175: test passed
gr2d-clear: -displays magenta window, and 'overlay support missing'
gr2d-context: - see attachment
gr3d-triangle - displays gamut triangle, then magenta window, and 'overlay support missing'

grate:
clear: magenta window, ERROR: grate_overlay_create: host1x_overlay_create() failed: -1
cube, cube-textured: 'CgDrv_Create: BLOB compiler is unavailable' (x2) 'grate_program_new() failed'
cube-textured2: displays jailbars scene keyed over floating cubes behind
cube-textured3 fails after loading to '24%' with ERROR: host1x bo_create_helper:237: host1x_bo_create failed; ERROR:grate_create_texture: failed to allocate texture 2048x2048 bpp:8 pitch:4096; Segmentation fault (AC100 display too small?)
interactive: floating cube responds correctly to keyboard commands, in 'face cull mode' the text on cube is mirrored except in 'none', can be flipped with '2' key, front face direction. '3' depth function blanks cube in some modes, maybe as intended.
quad: 'CgDrv_Create: BLOB compiler is unavailable' (x2) 'grate_program_new() failed'
stencil: "Stencil test works!"
texture-filter: works, stepping pixelation on cube
texture-wrap: runs, not sure what is expected behaviour!
triangle: 'CgDrv_Create: BLOB compiler is unavailable' (x2) 'grate_program_new() failed'
triangle-rotate: 'CgDrv_Create: BLOB compiler is unavailable' (x2) 'grate_program_new() failed'
gr2-context.txt

[digetx]

All results look good! Please ignore the failed tests because they depend on extra bits, and thus, expected to fail in a default userspace/kernel configuration.

The cube-textured3 requires a lot of free contiguous memory, you may try to get it working by adding cma=128M (maybe even 256M) into the kernel's cmdline arguments.

[emulti]

Does allocating larger cma reduce the amount of memory for non-gpu-related applications? I was thinking of trying reducing it from the current 64M if it does, memory is short on ac100.

I built the linux-grate 5.8.0-rc5-g81e240239be6 (after yesterday's commits)
Unfortunately on cold boot it hangs, before systemd journaling starts, and I don't have a serial console connection to capture the issue. But we probably know what it's from...
dmesg-5.8.0-rc5.txt

On warm boot there is an issue with a duplicate regulator name regulator.5:cpu0 (from the device tree, maybe cpufreq/DVFS related?) as in the attached dmesg.
Strange, Device tree binary is different size but there were no new dts commits I have found, maybe I am looking in the wrong place.

[digetx]

Larger CMA shouldn't reduce the amount of memory. It's a reusable (system) memory that could be swapped out during of contiguous allocation, it's not a carveout.

Hmm.. now I'm also seeing that there is some kernel problem using next-20200717:

 BUG: sleeping function called from invalid context at kernel/locking/mutex.c:281
 in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 12, name: kworker/0:1
 CPU: 0 PID: 12 Comm: kworker/0:1 Not tainted 5.8.0-rc5-next-20200717-00162-g4bcedc60754a #2833
 Hardware name: NVIDIA Tegra SoC (Flattened Device Tree)
 Workqueue: rcu_gp srcu_invoke_callbacks
 [<c010dad5>] (unwind_backtrace) from [<c0109481>] (show_stack+0x11/0x14)
 [<c0109481>] (show_stack) from [<c0469da9>] (dump_stack+0x8d/0x9c)
 [<c0469da9>] (dump_stack) from [<c013e67d>] (___might_sleep+0xed/0x11c)
 [<c013e67d>] (___might_sleep) from [<c09b8d6d>] (mutex_lock+0x1d/0x54)
 [<c09b8d6d>] (mutex_lock) from [<c055bd03>] (device_del+0x2b/0x2a4)
 [<c055bd03>] (device_del) from [<c055bfd9>] (__device_link_free_srcu+0x41/0x50)
 [<c055bfd9>] (__device_link_free_srcu) from [<c017037f>] (srcu_invoke_callbacks+0x9b/0x100)
 [<c017037f>] (srcu_invoke_callbacks) from [<c0133dbd>] (process_one_work+0x145/0x408)
 [<c0133dbd>] (process_one_work) from [<c0134179>] (worker_thread+0xf9/0x3c4)
 [<c0134179>] (worker_thread) from [<c0138ef3>] (kthread+0x10b/0x13c)
 [<c0138ef3>] (kthread) from [<c010015d>] (ret_from_fork+0x11/0x34)
 Exception stack(0xef159fb0 to 0xef159ff8)

It probably has the same root as yours regulator issue. I'll check if today's next still has that issue and will ping you once the problem will be resolved.

[digetx]

Please give a try to the recent grate-kernel update, I reverted the offending commits.

[emulti]

Yes, the reverted patches (5.8.0-rc6...) put things back as they were (host1xdmar: DMA blocked on first cold boot).
In testing with the previous kernel 5.8.0-rc5... I found that the results are not consistent from cold boot. It depends whether the system was powered off by software (holding down power button after 'poweroff', because the system is in unresponsive state after with power LED still on), or by removing the battery. So the condition of the EC varies.
This incomplete powerdown issue has been reported on kernels after at least 5.4, but is not present on 5.1.1 which I have used previously and tested again today. Perhaps this is linked to the incorrect initialisation of host1x. I will try and find out what changed in the nvec code between 5.1 and 5.4, but if doesn't look like much.
The issue 'sysfs: cannot create duplicate filename '/devices/virtual/devlink/regulator.5:cpu0' is not present any more in rc6. I changed the nvec driver to built-in rather than an module, it is loaded much earlier. uvcvideo driver is also now compiled-in.

[digetx]

Thank you for the report! I'll add some debug messages to the host1x driver that may help to figure out what's wrong, will ping you once debugging will be ready for the testing.

[digetx]

@emulti Could you please fetch a recent grate-kernel update and post kernel boot log? I added some debug messages which may shed some light on the host1x problem.