fix(build): add quiet to the kernel command line - #118
Open
dakejahl wants to merge 4 commits into
Open
Conversation
printk to a serial console is synchronous, so on the BSP's verbose default (CONSOLE_LOGLEVEL_DEFAULT=7) any driver logging above the 115200 UART's ~11.5 kB/s throttles the whole kernel. A PCIe AER correctable-error storm does exactly that and starves systemd until RuntimeWatchdogSec resets the board. quiet suppresses the AER lines while keeping KERN_ERR and worse, and leaves the ring buffer untouched. Raising the console baud was the alternative, rejected to keep long FTDI runs on the bench reliable.
… ring buffer quiet on the command line only survives until systemd-sysctl runs: L4T ships /etc/sysctl.d/30-nv-console-messages.conf (kernel.printk = 6 6 1 7, a nvidia-l4t-configs conffile) which puts the console threshold back to 6 a few seconds into userspace, and the KERN_WARNING flood this was meant to stop back on the wire. Measured on a JAJ fixture: quiet alone still gave a 1.77 MB boot console with 4289 AER lines, so it only ever bought the pre-userspace window. A 99- drop-in wins on sort order without touching NVIDIA's conffile, and the value is the stock Ubuntu one that 30- overrode. log_buf_len=4M is the other half. Suppressing a message keeps it off the UART but not out of the ring buffer, which at the 256K default wrapped inside a minute of boot - dmesg stopped reaching back to the driver probe lines the fixture tests read, which is what pushed those checks onto sysfs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Quiets the kernel's serial console on PAB, JAJ and PAB_V3, and widens the ring buffer so a flood no longer costs us the boot record. Three parts:
quietandlog_buf_len=4Mon the extlinuxAPPEND, and a sysctl drop-in that keeps the console threshold down once userspace is running.Problem
The BSP boots with
CONFIG_CONSOLE_LOGLEVEL_DEFAULT=7againstconsole=ttyTCU0,115200.printk()to a serial console is synchronous — it busy-waits for the UART FIFO — so the kernel is hard-capped at 11520 B/s of log output, and anything logging faster throttles the entire system.A marginal PCIe link makes that concrete. AER correctable errors (
RxErr,BadDLLP) are harmless to data — the link replays and NVMe keeps working — but each event emits three log lines, ~400 bytes, so roughly 35 ms of blocking UART transmission apiece. On the bench that measured 11441 B/s against the 11520 B/s ceiling: the kernel ran at about 1/5 real time, idle load sat near 5, andRuntimeWatchdogSec=120reset the board every couple of minutes because PID 1 could not pet the watchdog.quieton its own does not hold. L4T ships/etc/sysctl.d/30-nv-console-messages.conf—kernel.printk = 6 6 1 7, anvidia-l4t-configsconffile — whichsystemd-sysctlapplies a few seconds into userspace and which puts the console threshold back to 6. Measured on a JAJ fixture: withquieton/proc/cmdline,/proc/sys/kernel/printkstill read6 6 1 7and the boot console still carried 1.77 MB across 4289 AER lines. It only ever bought the pre-userspace window.Suppression also does nothing for the ring buffer: every message still lands there, and at the 256K default the buffer wrapped inside a minute of boot — at 1 min uptime it spanned six seconds — so
dmesgno longer reached the driver probe lines the fixture tests read back.Solution
quietsets the console threshold toCONSOLE_LOGLEVEL_QUIET(4).suppress_message_printing()drops a message whenlevel >= console_loglevel, and correctable AER logs atKERN_WARNING(4) on every line —aer.c:687andaer.c:719— so all of it is suppressed.KERN_ERRand worse still reach the console, unlikeloglevel=1, which would also hide uncorrectable AER, filesystem errors and OOM kills.99-ark-console-quiet.confcarries the same threshold into userspace, winning on sort order rather than editing NVIDIA's conffile. Its value is stock Ubuntu's, from the10-console-messages.confthat 30- overrode. With both in place the JAJ boot console went 1.77 MB → 52 kB, zero AER lines.log_buf_len=4Mkeepsdmesgandjournalctl -kusable through a flood, which is what the bench reads when a run needs explaining after the fact.Raising the console baud rate would relieve the same pressure, but the bench depends on long FTDI runs at 115200.
Patched at the extlinux layer rather than
CMDLINE_ADDso it can be flipped on a unit without a reflash. jetson-io copies the default entry'sAPPENDverbatim (extlinux.py:67,:100), so the entry it generates on the fixture inherits it. Runs outside the staging block and is idempotent, so--fastrebuilds pick it up too, and it aborts the build if the file no longer has exactly one activeAPPENDline.