Skip to content

fix(build): add quiet to the kernel command line - #118

Open
dakejahl wants to merge 4 commits into
mainfrom
fix/quiet-serial-console
Open

fix(build): add quiet to the kernel command line#118
dakejahl wants to merge 4 commits into
mainfrom
fix/quiet-serial-console

Conversation

@dakejahl

@dakejahl dakejahl commented Aug 16, 2026

Copy link
Copy Markdown
Collaborator

Summary

Quiets the kernel's serial console on PAB, JAJ and PAB_V3, and widens the ring buffer so a flood no longer costs us the boot record. Three parts: quiet and log_buf_len=4M on the extlinux APPEND, and a sysctl drop-in that keeps the console threshold down once userspace is running.

Problem

The BSP boots with CONFIG_CONSOLE_LOGLEVEL_DEFAULT=7 against console=ttyTCU0,115200. printk() to a serial console is synchronous — it busy-waits for the UART FIFO — so the kernel is hard-capped at 11520 B/s of log output, and anything logging faster throttles the entire system.

A marginal PCIe link makes that concrete. AER correctable errors (RxErr, BadDLLP) are harmless to data — the link replays and NVMe keeps working — but each event emits three log lines, ~400 bytes, so roughly 35 ms of blocking UART transmission apiece. On the bench that measured 11441 B/s against the 11520 B/s ceiling: the kernel ran at about 1/5 real time, idle load sat near 5, and RuntimeWatchdogSec=120 reset the board every couple of minutes because PID 1 could not pet the watchdog.

quiet on its own does not hold. L4T ships /etc/sysctl.d/30-nv-console-messages.confkernel.printk = 6 6 1 7, a nvidia-l4t-configs conffile — which systemd-sysctl applies a few seconds into userspace and which puts the console threshold back to 6. Measured on a JAJ fixture: with quiet on /proc/cmdline, /proc/sys/kernel/printk still read 6 6 1 7 and the boot console still carried 1.77 MB across 4289 AER lines. It only ever bought the pre-userspace window.

Suppression also does nothing for the ring buffer: every message still lands there, and at the 256K default the buffer wrapped inside a minute of boot — at 1 min uptime it spanned six seconds — so dmesg no longer reached the driver probe lines the fixture tests read back.

Solution

quiet sets the console threshold to CONSOLE_LOGLEVEL_QUIET (4). suppress_message_printing() drops a message when level >= console_loglevel, and correctable AER logs at KERN_WARNING (4) on every line — aer.c:687 and aer.c:719 — so all of it is suppressed. KERN_ERR and worse still reach the console, unlike loglevel=1, which would also hide uncorrectable AER, filesystem errors and OOM kills.

99-ark-console-quiet.conf carries the same threshold into userspace, winning on sort order rather than editing NVIDIA's conffile. Its value is stock Ubuntu's, from the 10-console-messages.conf that 30- overrode. With both in place the JAJ boot console went 1.77 MB → 52 kB, zero AER lines.

log_buf_len=4M keeps dmesg and journalctl -k usable through a flood, which is what the bench reads when a run needs explaining after the fact.

Raising the console baud rate would relieve the same pressure, but the bench depends on long FTDI runs at 115200.

Patched at the extlinux layer rather than CMDLINE_ADD so it can be flipped on a unit without a reflash. jetson-io copies the default entry's APPEND verbatim (extlinux.py:67, :100), so the entry it generates on the fixture inherits it. Runs outside the staging block and is idempotent, so --fast rebuilds pick it up too, and it aborts the build if the file no longer has exactly one active APPEND line.

printk to a serial console is synchronous, so on the BSP's verbose default
(CONSOLE_LOGLEVEL_DEFAULT=7) any driver logging above the 115200 UART's
~11.5 kB/s throttles the whole kernel. A PCIe AER correctable-error storm does
exactly that and starves systemd until RuntimeWatchdogSec resets the board.
quiet suppresses the AER lines while keeping KERN_ERR and worse, and leaves the
ring buffer untouched. Raising the console baud was the alternative, rejected to
keep long FTDI runs on the bench reliable.
… ring buffer

quiet on the command line only survives until systemd-sysctl runs: L4T ships
/etc/sysctl.d/30-nv-console-messages.conf (kernel.printk = 6 6 1 7, a
nvidia-l4t-configs conffile) which puts the console threshold back to 6 a few
seconds into userspace, and the KERN_WARNING flood this was meant to stop back
on the wire. Measured on a JAJ fixture: quiet alone still gave a 1.77 MB boot
console with 4289 AER lines, so it only ever bought the pre-userspace window.
A 99- drop-in wins on sort order without touching NVIDIA's conffile, and the
value is the stock Ubuntu one that 30- overrode.

log_buf_len=4M is the other half. Suppressing a message keeps it off the UART
but not out of the ring buffer, which at the 256K default wrapped inside a
minute of boot - dmesg stopped reaching back to the driver probe lines the
fixture tests read, which is what pushed those checks onto sysfs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant