Skip to content

fix(jaj): cap the PCIe FFC link at Gen2 - #120

Open
dakejahl wants to merge 1 commit into
mainfrom
fix/jaj-pcie-gen2
Open

fix(jaj): cap the PCIe FFC link at Gen2#120
dakejahl wants to merge 1 commit into
mainfrom
fix/jaj-pcie-gen2

Conversation

@dakejahl

@dakejahl dakejahl commented Aug 16, 2026

Copy link
Copy Markdown
Collaborator

Summary

Pins C7 — the controller behind JAJ's 16-pin PCIe FFC header — to 5 GT/s.

Problem

Schematic sheet 14 ("Pi 5 PCIE Connector") wires J7 with a single lane: PCIE2_CLK_P/N, PCIE2_RX0_P/N, PCIE2_TX0_P/N and nothing else. PCIE2_RX1/PCIE2_TX1 reach the SODIMM but are unrouted. So this is one lane over unshielded flat flex, on a pinout Raspberry Pi rates at 5 GT/s and documents 8 GT/s as opt-in and unguaranteed — over a run far shorter than the ~6" cable on the bench fixture.

Nothing pins the rate today. tegra234.dtsi gives C7 num-lanes = <8> and the stock p3768 carrier dtsi supplies two P2U phys without overriding it, so the link trains at 8 GT/s. On the fixture that produces a continuous correctable-AER storm — RxErr and BadDLLP at 210-240 events/s with the drive idle, summed over both ends of the link. Counted as one bit error apiece, which is the most generous reading, that is a BER near 3e-8: four to five orders of magnitude above the 1e-12 the Gen3 channel budget assumes. Measuring this off the console understates it by about 7x — the UART tops out at ~28 events/s and drops the rest — so take it from aer_dev_correctable, which increments whether or not the message reaches the wire. A PCIe link in L0 never goes quiet (logical-idle blocks plus periodic UpdateFC DLLPs), so that rate is the raw channel BER rather than anything about NVMe traffic. Every error is correctable and the link replays, but there is no margin left for temperature, voltage or cable flex.

The board contributes two more things at 8 GT/s. D23/D24/D25 put ESD arrays on all three high-speed pairs, which is shunt capacitance and a stub at 4 GHz Nyquist unless they are ultra-low-cap parts. AC coupling is fine and not a suspect: C171/C172 are 0.22 µF on the TX pair at the connector, inside the spec window.

Worth correcting a claim in ark_scripts#79 while this is in view: the x1 width is nominal, not a downgrade. The root port's x8 is the num-lanes default and the endpoint's x4 is the drive's own capability; neither describes what the FFC routes.

Solution

max-link-speed = <2> on pcie@141e0000 in the JAJ override. Halving the symbol rate drops Nyquist to 2.5 GHz, which buys a large amount of loss and jitter margin and cuts what the ESD capacitance costs — it aligns JAJ with the operating point the connector's originator actually certifies, rather than suppressing the symptom. Costs roughly 400 MB/s of ceiling on that link, ample for the FFC SSD's logging role. C1, C4 and C5 are untouched, so the M.2 Key-M slot keeps Gen3 x4.

Confirmed on the testlaptop9 fixture by the equivalent runtime retrain (setpci on Link Control 2): zero corrected errors over 90 s at 5 GT/s and over 60 s at 2.5 GT/s, against 209-229/s at 8 GT/s, and a control back at 8 GT/s brings them straight back. Numbers in the comment below. One gap the retrain cannot cover: it re-trains a link that already equalized at Gen3, so read current_link_speed and an aer_dev_correctable delta once on the first image built from this.

C7 drives the 16-pin FFC header, whose Raspberry Pi 5 pinout carries one
lane over unshielded flat flex. Raspberry Pi rates that connector at
5 GT/s and treats 8 GT/s as opt-in and unguaranteed; nothing in the stock
DT pins the rate, so the link trains at 8 GT/s with no margin and runs a
continuous correctable-AER storm.
@dakejahl

Copy link
Copy Markdown
Collaborator Author

Runtime confirmation on the testlaptop9 JAJ fixture. Forced with setpci -s 0007:00:00.0 CAP_EXP+30.w on the root port's Link Control 2 plus a retrain (CAP_EXP+10.w=20:20), console loglevel left alone so the before/after stays comparable. Error counts are TOTAL_ERR_COR deltas from aer_dev_correctable, root port and NVMe summed; loads are the 1-min average at the end of each window.

Target speed Corrected errors/s Console Idle load nvme1n1 seq read
8 GT/s 209 11427 B/s (99% of 11520) 3.3 752 MB/s
5 GT/s 0 over 90 s 0 0.7 399 MB/s
2.5 GT/s 0 over 60 s 0 0.2 205 MB/s
8 GT/s (control) 229 11422 B/s 2.8

The control is what rules out a lucky retrain. Reads are dd bs=1M count=2000 iflag=direct; the module M.2 (0004:*, Gen3 x4, same drive model) does 1.2 GB/s and this change does not touch it.

Three things the numbers add to the PR:

  • The x1 width is not a training downgrade, as the description says. It stays x1 at 5 GT/s and at 2.5 GT/s, so it tracks nothing about link margin.
  • Counting console lines understates the error rate by about 7x. 11427 B/s over ~400 B/event is 28.6 events/s, which is the UART's capacity rather than the channel's error rate, and the captures are full of ** N printk messages dropped **. Direct proof: with the console silenced (dmesg -n 4, 0 B/s on the wire) the counters still advanced at 241/s.
  • Gen1 is clean too, so Gen2 is not on a cliff edge — there is a step below if 5 GT/s ever proves marginal in the field.

Orthogonal, but it settles whether this and ark_jetson_kernel#118 are alternatives: with the link left at 8 GT/s and only the console quieted, flash_and_test_just_a_jetson.sh -j --skip-scan finishes in 133 s with 0 failures, against a bench mean of 145 s over 348 passing units. Loud, the same run never finished — it hung past 20 minutes in the jetson-io step with the board resetting under the systemd watchdog. So they fix different things and neither substitutes for the other: Gen2 is the right operating point for the header, and #118 is what keeps a link that degrades anyway showing up as a slow-but-passing run instead of a reset loop.

One consequence worth naming before this merges: afterwards nothing on a healthy unit will ever show anything about that link's margin. The correctable-AER count in the post-mortem digest on ark_scripts#79 becomes the only sentinel for it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant