sidecar thermal shutdown on mfg line

[nathanaelhuffman]

I got called to debug a system that had a sidecar in A2 after mupdate to 17.2.

 humility -a /data/local/images/sidecar/d/sp/build-sidecar-d-image-default-v1.0.56.zip --ip  fe80::aa40:25ff:fe05:8e00%dut2 ringbuf seq
humility: connecting to fe80::aa40:25ff:fe05:8e00%5
humility: ring buffer drv_oxide_vpd::__RINGBUF in sequencer:
humility: ring buffer drv_packrat_vpd_loader::__RINGBUF in sequencer:
humility: ring buffer drv_sidecar_seq_server::__RINGBUF in sequencer:
 NDX LINE      GEN    COUNT PAYLOAD
  17  368      406        1 TofinoSequencerPolicyUpdate(Disabled)
  18  350      406        1 TofinoSequencerTick(Disabled, A2 { error: None })
  19  368      406        1 TofinoSequencerPolicyUpdate(Disabled)
  20  350      406        1 TofinoSequencerTick(Disabled, A2 { error: None })
  21  368      406        1 TofinoSequencerPolicyUpdate(Disabled)
  22  350      406        1 TofinoSequencerTick(Disabled, A2 { error: None })
  23  368      406        1 TofinoSequencerPolicyUpdate(Disabled)
  24  350      406        1 TofinoSequencerTick(Disabled, A2 { error: None })
  25  368      406        1 TofinoSequencerPolicyUpdate(Disabled)
  26  350      406        1 TofinoSequencerTick(Disabled, A2 { error: None })
  27  368      406        1 TofinoSequencerPolicyUpdate(Disabled)
  28  350      406        1 TofinoSequencerTick(Disabled, A2 { error: None })
  29  368      406        1 TofinoSequencerPolicyUpdate(Disabled)
  30  350      406        1 TofinoSequencerTick(Disabled, A2 { error: None })
  31  368      406        1 TofinoSequencerPolicyUpdate(Disabled)
   0  350      407        1 TofinoSequencerTick(Disabled, A2 { error: None })
   1  368      407        1 TofinoSequencerPolicyUpdate(Disabled)
   2  350      407        1 TofinoSequencerTick(Disabled, A2 { error: None })
   3  368      407        1 TofinoSequencerPolicyUpdate(Disabled)
   4  350      407        1 TofinoSequencerTick(Disabled, A2 { error: None })
   5  368      407        1 TofinoSequencerPolicyUpdate(Disabled)
   6  350      407        1 TofinoSequencerTick(Disabled, A2 { error: None })
   7  368      407        1 TofinoSequencerPolicyUpdate(Disabled)
   8  350      407        1 TofinoSequencerTick(Disabled, A2 { error: None })
   9  368      407        1 TofinoSequencerPolicyUpdate(Disabled)
  10  350      407        1 TofinoSequencerTick(Disabled, A2 { error: None })
  11  368      407        1 TofinoSequencerPolicyUpdate(Disabled)
  12  350      407        1 TofinoSequencerTick(Disabled, A2 { error: None })
  13  368      407        1 TofinoSequencerPolicyUpdate(Disabled)
  14  350      407        1 TofinoSequencerTick(Disabled, A2 { error: None })
  15  368      407        1 TofinoSequencerPolicyUpdate(Disabled)
  16  350      407        1 TofinoSequencerTick(Disabled, A2 { error: None })

Looks liked a commanded power off of some kind, let's check thermal

humility -a /data/local/images/sidecar/d/sp/build-sidecar-d-image-default-v1.0.56.zip --ip  fe80::aa40:25ff:fe05:8e00%dut2 ringbuf thermal
humility: connecting to fe80::aa40:25ff:fe05:8e00%5
humility: ring buffer drv_i2c_devices::emc2305::__RINGBUF in thermal:
humility: ring buffer drv_i2c_devices::max31790::__RINGBUF in thermal:
humility: ring buffer task_thermal::__RINGBUF in thermal:
   TOTAL VARIANT
    6678 PowerDownAt
      97 ControlPwm
      11 AutoState(Boot)
       2 AutoState(Running)
       2 AutoState(Uncontrollable)
       1 AutoState(Overheated)
       8 FanAdded
       8 AddedDynamicInput
       2 PowerDownDueTo
       2 PowerModeChanged
       2 FanControllerInitialized
       1 Start
       1 ThermalMode(Auto)
       1 CriticalDueTo
       1 SetFanWatchdogOk
 NDX LINE      GEN    COUNT PAYLOAD
   8 1210      211        1 PowerDownAt(0x66e840)
   9 1210      211        1 PowerDownAt(0x66ec28)
  10 1210      211        1 PowerDownAt(0x66f010)
  11 1210      211        1 PowerDownAt(0x66f3f8)
  12 1210      211        1 PowerDownAt(0x66f7e8)
  13 1210      211        1 PowerDownAt(0x66fbc8)
  14 1210      211        1 PowerDownAt(0x66ffb0)
  15 1210      211        1 PowerDownAt(0x670398)
  16 1210      211        1 PowerDownAt(0x670780)
  17 1210      211        1 PowerDownAt(0x670b68)
  18 1210      211        1 PowerDownAt(0x670f50)
  19 1210      211        1 PowerDownAt(0x671338)
  20 1210      211        1 PowerDownAt(0x671720)
  21 1210      211        1 PowerDownAt(0x671b08)
  22 1210      211        1 PowerDownAt(0x671ef0)
  23 1210      211        1 PowerDownAt(0x6722d8)
  24 1210      211        1 PowerDownAt(0x6726c0)
  25 1210      211        1 PowerDownAt(0x672aa8)
  26 1210      211        1 PowerDownAt(0x672e90)
  27 1210      211        1 PowerDownAt(0x673278)
  28 1210      211        1 PowerDownAt(0x673660)
  29 1210      211        1 PowerDownAt(0x673a48)
  30 1210      211        1 PowerDownAt(0x673e30)
  31 1210      211        1 PowerDownAt(0x674218)
   0 1210      212        1 PowerDownAt(0x674600)
   1 1210      212        1 PowerDownAt(0x6749e8)
   2 1210      212        1 PowerDownAt(0x674dd0)
   3 1210      212        1 PowerDownAt(0x6751b8)
   4 1210      212        1 PowerDownAt(0x6755a0)
   5 1210      212        1 PowerDownAt(0x675988)
   6 1210      212        1 PowerDownAt(0x675d70)
   7 1210      212        1 PowerDownAt(0x676158)

This looks like a thermal shutdown. 2 things of note here: the repeated PowerDownAt seems like a bug, and it pushes the other more useful stuff out of the ring buffer. We did have a CriticalDueTo logged in history so I'm postulating that this was due to a xcvr read error. Additional attempts to go back to A0 did not work, as we were immediately shut down again:

humility -a /data/local/images/sidecar/d/sp/build-sidecar-d-image-default-v1.0.56.zip --ip  fe80::aa40:25ff:fe05:8e00%dut2 ringbuf seq
humility: connecting to fe80::aa40:25ff:fe05:8e00%5
humility: ring buffer drv_oxide_vpd::__RINGBUF in sequencer:
humility: ring buffer drv_packrat_vpd_loader::__RINGBUF in sequencer:
humility: ring buffer drv_sidecar_seq_server::__RINGBUF in sequencer:
 NDX LINE      GEN    COUNT PAYLOAD
  17  368      409        1 TofinoSequencerPolicyUpdate(Disabled)
  18  350      409        1 TofinoSequencerTick(Disabled, A2 { error: None })
  19  368      409        1 TofinoSequencerPolicyUpdate(Disabled)
  20  350      409        1 TofinoSequencerTick(Disabled, A2 { error: None })
  21  368      409        1 TofinoSequencerPolicyUpdate(Disabled)
  22  350      409        1 TofinoSequencerTick(Disabled, A2 { error: None })
  23  368      409        1 TofinoSequencerPolicyUpdate(Disabled)
  24  350      409        1 TofinoSequencerTick(Disabled, A2 { error: None })
  25  368      409        1 TofinoSequencerPolicyUpdate(Disabled)
  26  350      409        1 TofinoSequencerTick(Disabled, A2 { error: None })
  27  368      409        1 TofinoSequencerPolicyUpdate(Disabled)
  28  350      409        1 TofinoSequencerTick(Disabled, A2 { error: None })
  29  368      409        1 TofinoSequencerPolicyUpdate(Disabled)
  30  350      409        1 TofinoSequencerTick(Disabled, A2 { error: None })
  31  368      409        1 TofinoSequencerPolicyUpdate(Disabled)
   0  350      410        1 TofinoSequencerTick(Disabled, A2 { error: None })
   1  368      410        1 TofinoSequencerPolicyUpdate(Disabled)
   2  350      410        1 TofinoSequencerTick(Disabled, A2 { error: None })
   3  368      410        1 TofinoSequencerPolicyUpdate(Disabled)
   4  350      410        1 TofinoSequencerTick(Disabled, A2 { error: None })
   5  368      410        1 TofinoSequencerPolicyUpdate(Disabled)
   6  350      410        1 TofinoSequencerTick(Disabled, A2 { error: None })
   7  368      410        1 TofinoSequencerPolicyUpdate(Disabled)
   8  350      410        1 TofinoSequencerTick(Disabled, A2 { error: None })
   9  403      410        1 ClearingTofinoSequencerFault(None)
  10  368      410        1 TofinoSequencerPolicyUpdate(LatchOffOnFault)
  11  368      410        1 TofinoSequencerPolicyUpdate(Disabled)
  12  350      410        1 TofinoSequencerTick(Disabled, A2 { error: None })
  13  368      410        1 TofinoSequencerPolicyUpdate(Disabled)
  14  350      410        1 TofinoSequencerTick(Disabled, A2 { error: None })
  15  368      410        1 TofinoSequencerPolicyUpdate(Disabled)
  16  350      410        1 TofinoSequencerTick(Disabled, A2 { error: None })

I took a hubris dump for further debug stored at /staff/core/hubris-2369

and proceed to ignition cycle the sidecar at which point it came back online with no problem.

Some thoughts:

1. This shouldn't be so sticky as to require an ignition cycle.
2. Spamming the ring buf with PowerDownAt and a new timestamp is really unhelpful
3. If this is a xcvr temp issue as suspected, it is most probably incorrect behavior to totally shut down the switch!

[hawkw]

• Spamming the ring buf with PowerDownAt and a new timestamp is really unhelpful

Agreed, that's really unfortunate. Seems that it will just keep doing that every time the thermal loop's timer fires and it remains in Uncontrollable:

hubris/task/thermal/src/control.rs

Lines 1346 to 1363 in 7cb5e03

	ThermalControlState::Uncontrollable => ControlResult::PowerDown,
	};
	match control_result {
	ControlResult::Pwm(target_pwm) => {
	// Send the new RPM to all of our fans
	ringbuf_entry!(Trace::ControlPwm(target_pwm.0));
	self.set_pwm(target_pwm)?;
	}
	ControlResult::PowerDown => {
	ringbuf_entry!(Trace::PowerDownAt(sys_get_timer().now));
	*self.prev_err_blackbox = *self.err_blackbox;
	self.err_blackbox.clear();
	if let Err(e) = self.bsp.power_down() {
	ringbuf_entry!(Trace::PowerDownFailed(e));
	}
	self.set_pwm(PWMDuty(0))?;
	}

What I'm a bit surprised by is that we didn't see it keep spamming PowerDownAt while debugging thermal shutdowns on Cosmo last week. I think that's because when a Cosmo shut itself down,, this check triggers:

hubris/task/thermal/src/control.rs

Lines 1097 to 1102 in 7cb5e03

	let prev_power_mode = self.power_mode;
	self.power_mode = self.bsp.power_mode();
	if prev_power_mode != self.power_mode {
	ringbuf_entry!(Trace::PowerModeChanged(self.power_mode));
	self.reset_state();
	}

because we transitioned from A0 to A2. I wonder if that didn't happen here because the system was already in A2 when it "overheated" due to the hot transceiver?

• If this is a xcvr temp issue as suspected, it is most probably incorrect behavior to totally shut down the switch!

Yeah, that's real bad. We already have a ticket tracking that (#2355), but if it's happened again, we should probably bump the priority up on fixing it. Unfortunately the thermal loop will probably need to change substantially to fix that.

[twinfees]

Not sure if same issue was hit, but had another sidecar (sw0) in A2 after mfg mupdate to 17.2 (related to https://github.com/oxidecomputer/mfg-troubleshooting/issues/378).

Captured a dump and also added it to /staff/core/hubris-2369 (filename: hubris.core.0-mfg378).

Figured I'd update here first, but if this is a different problem, happy to move to a new issue.

[Aaron-Hartwig]

@hawkw per our convo yesterday, here are the concrete temperature details. This is all in line with what we talked about.

QSFP modules are specified by SFF-8679 QSFP+ 4x Hardware and Electrical Specification. Per Rev 1.9 Section 8.1 defines the case temperature range as 0 to 70C for Standard class devices. This is obviously wider for the Extended and Industrial classes, but we should try for the most stringent.

Lower speed modules (generally 100G and slower) have a SFF-8636 compliant management interface. Per Rev 2.11 Section 6.2.4 reports that the reported temperatures are internally measured.

Higher speed modules (generally 200G and higher) have a CMIS compliant management interface. Per Rev 5.3 Section 8.2.5 reports that the reported temperatures are internally measured.