CubeCell only handles first LoRaWAN MAC downlink following its first uplink

[pmurman]

Hi,
Board: CubeCell v1 and v2 dev board
LoRaWAN: 1.10 + 1.04, OTAA, ADR enabled
Dev: PlatformIO
RL: 7.6.0

Some CubeCell boards stop 'seeing' MAC downlinks sent by TTN after successfully receiving the first one.
Individual clock drift corrections are applied to each CubeCell.
Clock drift for the ones that are now failing is stable and similar to the clock drift of the good guys.

Any suggestions what I could try to come closer to what may be causing this?
(I did enable RL debugging (1st and 2nd option), but my app didn't like that causing hang-up during OTAA.)

With other timing/rollover issues out of the way now, having CubeCells running RLLoRaWAN seems within reach...

[StevenCellist]

As mentioned in #1691 (comment), converting to a discussion.

[StevenCellist]

Can you share a couple of logs from those devices, including the terminal timestamp from your PC? So using the DEBUG_PROTOCOL flag in BuildOptUser.h and enabling timestamps in your terminal program. Then we can check out most of the things that happen on your device.

[pmurman]

(No worries, we Dutch are used to what others consider rude/offensive, and we call 'direct' ;)

The great work is absolutely on you, @olivierouellet and not to forget @bns!! Without all of your help, I'd still be wandering aimlessly in the woods...

I wrote up my findings here perhaps some of it can be used for the wiki.
If I can be of assistance with this, or with further testing/experimentation, please let me know!

What surprised me is that although CubeCell clock drift is quite temperature dependent, the device I put in my freezer at -18oC coming from +20oC kept sending uplinks all the time, without any misses. (I did not check for missing MAC downlinks though, but it was on air already for a day before that time and the LNS may not have sent any MAC commands during the freezer test.)
Is OTAA more timing (thus temperature) critical than sending uplinks?

To @olivierouellet: you also use clock drift correction. Is temperature dependency not an issue for your devices?

[StevenCellist]

Right.. the 'u' in the surname instead of 'oe' slightly threw me off but my suspicion was right that you're Dutch too ;)

And just for completeness' sake.. bns on the Heltec forum (which is a totally random user on GitHub) is me here.

The device clock could drift by 50% and still uplink just fine. Uplinks are stateless, i.e. completely independent of the previous uplink. It's just the downlink window that may shift. This can be compensated by a wider scanGuard if you can afford the additional Rx power consumption, or a TCXO for timing which is how @olivierouellet is preventing his clock from drifting with temperature.

If your LNS uses a 1s Rx1 delay, then yes, OTAA is more timing critical because it uses a 5s Rx1 delay and in 5s you drift more than in 1s. Plus, you immediately see that the JoinAccept is missed because the device is going to send another JoinRequest. You don't get that immediate feedback for missing downlinks except that the LNS is going to keep on sending downlinks which is the worst thing that can happen in LoRaWAN. So if you keep an eye on the console and notice repeated downlinks hours after your device has joined, then it's guaranteed missing them.

[pmurman]

bns on the Heltec forum is me here.

Ha, ha! You got me there.

Thanks for explaining timing considerations - very insightful.

[olivierouellet]

To @olivierouellet: you also use clock drift correction. Is temperature dependency not an issue for your devices?

I was using it because of a totally unrelated bug. I got rid of the drift correction and everything is working fine on my STM32WLE since the bug fix. As Steven said, the TCXO on my board helps to prevent drifts related to temperature changes.

[pmurman]

Ah, I see. (STM32WLE5 - very nice, makes me jealous...)

[olivierouellet]

Did you try to change scanGuard default value ? For a board having timing issues, I'd say the default 10 ms is probably not enough. The comment suggest to increase the value to at least 50 for debugging purpose. I'd try up to max value (see warning below) to see if it changes something.

    /*!
      \brief Rx window padding in milliseconds
      according to the spec, the Rx window must be at least enough time to effectively detect a preamble
      but we pad it a bit on both sides (start and end) to make sure it is wide enough
      The larger this number the more power will be consumed! So be careful about changing it.
      For debugging purposes 50 is a reasonable start, but for production devices it should
      be as low as possible.
      0 is a valid time.

      500 is the **maximum** value, but it is not a good idea to go anywhere near that.
      If you have to go above 50 you probably have a bug somewhere. Check your device timing.
    */
    RadioLibTime_t scanGuard = 10;

[pmurman]

Thanks for your suggestion. Turns out there must be something else going on some boards, interfering with the radio.
Which I find good news as it appears not to be yet another CubeCell firmware / timing surprise.