Rekeying failure on a busy link

[MichelStam]

Guys,

I've been testing an annoying bug I've been having on a mesh of (currently) 2 units for the past week or so, but I cannot seem to find what causes it.

It happens when I run an iperf test between the units. At or around the time the SAE lifetime expires, a rekey occurs, after which traffic between the units stops.
Sometimes a packet arrives about a key lifetime later, but it does not get stable anymore.

If I leave the link idle (no iperf test, just some pings), then this problem does not seem to occur.

Looking at the debug traces from meshd-nl80211I can find no fault. I also looked at the key material sent down to the ath9k driver (printk's in the kernel driver), but even reading back those registers does not indicate to me that there's a fault.

Both units use an ath9k Atheros card; One is an AzureWave AR5B95, the other is a Compex WLE200N2-23. I have also observed the problem on Compex WLE350NX cards, so I am guessing this is not hardware related.

I set up both units with the attached config below;
meshd.txt

The kernel I use 4.4.11, but I've seen the same problem with 3.10.49.
The compat-wireless 2016-01-10 driver set used by OpenWRT seems to have the same problem with the old 3.10.34 kernel I run on that system.

The iperf setup is (using 2.0.5):

• One system running iperf -s -u -p 6969 -i 5
• One system running iperf -c -u -p 6969 -i 5 -t 86400 -b 100M

I create the mesh interfaces by:

• iw phy phy0 interface add mesh0 type mp
• ifconfig mesh0 IP MASK up
• meshd-nl80211 -c meshd.txt -i mesh0

Right now the key lifetime is at 60 seconds for problem reproduction, but I have seen the same problem on a link with a key lifetime of 3600 seconds; the link then dies at that time.

Can anyone give me a couple of pointers where to look, or maybe help me out?

Regards,

Michel Stam

[fhuberts]

I think this is related to #39

[MichelStam]

I have not seen the messages:
'confirm did not verify!'
anywhere in the logs, so I'm not sure about this.

Reading back I suppose I should have attached a log from one of the nodes as well. Its attached now.
meshd-log.txt

I just noticed your commit, I will test again with the latest master tomorrow morning.

[alexgrin]

This appears to be a long-standing issue with ath9k. I've submitted this as a bug to ath9k mailing list last year ( https://www.mail-archive.com/ath9k-devel%40lists.ath9k.org/msg13595.html ) and there was an earlier report of very similar issues ( http://lists.shmoo.com/pipermail/hostap/2014-November/031377.html ). I was not able to get any traction. I've gone as far as reading the key back from the card registers and it matches what's expected. Our workaround was to have a unicast probe between the nodes that occurs right after rekey and, if it fails, rekey.

[MichelStam]

@alexgrin:
I'm inclined to believe you on that, looking at the debugging you did. Do you have a patch of that workaround somewhere?

@fhuberts:
I've tested with your patches, no changes as far as the bug is concerned.

[fhuberts]

On 23/06/16 18:10, Alexis Green wrote:

This appears to be a long-standing issue with ath9k. I've submitted this
as a bug to ath9k mailing list last year (
https://www.mail-archive.com/ath9k-devel%40lists.ath9k.org/msg13595.html
) and there was an earlier report of very similar issues (
http://lists.shmoo.com/pipermail/hostap/2014-November/031377.html ). I
was not able to get any traction. I've gone as far as reading the key
back from the card registers and it matches what's expected. Our
workaround was to have a unicast probe between the nodes that occurs
right after rekey and, if it fails, rekey.

Would you mind sharing the code of your workaround?

[alexgrin]

The code is pretty awful looking but I'll see if I can button it up this
weekend and push up to my repo.

On Fri, Jun 24, 2016 at 6:10 AM, Ferry Huberts notifications@github.com
wrote:

On 23/06/16 18:10, Alexis Green wrote:

This appears to be a long-standing issue with ath9k. I've submitted this
as a bug to ath9k mailing list last year (
https://www.mail-archive.com/ath9k-devel%40lists.ath9k.org/msg13595.html
) and there was an earlier report of very similar issues (
http://lists.shmoo.com/pipermail/hostap/2014-November/031377.html ). I
was not able to get any traction. I've gone as far as reading the key
back from the card registers and it matches what's expected. Our
workaround was to have a unicast probe between the nodes that occurs
right after rekey and, if it fails, rekey.

Would you mind sharing the code of your workaround?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#42 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/ABLZNlU0RFhCO7ZEYRMhnEpY7l_n6wjmks5qO9dDgaJpZM4I81o7
.

[MichelStam]

@alexgrin: I was looking at the workaround you described, I assume that you use the NL80211_CMD_PROBE_CLIENT netlink call for this? This would probably require a small patch in net/wireless/nl80211.c to make this work (this call is only allowed for AP and P2P interfaces, by default). Or do you probe differently?

[alexgrin]

Nope, it's nowhere near as awesome as you think. Authsae does a multicast ping (layer3) after rekey and waits to hear a unicast response from the a device with MAC address of the peer we just rekeyed with. If there's no response, rekey is triggered. You have to specify the interface for multicast to the daemon for this to work. It's a pretty nasty hackjob and I'll post the code as is (-ish) soon.

[alexgrin]

Here's the yuckyness - uniumwifi@a1591d3

[fhuberts]

thanks for sharing!

[MichelStam]

Thank you for the patch! I'll have a look at it.

Sorry for the radio silence, was testing a patch which may help find a solution.
It works by resetting the ath9k chip when a new key is installed. It does seem to cause a LOT of authsae renegotiation traffic.
The link does keep forwarding traffic.
Although it is a start, I would hardly call this a nice patch, its the equivalent of using buckshot to swat mosquitoes.

Would someone mind checking it out and maybe suggesting a better approach?

ath9k-install_key-buckshot.diff.txt

[MichelStam]

I've posted the issue on the ath9k-devel list as well, hopefully I can stir something up/get the help to address this.

https://lists.ath9k.org/pipermail/ath9k-devel/2016-July/014676.html

[MichelStam]

So far 0 response, tried to bump it one time with no effect.

For the forseeable future, I've chosen to use software encryption.

[chunyeow]

Please note that your maximum achievable throughput will degrade if using
software encryption.

On Wed, Jul 13, 2016 at 11:41 PM, MichelStam notifications@github.com
wrote:

So far 0 response, tried to bump it one time with no effect.

For the forseeable future, I've chosen to use software encryption.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#42 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/ABBewvbaVf5ZQaXzL1faMdgbnG0gX9vuks5qVQdGgaJpZM4I81o7
.

[MichelStam]

Hello Chun-Yeow,

I agree, there is a measurable performance drop of about 2 Mbps. Luckily, for this particular application, high bandwidth is not the most important, but link stability is.
I see it as a temporary solution so the project can move forward at my employers' side. When a fix is available, we're definitely switching back to hw encryption.

On a side note, I seem to have gotten a little traction on the ath9k-devel list; Adrian Chadd has taken a look at the Atheros reference driver, which seems to have a fix called ATH_SUPPORT_KEYPLUMB_WAR. This reinserts the key when there's Rx decryption errors.
This fix may have to be ported into the ath9k driver.

Is it maybe an idea for those of you that have run into this issue as well at some point or the other to pitch in on the ath9k-devel list?

Cheers,

Michel