Mediators should not enqueue keep-alives for handovers with a backlog

[salcock]

Keep alives are currently appended to the end of the buffer of data to send to the handover.

In the situation where we have a agency that has been absent for some time and therefore is having records queued up for it on the mediator, this leads to some keep alive failures.

If the keep alive response timeout is too low, then the keep alive may not even be sent to the agency before the timer expires, causing the handover to be dropped while clearing the backlog. The handover will likely reconnect and carry on, but it's not ideal and will likely cause concern for both users and agencies to see these connections continuing to drop.

Instead, we should avoid queuing a keep alive as long as there is data in the buffer to send via that handover, since keep alives should only be sent when the outgoing data connection is idle (as opposed to there being no incoming data for that handover coming from the collectors). We can instead restart the keep alive timer once the buffer is empty again.

[salcock]

#46 includes some changes to help address this problem, but it turns out that the real root cause of the problem is a bit more complicated.

I'm still seeing KA response failures in contrived scenarios, even though the send() call completed successfully and there are no bytes remaining in the application buffer to send. It appears that the KA can get stuck inside a kernel buffer somewhere (presumably waiting for space in either the send window or the client's receive window?) and therefore doesn't reach the client before the response timer can expire. Since we have no easy way of knowing when the KA actually gets sent, we can't postpone starting the response timer until we're sure that the KA has hit the wire.

I think in practical terms this is a big enough edge case that it doesn't really require an immediate fix. To trigger this, you'd need to meet a number of conditions:

• a short KA response timeout and/or a large backlog of buffered records to send to the client.
• a client that is processing the records relatively slowly, such that windows are being filled up (e.g. tracepktdump dumping all received records to an active terminal).
• there are no new records for the client agency being received by the mediator while it clears the backlog (otherwise no keepalive will be triggered).

So, I'm going to remove this from the 1.0.4 milestone but leave the issue open so I don't forget about this entirely.

One possible solution could revolve around the use of SO_TIMESTAMPING to get the timestamp once the packet was sent -- no timestamp == not sent yet.