Send-Q Back Logged

[stevanvv]

Having an odd issue where the Send-Q for the TCP Socket is back-logged, and astmanproxy crashes. This happens when the client connected to the proxy loses internet connectivity unexpectedly, the proxy doesn't know how to handle that sudden disconnect. Any ideas on how we can address this?

[davies147]

Hi, I thought I fixed this a long time ago. Which send Q? Client or asterisk? In theory, that will cause a socket error, and that connection will be closed, but astmanproxy will continue. Are you able to see what response code you get from the attempt to write? Perhaps I am not catching it correctly? Thanks, Steve

…

On Fri, 17 May 2019 at 17:40, Stevan Veselinovc ***@***.***> wrote: Having an odd issue where the Send-Q for the TCP Socket is back-logged, and astmanproxy crashes. This happens when the client connected to the proxy loses internet connectivity unexpectedly, the proxy doesn't know how to handle that sudden disconnect. Any ideas on how we can address this? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#10?email_source=notifications&email_token=AAADS23RH6G4KW64JRNLJD3PV3NXRA5CNFSM4HNWZHV2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GUOGPXQ>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAADS2ZTGLYEWPOJDP4HVGTPV3NXRANCNFSM4HNWZHVQ> .

[stevanvv]

Hi Steve,
Thanks for reply.. The client (proxy) shows the backlog in the Send-Q, the software that uses the proxy just sits and waits in a TIME_WAIT. I'll work on getting you the response code today and get back to you.

Thanks,
Steve

[stevanvv]

Hi Steve,

The proxy is not displaying any error messages in its log or while running in verbose. The issue that occurs when one of the tcp sockets begins to build a large send-q is that the other clients using the same proxy no longer function either, as if the proxy stops sending events all together.

[davies147]

Hi, Just to be clear, this is a send-Q within astmanproxy, where it is trying to send to a 'client' (ie not asterisk) and that client is unable to respond, so the write is failing? Also, are you using the "standard" format, or one of the others?

…

On Wed, 22 May 2019 at 07:54, Stevan Veselinovc ***@***.***> wrote: Hi Steve, The proxy is not displaying any error messages in its log or while running in verbose. The issue that occurs when one of the tcp sockets begins to build a large send-q is that the other clients using the same proxy no longer function either, as if the proxy stops sending events all together. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#10?email_source=notifications&email_token=AAADS25GXTQZNLT74NCTD7TPWTUZRA5CNFSM4HNWZHV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODV6CVXY#issuecomment-494676703>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAADS2ZAX5CTFJ6BI5ENA4TPWTUZRANCNFSM4HNWZHVQ> .

[stevanvv]

Hi Steven,

That's correct, that is the Send-Q i'm talking about. We are only communicating in the standard format.

This is what I am seeing now running the proxy in debug mode:

May 23 11:41:55: asterisk@127.0.0.1: attempting read...
May 23 11:41:55: Read error -1 getting line
May 23 11:41:55: Returning standard block of 0 lines, res -1
May 23 11:41:55: standard_read result = -1
May 23 11:41:55: Connection closed: [software client ip address]
May 23 11:41:55: Read error -1 getting line
May 23 11:41:55: Returning standard block of 0 lines, res -1
May 23 11:41:55: standard_read result = -1
May 23 11:41:55: Freed entire stack.
May 23 11:41:55: --- exiting session_do thread ---
May 23 11:41:55: Connection closed: [software client ip address]

It seems like on the read from asterisk, once it fails it drops all connections, then the send-q begins to build up

[davies147]

Okay, so the closing of the client session when we lose asterisk connectivity is absolutely intended. We cannot handle a client if we have no server to talk to. Are you able to solve the problem with losing connection to asterisk? I would normally stop astmanproxy entirely before I stop asterisk. It is possible that there is a failure to cleanly close the client connection down. Can you show me a copy of the output of the following command when everything is all stuck - I am showing my example below: root@dev-1:/var/www/html# netstat -tnp | grep -e Q -e astmanproxy Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.1:5038 127.0.0.1:60212 ESTABLISHED 19867/astmanproxy tcp 0 0 127.0.0.1:37786 127.0.0.1:5037 ESTABLISHED 19867/astmanproxy tcp 0 0 10.0.0.32:5038 172.25.11.141:45678 ESTABLISHED 19867/astmanproxy Thanks, Steve

…

On Thu, 23 May 2019 at 20:00, Stevan Veselinovc ***@***.***> wrote: Hi Steven, That's correct, that is the Send-Q i'm talking about. We are only communicating in the standard format. This is what I am seeing now running the proxy in debug mode: May 23 11:41:55: ***@***.***: attempting read... May 23 11:41:55: Read error -1 getting line May 23 11:41:55: Returning standard block of 0 lines, res -1 May 23 11:41:55: standard_read result = -1 May 23 11:41:55: Connection closed: [software client ip address] May 23 11:41:55: Read error -1 getting line May 23 11:41:55: Returning standard block of 0 lines, res -1 May 23 11:41:55: standard_read result = -1 May 23 11:41:55: Freed entire stack. May 23 11:41:55: --- exiting session_do thread --- May 23 11:41:55: Connection closed: [software client ip address] It seems like on the read from asterisk, once it fails it drops all connections, then the send-q begins to build up — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#10?email_source=notifications&email_token=AAADS264C2DHEAIQO5X4JO3PW3SVVA5CNFSM4HNWZHV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWDFTOA#issuecomment-495344056>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAADS25BDGGSUFZR6O43MADPW3SVVANCNFSM4HNWZHVQ> .

[stevanvv]

Hi Davies,

Sorry this took so long to get back to you! I finally got down to the error message, it is:

corrupted unsupported chunks

following this error message, astmanproxy crashes

[davies147]

Oh, I have no idea where that error message might come from! Can you elaborate on how got found that message? Regards, Steve

…

On Wed, 12 Jun 2019 at 22:46, Stevan Veselinovc ***@***.***> wrote: Hi Davies, Sorry this took so long to get back to you! I finally got down to the error message, it is: corrupted unsupported chunks following this error message, astmanproxy crashes — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#10?email_source=notifications&email_token=AAADS26JLBL63IN2C6DN44LP2FVEPA5CNFSM4HNWZHV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXR4PZQ#issuecomment-501467110>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAADS2ZDXIUSGAEX6DAHQ6LP2FVEPANCNFSM4HNWZHVQ> .