Exception ResponseNotReady

[andrenam]

Hi,

I stumbled upon your efforts with pywebcorp. Thanks for that. I gave it a try with
python 3.6.7 on win64, requests 2.21.0 (downgraded to 2.13.0 for testing an old version at the time of ~April 2017). With this code, I get the following error:

import requests
import pywebcorp.patch

r = requests.get(url='https://httpbin.org/get', verify=False)
print(r.text)

Traceback (most recent call last):
  File "C:\Users\USERNAME\conda_envs\pywebcorp\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 594, in urlopen
    self._prepare_proxy(conn)
  File "C:\Users\USERNAME\conda_envs\pywebcorp\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 810, in _prepare_proxy
    conn.connect()
  File "C:\Users\USERNAME\conda_envs\pywebcorp\lib\site-packages\pywebcorp\patch.py", line 95, in connect
    return super(NTLM_HTTPS, self).connect()
  File "C:\Users\USERNAME\conda_envs\pywebcorp\lib\site-packages\requests\packages\urllib3\connection.py", line 294, in connect
    self._tunnel()
  File "C:\Users\USERNAME\conda_envs\pywebcorp\lib\site-packages\pywebcorp\patch.py", line 78, in _tunnel
    return self._tunnel()
  File "C:\Users\USERNAME\conda_envs\pywebcorp\lib\site-packages\pywebcorp\patch.py", line 67, in _tunnel
    r = self.getresponse()
  File "C:\Users\USERNAME\conda_envs\pywebcorp\lib\site-packages\pywebcorp\patch.py", line 54, in getresponse
    return self.getresponse()
  File "C:\Users\USERNAME\conda_envs\pywebcorp\lib\http\client.py", line 1321, in getresponse
    raise ResponseNotReady(self.__state)
http.client.ResponseNotReady: Request-sent

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\USERNAME\conda_envs\pywebcorp\lib\site-packages\requests\adapters.py", line 423, in send
    timeout=timeout
  File "C:\Users\USERNAME\conda_envs\pywebcorp\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 649, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "C:\Users\USERNAME\conda_envs\pywebcorp\lib\site-packages\requests\packages\urllib3\util\retry.py", line 347, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "C:\Users\USERNAME\conda_envs\pywebcorp\lib\site-packages\requests\packages\urllib3\packages\six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "C:\Users\USERNAME\conda_envs\pywebcorp\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 594, in urlopen
    self._prepare_proxy(conn)
  File "C:\Users\USERNAME\conda_envs\pywebcorp\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 810, in _prepare_proxy
    conn.connect()
  File "C:\Users\USERNAME\conda_envs\pywebcorp\lib\site-packages\pywebcorp\patch.py", line 95, in connect
    return super(NTLM_HTTPS, self).connect()
  File "C:\Users\USERNAME\conda_envs\pywebcorp\lib\site-packages\requests\packages\urllib3\connection.py", line 294, in connect
    self._tunnel()
  File "C:\Users\USERNAME\conda_envs\pywebcorp\lib\site-packages\pywebcorp\patch.py", line 78, in _tunnel
    return self._tunnel()
  File "C:\Users\USERNAME\conda_envs\pywebcorp\lib\site-packages\pywebcorp\patch.py", line 67, in _tunnel
    r = self.getresponse()
  File "C:\Users\USERNAME\conda_envs\pywebcorp\lib\site-packages\pywebcorp\patch.py", line 54, in getresponse
    return self.getresponse()
  File "C:\Users\USERNAME\conda_envs\pywebcorp\lib\http\client.py", line 1321, ingetresponse
    raise ResponseNotReady(self.__state)
requests.packages.urllib3.exceptions.ProtocolError: ('Connection aborted.', ResponseNotReady('Request-sent',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "Y:\python\pywebcorp\test.py", line 7, in <module>
    r = requests.get(url='https://httpbin.org/get', verify=False)
  File "C:\Users\USERNAME\conda_envs\pywebcorp\lib\site-packages\requests\api.py", line 70, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Users\USERNAME\conda_envs\pywebcorp\lib\site-packages\requests\api.py", line 56, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\USERNAME\conda_envs\pywebcorp\lib\site-packages\requests\sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\USERNAME\conda_envs\pywebcorp\lib\site-packages\requests\sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\USERNAME\conda_envs\pywebcorp\lib\site-packages\requests\adapters.py", line 473, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ResponseNotReady('Request-sent',))

Do you have any idea that might help me?

[benjimin]

Thanks @andrenam . You could try checking out the test branch, and reporting what output test.py produces?

(pywebcorp ought to work if you are on windows, your OS is configured to report the proxy correctly, the proxy supports NTLM protocol, and your OS and proxy are both using a shared system for credential authentication, unless there has been a significant mismatch in dependency versions.. So I'm not yet sure what the root issue is in your case.)

[andrewisplinghoff]

I had exactly the same issue. What helped in my case was to change the line

pywebcorp/pywebcorp/patch.py

Line 46 in 1434199

r.read(amt=0) # change state (finish with this response to permit the next)

to the following:

r.read()

It seems that amt=0 is the culprit here. Not reading the full response makes the next request fail (https://stackoverflow.com/a/10876180/1491297).

I'm using Python 3.7.1, Windows 7, requests 2.21.0.

[andrewisplinghoff]

I've had another issue: only the very first HTTPS connection worked correctly and all subsequent requests failed. The problem here was that a mutable default argument ({}) is used in the headers argument of the NTLMmixin.request() method.

To fix this issue, change

pywebcorp/pywebcorp/patch.py

Lines 20 to 21 in 1434199

	def request(self, method, url, body=None, headers={}):
	"""Cache request"""

to the following:

def request(self, method, url, body=None, headers=None):
    """Cache request"""
    if headers is None:
        headers = {}

What happens without this change is that in the call from

pywebcorp/pywebcorp/patch.py

Line 66 in 1434199

self.request('CONNECT', tunnel_resource)

the Proxy-Authorization header from the previous request gets sent although in the method call no headers were given (headers argument is left out).

More info on why you should be hesitant to use mutable default arguments: https://stackoverflow.com/questions/1132941/least-astonishment-and-the-mutable-default-argument