Every X seconds web dashboard queries status of all the requests. That creates a lot of traffic on the servicex backend.
When something is not right at the backend this request fails and then very soon app fails with:
"HTTPConnectionPool(host='xaod-minio', port=9000): Max retries exceeded with url: /333ccae1-0424-41ef-af9a-94a313db142b (Caused by ResponseError('too many 500 error responses',)) 192.168.3.183 - - [27/Sep/2021:15:44:34 +0000] "POST //servicex/transformation HTTP/1.1" 500 36 "-" "Python/3.8 aiohttp/3.7.4.post0" 192.168.3.183 - - [27/Sep/2021:15:49:48 +0000] "POST //token/refresh HTTP/1.1" 200 338 "-" "Python/3.8 aiohttp/3.7.4.post0" object store <servicex.object_store_manager.ObjectStoreManager object at 0x7f1d36d5f6d8> File "/home/servicex/servicex/resources/submit_transformation_request.py", line 126, in post self.object_store.create_bucket(request_id) File "/home/servicex/servicex/object_store_manager.py", line 38, in create_bucket self.minio_client.make_bucket(bucket_name) File "/usr/local/lib/python3.6/site-packages/minio/api.py", line 631, in make_bucket headers=headers, File "/usr/local/lib/python3.6/site-packages/minio/api.py", line 278, in _url_open preload_content=preload_content, File "/usr/local/lib/python3.6/site-packages/urllib3/poolmanager.py", line 375, in urlopen response = conn.urlopen(method, u.request_uri, **kw) File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 859, in urlopen **response_kw File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 859, in urlopen **response_kw File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 859, in urlopen **response_kw [Previous line repeated 2 more times] File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 836, in urlopen retries = retries.increment(method, url, response=response, _pool=self) File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 574, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) HTTPConnectionPool(host='xaod-minio', port=9000): Max retries exceeded with url: /c48217fc-9a27-4390-af8e-c306494d5a99 (Caused by ResponseError('too many 500 error responses',))"
Every X seconds web dashboard queries status of all the requests. That creates a lot of traffic on the servicex backend.
When something is not right at the backend this request fails and then very soon app fails with:
"HTTPConnectionPool(host='xaod-minio', port=9000): Max retries exceeded with url: /333ccae1-0424-41ef-af9a-94a313db142b (Caused by ResponseError('too many 500 error responses',)) 192.168.3.183 - - [27/Sep/2021:15:44:34 +0000] "POST //servicex/transformation HTTP/1.1" 500 36 "-" "Python/3.8 aiohttp/3.7.4.post0" 192.168.3.183 - - [27/Sep/2021:15:49:48 +0000] "POST //token/refresh HTTP/1.1" 200 338 "-" "Python/3.8 aiohttp/3.7.4.post0" object store <servicex.object_store_manager.ObjectStoreManager object at 0x7f1d36d5f6d8> File "/home/servicex/servicex/resources/submit_transformation_request.py", line 126, in post self.object_store.create_bucket(request_id) File "/home/servicex/servicex/object_store_manager.py", line 38, in create_bucket self.minio_client.make_bucket(bucket_name) File "/usr/local/lib/python3.6/site-packages/minio/api.py", line 631, in make_bucket headers=headers, File "/usr/local/lib/python3.6/site-packages/minio/api.py", line 278, in _url_open preload_content=preload_content, File "/usr/local/lib/python3.6/site-packages/urllib3/poolmanager.py", line 375, in urlopen response = conn.urlopen(method, u.request_uri, **kw) File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 859, in urlopen **response_kw File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 859, in urlopen **response_kw File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 859, in urlopen **response_kw [Previous line repeated 2 more times] File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 836, in urlopen retries = retries.increment(method, url, response=response, _pool=self) File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 574, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) HTTPConnectionPool(host='xaod-minio', port=9000): Max retries exceeded with url: /c48217fc-9a27-4390-af8e-c306494d5a99 (Caused by ResponseError('too many 500 error responses',))"