Some DB issue that crashes the app.

[ivukotic]

Sometimes just logging in would crash the app.

This is what I find in the logs:

172.16.140.84 - - [10/May/2022:12:01:31 +0000] "GET / HTTP/1.1" 200 3098 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) coc_coc_browser/80.0.180 Chrome/74.0.3729.180 Safari/537.36"
[2022-05-10 12:42:50,629] ERROR in app: Exception on /token/refresh [POST]
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1820, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functionsrule.endpoint
File "/usr/local/lib/python3.6/site-packages/flask_restful/init.py", line 467, in wrapper
resp = resource(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/flask/views.py", line 89, in view
return self.dispatch_request(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/flask_restful/init.py", line 582, in dispatch_request
resp = meth(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/flask_jwt_extended/view_decorators.py", line 160, in wrapper
return fn(*args, **kwargs)
File "/home/servicex/servicex/resources/users/token_refresh.py", line 38, in post
user = UserModel.find_by_sub(get_jwt_identity())
File "/home/servicex/servicex/models.py", line 74, in find_by_sub
return cls.query.filter_by(sub=sub).first()
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2819, in first
return self.limit(1)._iter().first()
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2906, in _iter
execution_options={"_sa_orm_load_options": self.load_options},
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1692, in execute
result = conn._execute_20(statement, params or {}, execution_options)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 326, in _execute_on_connection
self, multiparams, params, execution_options
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1508, in _execute_clauseelement
cache_hit=cache_hit,
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1863, in execute_context
e, statement, parameters, cursor, context
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2044, in handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from=e
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 207, in raise
raise exception
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1820, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

[SQL: SELECT users.admin AS users_admin, users.created_at AS users_created_at, users.email AS users_email, users.experiment AS users_experiment, users.id AS users_id, users.institution AS users_institution, users.name AS users_name, users.pending AS users_pending, users.refresh_token AS users_refresh_token, users.sub AS users_sub, users.updated_at AS users_updated_at
FROM users
WHERE users.sub = %(sub_1)s
LIMIT %(param_1)s]
[parameters: {'sub_1': 'a51dbd7e-d274-11e5-9b11-9be347a09ce0', 'param_1': 1}]
(Background on this error at: https://sqlalche.me/e/14/e3q8)

[BenGalewsky]

@ivukotic can you see if the Postgres pod is ok? That would seem like the most likely issue with this error.

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

[ivukotic]

Seems postgres is OK. It did have 10 restarts but all before 23rd May.
We should increase request on its memory. We request 256MB and currently using 442MB.