Airflow Web服务器未响应（ome工作人员似乎已死亡，gunicorn未按预期重新启动他们）

我们最近开始看到Web服务器停止响应。运行

systemctl status webserver

时，该服务似乎仍在运行，但日志中充满了错误，wb服务没有响应。中的错误

Sep 14 06:56:45 semaf1-dk1.mid.dom airflow[1833]: [2020-09-14 06:56:45,662] {{cli.py:990}} ERROR - [0 / 0] Some workers seem to have died and gunicorn did not restart them as expected
Sep 14 06:56:56 semaf1-dk1.mid.dom airflow[1833]: [2020-09-14 06:56:56,701] {{cli.py:990}} ERROR - [0 / 0] Some workers seem to have died and gunicorn did not restart them as expected
Sep 14 06:57:07 semaf1-dk1.mid.dom airflow[1833]: [2020-09-14 06:57:07,738] {{cli.py:990}} ERROR - [0 / 0] Some workers seem to have died and gunicorn did not restart them as expected

我在日志中倒退到了这些开始发生之前的最后一行（

journalctl-u airflow webserver-a | grep-v“一些工人似乎已经死亡，gunicorn没有按预期重新启动他们”| tail-n 100

），我发现的最后一个错误如下所示：

Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: [2020-09-13 05:12:17,503] {{base.py:601}} ERROR - Add Permission on Menu Error: (pyodbc.OperationalError) ('08S01', '[08S01] [Microsoft][ODBC Driver 17 for SQL Server]Communication link failure (0) (SQLExecDirectW)')
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: [SQL: SELECT TOP 1 ab_view_menu.id AS ab_view_menu_id, ab_view_menu.name AS ab_view_menu_name
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: FROM ab_view_menu
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: WHERE ab_view_menu.name = ?]
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: [parameters: ('Logs',)]
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: (Background on this error at: http://sqlalche.me/e/13/e3q8)
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: [2020-09-13 05:12:17,504] {{base.py:600}} ERROR - (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: [SQL: SELECT TOP 1 ab_view_menu.id AS ab_view_menu_id, ab_view_menu.name AS ab_view_menu_name
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: FROM ab_view_menu
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: WHERE ab_view_menu.name = ?]
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: [parameters: [immutabledict({})]]
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: Traceback (most recent call last):
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: File "/usr/local/airflow/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1203, in _execute_context
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: conn = self._revalidate_connection()
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: File "/usr/local/airflow/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 473, in _revalidate_connection
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: "Can't reconnect until invalid "
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: sqlalchemy.exc.InvalidRequestError: Can't reconnect until invalid transaction is rolled back
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: The above exception was the direct cause of the following exception:
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: Traceback (most recent call last):
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: File "/usr/local/airflow/lib64/python3.6/site-packages/flask_appbuilder/base.py", line 598, in _add_permissions_menu
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: self.sm.add_permissions_menu(name)
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: File "/usr/local/airflow/lib64/python3.6/site-packages/flask_appbuilder/security/manager.py", line 1211, in add_permissions_menu
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: self.add_view_menu(view_menu_name)
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: File "/usr/local/airflow/lib64/python3.6/site-packages/flask_appbuilder/security/sqla/manager.py", line 431, in add_view_menu
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: view_menu = self.find_view_menu(name)
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: File "/usr/local/airflow/lib64/python3.6/site-packages/flask_appbuilder/security/sqla/manager.py", line 420, in find_view_menu
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: return self.get_session.query(self.viewmenu_model).filter_by(name=name).first()
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: File "/usr/local/airflow/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3397, in first
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: ret = list(self[0:1])
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: File "/usr/local/airflow/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3171, in __getitem__
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: return list(res)
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: File "/usr/local/airflow/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3503, in __iter__
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: return self._execute_and_instances(context)
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: File "/usr/local/airflow/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3528, in _execute_and_instances
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: result = conn.execute(querycontext.statement, self._params)
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: File "/usr/local/airflow/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1014, in execute
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: return meth(self, multiparams, params)
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: File "/usr/local/airflow/lib64/python3.6/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: return connection._execute_clauseelement(self, multiparams, params)
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: File "/usr/local/airflow/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1133, in _execute_clauseelement
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: distilled_params,
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: File "/usr/local/airflow/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1208, in _execute_context
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: e, util.text_type(statement), parameters, None, None
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: File "/usr/local/airflow/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1512, in _handle_dbapi_exception
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: sqlalchemy_exception, with_traceback=exc_info[2], from_=e
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: File "/usr/local/airflow/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: raise exception
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: File "/usr/local/airflow/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1203, in _execute_context
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: conn = self._revalidate_connection()
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: File "/usr/local/airflow/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 473, in _revalidate_connection
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: "Can't reconnect until invalid "
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: sqlalchemy.exc.StatementError: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: [SQL: SELECT TOP 1 ab_view_menu.id AS ab_view_menu_id, ab_view_menu.name AS ab_view_menu_name
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: FROM ab_view_menu
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: WHERE ab_view_menu.name = ?]
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: [parameters: [immutabledict({})]]
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: [2020-09-13 05:12:17,505] {{base.py:601}} ERROR - Add Permission on Menu Error: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: [SQL: SELECT TOP 1 ab_view_menu.id AS ab_view_menu_id, ab_view_menu.name AS ab_view_menu_name
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: FROM ab_view_menu
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: WHERE ab_view_menu.name = ?]
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: [parameters: [immutabledict({})]]
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: [2020-09-13 05:12:17,508] {{base.py:414}} INFO - Registering class SlaMissModelView on menu SLA Misses
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: [2020-09-13 05:12:17,508] {{baseviews.py:266}} INFO - Registering route /slamiss/action/<string:name>/<pk> ['GET', 'POST']
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: [2020-09-13 05:12:17,508] {{baseviews.py:266}} INFO - Registering route /slamiss/action_post ['POST']
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: [2020-09-13 05:12:17,510] {{baseviews.py:266}} INFO - Registering route /slamiss/add ['GET', 'POST']
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: [2020-09-13 05:12:17,510] {{baseviews.py:266}} INFO - Registering route /slamiss/api ['GET']
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: [2020-09-13 05:12:17,510] {{baseviews.py:266}} INFO - Registering route /slamiss/api/column/add/<col_name> ['GET']
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: [2020-09-13 05:12:17,510] {{baseviews.py:266}} INFO - Registering route /slamiss/api/column/edit/<col_name> ['GET']
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: [2020-09-13 05:12:17,511] {{baseviews.py:266}} INFO - Registering route /slamiss/api/create ['POST']
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: [2020-09-13 05:12:17,511] {{baseviews.py:266}} INFO - Registering route /slamiss/api/delete/<pk> ['DELETE']
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: [2020-09-13 05:12:17,511] {{baseviews.py:266}} INFO - Registering route /slamiss/api/get/<pk> ['GET']
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: [2020-09-13 05:12:17,511] {{baseviews.py:266}} INFO - Registering route /slamiss/api/read ['GET']
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: [2020-09-13 05:12:17,511] {{baseviews.py:266}} INFO - Registering route /slamiss/api/readvalues ['GET']
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]: [2020-09-13 05:12:17,512] {{baseviews.py:266}} INFO - Registering route /slamiss/api/update/<pk> ['PUT']

Sep 13 05:12:17 semaf1-dk1.mid.dom[1833]：[2020-09-13 05:12:17503]{{base.py:601}错误-在菜单上添加权限错误：（pyodbc.OperationalError）（'08S01'，'08S01][Microsoft][SQL Server的ODBC驱动程序17]通信链路故障（0）（SQLExecDirectW）'
Sep 13 05:12:17 semaf1-dk1.mid.dom气流[1833]：[SQL:选择顶部1 ab_视图菜单id作为ab_视图菜单id，选择ab_视图菜单名称作为ab_视图菜单名称
Sep 13 05:12:17 semaf1-dk1.mid.dom气流[1833]：从ab_视图菜单
Sep 13 05:12:17 semaf1-dk1.mid.dom气流[1833]：其中ab_view_menu.name=？]
Sep 13 05:12:17 semaf1-dk1.mid.dom气流[1833]：[参数：（'Logs'，）]
Sep 13 05:12:17 semaf1-dk1.mid.dom气流[1833]：（此错误的背景：http://sqlalche.me/e/13/e3q8)
Sep 13 05:12:17 semaf1-dk1.mid.dom[1833]：[2020-09-13 05:12:17504]{{base.py:600}错误-（sqlalchemy.exc.InvalidRequestError）在回滚无效事务之前无法重新连接
Sep 13 05:12:17 semaf1-dk1.mid.dom气流[1833]：[SQL:选择顶部1 ab_视图菜单id作为ab_视图菜单id，选择ab_视图菜单名称作为ab_视图菜单名称
Sep 13 05:12:17 semaf1-dk1.mid.dom气流[1833]：从ab_视图菜单
Sep 13 05:12:17 semaf1-dk1.mid.dom气流[1833]：其中ab_view_menu.name=？]
Sep 13 05:12:17 semaf1-dk1.mid.dom气流[1833]：[参数：[immutabledict（{}）]]
Sep 13 05:12:17 semaf1-dk1.mid.dom气流[1833]：回溯（最近一次呼叫最后一次）：
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]：文件“/usr/local/afflow/lib64/python3.6/site packages/sqlalchemy/engine/base.py”，第1203行，在执行上下文中
Sep 13 05:12:17 semaf1-dk1.mid.dom气流[1833]：conn=self.\u revalidate\u connection（）
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]：文件“/usr/local/afflow/lib64/python3.6/site packages/sqlalchemy/engine/base.py”，第473行，在重新验证连接中
Sep 13 05:12:17 semaf1-dk1.mid.dom[1833]：“在无效之前无法重新连接”
Sep 13 05:12:17 semaf1-dk1.mid.dom[1833]：sqlalchemy.exc.InvalidRequestError:在回滚无效事务之前无法重新连接
Sep 13 05:12:17 semaf1-dk1.mid.dom气流[1833]：上述异常是以下异常的直接原因：
Sep 13 05:12:17 semaf1-dk1.mid.dom气流[1833]：回溯（最近一次呼叫最后一次）：
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]：文件“/usr/local/afflow/lib64/python3.6/site packages/flask\u appbuilder/base.py”，第598行，在“添加权限”菜单中
Sep 13 05:12:17 semaf1-dk1.mid.dom[1833]：self.sm.add\u permissions\u菜单（名称）
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]：文件“/usr/local/afflow/lib64/python3.6/site packages/flask\u appbuilder/security/manager.py”，第1211行，在添加权限菜单中
Sep 13 05:12:17 semaf1-dk1.mid.dom气流[1833]：self.add\u视图菜单（视图菜单名称）
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]：文件“/usr/local/afflow/lib64/python3.6/site packages/flask\u appbuilder/security/sqla/manager.py”，第431行，在“添加视图”菜单中
Sep 13 05:12:17 semaf1-dk1.mid.dom气流[1833]：查看菜单=自我查找查看菜单（名称）
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]：文件“/usr/local/afflow/lib64/python3.6/site packages/flask\u appbuilder/security/sqla/manager.py”，第420行，在“查找视图”菜单中
Sep 13 05:12:17 semaf1-dk1.mid.dom[1833]：返回self.get\u session.query（self.viewmenu\u model）.筛选依据（name=name）.first（）
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]：文件“/usr/local/afflow/lib64/python3.6/site packages/sqlalchemy/orm/query.py”，第3397行，第一行
Sep 13 05:12:17 semaf1-dk1.mid.dom气流[1833]：ret=list（self[0:1]）
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]：文件“/usr/local/airflow/lib64/python3.6/site packages/sqlalchemy/orm/query.py”，第3171行，在__
Sep 13 05:12:17 semaf1-dk1.mid.dom气流[1833]：返回列表（res）
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]：文件“/usr/local/airflow/lib64/python3.6/site packages/sqlalchemy/orm/query.py”，第3503行，在__
Sep 13 05:12:17 semaf1-dk1.mid.dom[1833]：返回self.\u执行\u和\u实例（上下文）
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]：文件“/usr/local/afflow/lib64/python3.6/site packages/sqlalchemy/orm/query.py”，第3528行，在_execute_和_实例中
Sep 13 05:12:17 semaf1-dk1.mid.dom[1833]：结果=conn.execute（querycontext.statement，self.\u参数）
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]：文件“/usr/local/afflow/lib64/python3.6/site packages/sqlalchemy/engine/base.py”，执行中第1014行
Sep 13 05:12:17 semaf1-dk1.mid.dom气流[1833]：返回方法（自身、多图、参数）
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]：文件“/usr/local/afflow/lib64/python3.6/site packages/sqlalchemy/sql/elements.py”，第298行，在连接上执行
Sep 13 05:12:17 semaf1-dk1.mid.dom气流[1833]：返回连接执行子句元素（self、multiparams、params）
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]：文件“/usr/local/airflow/lib64/python3.6/site packages/sqlalchemy/engine/base.py”，第1133行，在执行条款元素中
Sep 13 05:12:17 semaf1-dk1.mid.dom气流[1833]：蒸馏参数，
Sep 13 05:12:17 semaf1-dk1.mid.dom airflow[1833]：文件“/usr/local/afflow/lib64/python3.6/site packages/sqlalchemy/engine/base.py”，第1208行，在执行上下文中
Sep 13 05:12:17 semaf1-dk1.mid.dom气流[1833]：e，util.text_类型（语句），参数
[2020-09-13 05:12:17,505] {{base.py:601}} ERROR - Add Permission on Menu Error:
(sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back