Kubernetes 气流服务尚未开始
我们正在尝试部署airflow(1.9.0)版本,该版本使用Postgres作为其DB,并使用Redis实现Kubernetes中吊舱之间的连接 同样的设置在暂存环境中运行良好,在prod环境中失败,经过调查,我了解到image 9.6的Postgres基本版本最近发生了更改,这会导致任何问题吗 附件是postgress和Web服务器的日志(气流) 尝试使用不同的姿势图像,但仍然相同 Web服务器日志Kubernetes 气流服务尚未开始,kubernetes,webserver,airflow,postgres-9.6,Kubernetes,Webserver,Airflow,Postgres 9.6,我们正在尝试部署airflow(1.9.0)版本,该版本使用Postgres作为其DB,并使用Redis实现Kubernetes中吊舱之间的连接 同样的设置在暂存环境中运行良好,在prod环境中失败,经过调查,我了解到image 9.6的Postgres基本版本最近发生了更改,这会导致任何问题吗 附件是postgress和Web服务器的日志(气流) 尝试使用不同的姿势图像,但仍然相同 Web服务器日志 Collecting botocore Downloading https://files
Collecting botocore
Downloading https://files.pythonhosted.org/packages/a1/b0/7a8794d914b95ef3335a5a4ba20595b46081dbd1e29f13812eceacf091ca/botocore-1.12.215-py2.py3-none-any.whl (5.7MB)
Collecting docutils<0.16,>=0.10 (from botocore)
Downloading https://files.pythonhosted.org/packages/22/cd/a6aa959dca619918ccb55023b4cb151949c64d4d5d55b3f4ffd7eee0c6e8/docutils-0.15.2-py3-none-any.whl (547kB)
Collecting jmespath<1.0.0,>=0.7.1 (from botocore)
Downloading https://files.pythonhosted.org/packages/83/94/7179c3832a6d45b266ddb2aac329e101367fbdb11f425f13771d27f225bb/jmespath-0.9.4-py2.py3-none-any.whl
Requirement already satisfied: urllib3<1.26,>=1.20; python_version >= "3.4" in /usr/lib/python3/dist-packages (from botocore) (1.22)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1; python_version >= "2.7" in /usr/local/lib/python3.6/dist-packages (from botocore) (2.8.0)
Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil<3.0.0,>=2.1; python_version >= "2.7"->botocore) (1.11.0)
Installing collected packages: docutils, jmespath, botocore
Successfully installed botocore-1.12.215 docutils-0.15.2 jmespath-0.9.4
Multi-tenant details not configured in this instance - Exiting
Cluster "abc" set.
User "abc@airflow.com" set.
Context "abc" created.
Switched to context "cedp".
[2019-08-26 14:01:03,391] {{driver.py:124}} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2019-08-26 14:01:03,415] {{driver.py:124}} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 17, in <module>
from airflow.bin.cli import CLIFactory
File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 47, in <module>
from airflow import jobs, settings
File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 64, in <module>
class BaseJob(Base, LoggingMixin):
File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 96, in BaseJob
executor=executors.GetDefaultExecutor(),
File "/usr/local/lib/python2.7/dist-packages/airflow/executors/__init__.py", line 42, in GetDefaultExecutor
DEFAULT_EXECUTOR = _get_executor(executor_name)
File "/usr/local/lib/python2.7/dist-packages/airflow/executors/__init__.py", line 60, in _get_executor
from airflow.executors.celery_executor import CeleryExecutor
File "/usr/local/lib/python2.7/dist-packages/airflow/executors/celery_executor.py", line 18, in <module>
from celery import Celery
File "/usr/local/lib/python2.7/dist-packages/celery/local.py", line 509, in __getattr__
module = __import__(self._object_origins[name], None, None, [name])
File "/usr/local/lib/python2.7/dist-packages/celery/app/__init__.py", line 5, in <module>
from celery import _state
File "/usr/local/lib/python2.7/dist-packages/celery/_state.py", line 17, in <module>
from celery.utils.threads import LocalStack
File "/usr/local/lib/python2.7/dist-packages/celery/utils/__init__.py", line 9, in <module>
from .nodenames import worker_direct, nodename, nodesplit
File "/usr/local/lib/python2.7/dist-packages/celery/utils/nodenames.py", line 9, in <module>
from kombu.entity import Exchange, Queue
File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 9, in <module>
from .serialization import prepare_accept_content
File "/usr/local/lib/python2.7/dist-packages/kombu/serialization.py", line 456, in <module>
for ep, args in entrypoints('kombu.serializers'): # pragma: no cover
File "/usr/local/lib/python2.7/dist-packages/kombu/utils/compat.py", line 89, in entrypoints
for ep in importlib_metadata.entry_points().get(namespace, [])
File "/usr/local/lib/python2.7/dist-packages/importlib_metadata/__init__.py", line 456, in entry_points
ordered = sorted(eps, key=by_group)
File "/usr/local/lib/python2.7/dist-packages/importlib_metadata/__init__.py", line 454, in <genexpr>
dist.entry_points for dist in distributions())
File "/usr/local/lib/python2.7/dist-packages/importlib_metadata/__init__.py", line 364, in <genexpr>
cls._search_path(path, pattern)
File "/usr/local/lib/python2.7/dist-packages/importlib_metadata/__init__.py", line 373, in _switch_path
return pathlib.Path(path)
File "/usr/local/lib/python2.7/dist-packages/pathlib2/__init__.py", line 1256, in __new__
self = cls._from_parts(args, init=False)
File "/usr/local/lib/python2.7/dist-packages/pathlib2/__init__.py", line 898, in _from_parts
drv, root, parts = self._parse_args(args)
File "/usr/local/lib/python2.7/dist-packages/pathlib2/__init__.py", line 891, in _parse_args
return cls._flavour.parse_parts(parts)
File "/usr/local/lib/python2.7/dist-packages/pathlib2/__init__.py", line 250, in parse_parts
parsed.append(intern(x))
TypeError: can't intern subclass of string
[2019-08-26 14:01:04,253] {{driver.py:124}} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2019-08-26 14:01:04,277] {{driver.py:124}} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
Traceback (most recent call last):
File "/usr/local/airflow/set_auth.py", line 16, in <module>
session.commit()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 927, in commit
self.transaction.commit()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 467, in commit
self._prepare_impl()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 447, in _prepare_impl
self.session.flush()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2209, in flush
self._flush(objects)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2329, in _flush
transaction.rollback(_capture_exception=True)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2293, in _flush
flush_context.execute()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 389, in execute
rec.execute(self)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 548, in execute
uow
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 181, in save_obj
mapper, table, insert)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 835, in _emit_insert_statements
execute(statement, params)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 945, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement
compiled_sql, distilled_params
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
context)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1402, in _handle_dbapi_exception
exc_info
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
context)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 470, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "users" does not exist
LINE 1: INSERT INTO users (username, email, password) VALUES ('admin...
^
[SQL: 'INSERT INTO users (username, email, password) VALUES (%(username)s, %(email)s, %(password)s) RETURNING users.id'] [parameters: {'username': 'admin', 'password': '$2b$12$F.8CTth9cL5G9f.pd180Duz/nC8S5KwTctwf/jG1Y/QB8PZagkTa.', 'email': 'abc@airflow.com'}]
[2019-08-26 14:01:15,229] {{driver.py:124}} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2019-08-26 14:01:15,253] {{driver.py:124}} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 17, in <module>
from airflow.bin.cli import CLIFactory
File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 47, in <module>
from airflow import jobs, settings
File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 64, in <module>
class BaseJob(Base, LoggingMixin):
File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 96, in BaseJob
executor=executors.GetDefaultExecutor(),
File "/usr/local/lib/python2.7/dist-packages/airflow/executors/__init__.py", line 42, in GetDefaultExecutor
DEFAULT_EXECUTOR = _get_executor(executor_name)
File "/usr/local/lib/python2.7/dist-packages/airflow/executors/__init__.py", line 60, in _get_executor
from airflow.executors.celery_executor import CeleryExecutor
File "/usr/local/lib/python2.7/dist-packages/airflow/executors/celery_executor.py", line 18, in <module>
from celery import Celery
File "/usr/local/lib/python2.7/dist-packages/celery/local.py", line 509, in __getattr__
module = __import__(self._object_origins[name], None, None, [name])
File "/usr/local/lib/python2.7/dist-packages/celery/app/__init__.py", line 5, in <module>
from celery import _state
File "/usr/local/lib/python2.7/dist-packages/celery/_state.py", line 17, in <module>
from celery.utils.threads import LocalStack
File "/usr/local/lib/python2.7/dist-packages/celery/utils/__init__.py", line 9, in <module>
from .nodenames import worker_direct, nodename, nodesplit
File "/usr/local/lib/python2.7/dist-packages/celery/utils/nodenames.py", line 9, in <module>
from kombu.entity import Exchange, Queue
File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 9, in <module>
from .serialization import prepare_accept_content
File "/usr/local/lib/python2.7/dist-packages/kombu/serialization.py", line 456, in <module>
for ep, args in entrypoints('kombu.serializers'): # pragma: no cover
File "/usr/local/lib/python2.7/dist-packages/kombu/utils/compat.py", line 89, in entrypoints
for ep in importlib_metadata.entry_points().get(namespace, [])
File "/usr/local/lib/python2.7/dist-packages/importlib_metadata/__init__.py", line 456, in entry_points
ordered = sorted(eps, key=by_group)
File "/usr/local/lib/python2.7/dist-packages/importlib_metadata/__init__.py", line 454, in <genexpr>
dist.entry_points for dist in distributions())
File "/usr/local/lib/python2.7/dist-packages/importlib_metadata/__init__.py", line 364, in <genexpr>
cls._search_path(path, pattern)
File "/usr/local/lib/python2.7/dist-packages/importlib_metadata/__init__.py", line 373, in _switch_path
return pathlib.Path(path)
File "/usr/local/lib/python2.7/dist-packages/pathlib2/__init__.py", line 1256, in __new__
self = cls._from_parts(args, init=False)
File "/usr/local/lib/python2.7/dist-packages/pathlib2/__init__.py", line 898, in _from_parts
drv, root, parts = self._parse_args(args)
File "/usr/local/lib/python2.7/dist-packages/pathlib2/__init__.py", line 891, in _parse_args
return cls._flavour.parse_parts(parts)
File "/usr/local/lib/python2.7/dist-packages/pathlib2/__init__.py", line 250, in parse_parts
parsed.append(intern(x))
TypeError: can't intern subclass of string
Postgress LOg
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default timezone ... Etc/UTC
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
pg_ctl -D /var/lib/postgresql/data -l logfile start
waiting for server to start....LOG: database system was shut down at 2019-08-26 13:42:41 UTC
LOG: MultiXact member wraparound protections are now enabled
LOG: database system is ready to accept connections
LOG: autovacuum launcher started
done
server started
CREATE DATABASE
/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
LOG: received fast shutdown request
waiting for server to shut down...LOG: aborting any active transactions
.LOG: autovacuum launcher shutting down
LOG: shutting down
LOG: database system is shut down
done
server stopped
PostgreSQL init process complete; ready for start up.
LOG: database system was shut down at 2019-08-26 13:42:43 UTC
LOG: MultiXact member wraparound protections are now enabled
LOG: database system is ready to accept connections
LOG: autovacuum launcher started
LOG: incomplete startup packet
LOG: incomplete startup packet
ERROR: relation "users" does not exist at character 13
STATEMENT: INSERT INTO users (username, email, password) VALUES ('admin', 'abc@airflow.com', '$2b$12$Vmkgo0OBgjLmylPMi3yrCOhVIWhWAgrEpCCojRZw0weeP..3nneg.') RETURNING users.id
LOG: incomplete startup packet
LOG: incomplete startup packet
LOG: incomplete startup packet
LOG: incomplete startup packet
ERROR: relation "users" does not exist at character 13
STATEMENT: INSERT INTO users (username, email, password) VALUES ('admin', 'abc@airflow.com', '$2b$12$y2DtC8uEM5coowQZP3GZsOIw/QFkqKZqvV4TcOkCSJ0wM.QbiwbA2') RETURNING users.id
LOG: incomplete startup packet
LOG: incomplete startup packet
LOG: incomplete startup packet
LOG: incomplete startup packet
ERROR: relation "users" does not exist at character 13
STATEMENT: INSERT INTO users (username, email, password) VALUES ('admin', 'abc@airflow.com', '$2b$12$0gj.4OfVy5y.xVt2FpVny.mRfCD/1wYnAbdMA22Xj4aI54tATo4Nu') RETURNING users.id
LOG: incomplete startup packet
LOG: incomplete startup packet
ERROR: relation "users" does not exist at character 13
STATEMENT: INSERT INTO users (username, email, password) VALUES ('admin', 'abc@airflow.com', '$2b$12$KXKdhuhdt5rmehEPxuX1He8uwE2fvgMcWoS4rg4oGzL5xWfn8Cgd6') RETURNING users.id
LOG: incomplete startup packet
LOG: incomplete startup packet
LOG: incomplete startup packet
ERROR: relation "users" does not exist at character 13
STATEMENT: INSERT INTO users (username, email, password) VALUES ('admin', 'abc@airflow.com', '$2b$12$QVtj0DHd6uLOnlIlwbE3kezYDzP.Y8m/Ln9H9of77pEKCihOiLhnq') RETURNING users.id
LOG: incomplete startup packet
收集botocore
正在下载https://files.pythonhosted.org/packages/a1/b0/7a8794d914b95ef3335a5a4ba20595b46081dbd1e29f13812eceacf091ca/botocore-1.12.215-py2.py3-none-any.whl (5.7MB)
收集文档=0.10(从botocore)
正在下载https://files.pythonhosted.org/packages/22/cd/a6aa959dca619918ccb55023b4cb151949c64d4d5d55b3f4ffd7eee0c6e8/docutils-0.15.2-py3-none-any.whl (547kB)
正在收集jmespath=0.7.1(从botocore)
正在下载https://files.pythonhosted.org/packages/83/94/7179c3832a6d45b266ddb2aac329e101367fbdb11f425f13771d27f225bb/jmespath-0.9.4-py2.py3-none-any.whl
已经满足的要求:urllib3=1.20;python_version>=/usr/lib/python3/dist包中的“3.4”(来自botocore)(1.22)
已经满足的需求:python dateutil=2.1;python_version>=/usr/local/lib/python3.6/dist-packages中的“2.7”(来自botocore)(2.8.0)
已满足要求:六个>=1.5 in/usr/lib/python3/dist包(来自python dateutil=2.1;python_版本>=“2.7”->botocore)(1.11.0)
安装收集的软件包:docutils、jmespath、botocore
已成功安装botocore-1.12.215 docutils-0.15.2 jmespath-0.9.4
此实例中未配置多租户详细信息-正在退出
集群“abc”集合。
用户“abc@airflow.com”“好的。
上下文“abc”已创建。
切换到上下文“cedp”。
[2019-08-26 14:01:03391]{{driver.py:124}}INFO-从/usr/lib/python2.7/lib2to3/grammar.txt生成语法表
[2019-08-26 14:01:03415]{{driver.py:124}}INFO-从/usr/lib/python2.7/lib2to3/patternegrammar.txt生成语法表
回溯(最近一次呼叫最后一次):
文件“/usr/local/bin/afflow”,第17行,在
从afflow.bin.cli导入CLIFactory
文件“/usr/local/lib/python2.7/dist-packages/afflow/bin/cli.py”,第47行,在
从导入作业、设置
文件“/usr/local/lib/python2.7/dist-packages/afflow/jobs.py”,第64行,在
类BaseJob(Base,LoggingMixin):
BaseJob中的第96行文件“/usr/local/lib/python2.7/dist-packages/afflow/jobs.py”
executor=executors.GetDefaultExecutor(),
文件“/usr/local/lib/python2.7/dist包/气流/executors/\u init\uuuy”,第42行,GetDefaultExecutor
默认执行者=\u获取执行者(执行者名称)
文件“/usr/local/lib/python2.7/dist packages/afflow/executors/__init__.py”,第60行,in_get_executor
从airflow.executors.celery\u executor导入芹菜executor
文件“/usr/local/lib/python2.7/dist packages/afflow/executors/celery_executor.py”,第18行,在
从芹菜进口芹菜
文件“/usr/local/lib/python2.7/dist packages/芹菜/local.py”,第509行,在__
模块=\uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu导入(self.\u对象\u来源[名称],无,无,[名称])
文件“/usr/local/lib/python2.7/dist-packages/芹菜/app/_-init__.py”,第5行,在
来自西芹进口州
文件“/usr/local/lib/python2.7/dist packages/芹菜/_state.py”,第17行,在
从celery.utils.threads导入LocalStack
文件“/usr/local/lib/python2.7/dist-packages/cellery/utils/__init__.py”,第9行,在
from.nodenames导入worker\u direct、nodename、nodesplit
文件“/usr/local/lib/python2.7/dist packages/cellery/utils/nodenames.py”,第9行,在
从kombu.entity导入交换,队列
文件“/usr/local/lib/python2.7/dist-packages/kombu/entity.py”,第9行,在
from.serialization导入准备\u接受\u内容
文件“/usr/local/lib/python2.7/dist-packages/kombu/serialization.py”,第456行,在
对于ep,入口点中的参数('kombu.serializers'):#pragma:无封面
文件“/usr/local/lib/python2.7/dist-packages/kombu/utils/compat.py”,第89行,入口点
对于importlib_metadata.entry_points().get(命名空间,[])中的ep
文件“/usr/local/lib/python2.7/dist packages/importlib_metadata/_init__.py”,第456行,入口点
已排序=已排序(eps,键=按组)
文件“/usr/local/lib/python2.7/dist packages/importlib_metadata/_init__.py”,第454行,在
dist.entry_分发中dist的点()
文件“/usr/local/lib/python2.7/dist packages/importlib_metadata/_init__.py”,第364行,在
cls.\u搜索\u路径(路径、模式)
文件“/usr/local/lib/python2.7/dist packages/importlib_metadata/_init__.py”,第373行,在开关路径中
返回pathlib.Path(路径)
文件“/usr/local/lib/python2.7/dist-packages/pathlib2/_-init__uuuuuu.py”,第1256行,在新的__
self=cls.\u来自零件(args,init=False)
文件“/usr/local/lib/python2.7/dist-packages/pathlib2/_-init___;.py”,第898行,位于from_u-parts中
drv,root,parts=self.\u parse\u args(args)
文件“/usr/local/lib/python2.7/dist-packages/pathlib2/_-init___;.py”,第891行,在解析参数中
返回cls.\u flavor.parse\u部分(parts)
文件“/usr/local/lib/python2.7/dist-packages/pathlib2/_-init__uuu.py”,第250行,在parse\u部分中
解析。追加(实习生(x))
TypeError:无法插入字符串的子类
[2019-08-26 14:01:04253]{{driver.py:124}}INFO-从/usr/lib/python2.7/lib2to3/grammar.txt生成语法表
[2019-08-26 14:01:04277]{{driver.py:124}}INFO-从/usr/lib/python2.7/lib2to3/patternegrammar.txt生成语法表
回溯(最近一次呼叫最后一次):
文件“/usr/local/afflow/set_auth.py”,第16行,在
session.commit()
提交中的文件“/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py”,第927行
self.transaction.commit()
提交中的文件“/usr/local/lib/python2.7/dist packages/sqlalchemy/orm/session.py”,第467行
self.\u prepare\u impl()
文件“/usr/local/lib/python2.7/dist-packages/sqlalchemy/or