Python 无法启动Airflow worker/flower,需要澄清气流结构,以确认安装正确
在不同的计算机上运行辅助程序会导致以下指定的错误。我已经跟踪并同步了dags文件夹 我还想确认RabbitMQ和PostgreSQL只需要安装在Airflow core机器上,而不需要安装在workers上(workers只连接到core) 设置的规格如下所示: 核心/服务器计算机 已安装以下组件:Python 无法启动Airflow worker/flower,需要澄清气流结构,以确认安装正确,python,rabbitmq,celery,airflow,Python,Rabbitmq,Celery,Airflow,在不同的计算机上运行辅助程序会导致以下指定的错误。我已经跟踪并同步了dags文件夹 我还想确认RabbitMQ和PostgreSQL只需要安装在Airflow core机器上,而不需要安装在workers上(workers只连接到core) 设置的规格如下所示: 核心/服务器计算机 已安装以下组件: Python2.7和 气流(气流\主气流=~/气流) 芹菜 心理学2 兔子 PostgreSQL 气流中的配置。cfg: sql\u alchemy\u conn=postgresql+p
- Python2.7和
- 气流(气流\主气流=~/气流)
- 芹菜
- 心理学2
- 兔子
- PostgreSQL
sql\u alchemy\u conn=postgresql+psycopg2://用户名:password@192.168.1.2:5432/气流
executor=CeleryExecutor
broker\u url=amqp://username:password@192.168.1.2:5672/
celery\u result\u backend=postgresql+psycopg2://用户名:password@192.168.1.2:5432/气流
- RabbitMQ正在运行
- 可以连接到PostgreSQL并已确认已创建表
- 可以启动和查看Web服务器(包括自定义DAG)
- Python2.7和
- 气流(气流\主气流=~/气流)
- 芹菜
- 心理学2
sql\u alchemy\u conn=postgresql+psycopg2://用户名:password@192.168.1.2:5432/气流
executor=CeleryExecutor
broker\u url=amqp://username:password@192.168.1.2:5672/
celery\u result\u backend=postgresql+psycopg2://用户名:password@192.168.1.2:5432/气流
气流花时
:
ubuntu@airflow_client:~/airflow$ airflow flower
[2016-06-13 04:19:42,814] {__init__.py:36} INFO - Using executor CeleryExecutor
Traceback (most recent call last):
File "/home/ubuntu/anaconda2/bin/airflow", line 15, in <module>
args.func(args)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/airflow/bin/cli.py", line 576, in flower
os.execvp("flower", ['flower', '-b', broka, port, api])
File "/home/ubuntu/anaconda2/lib/python2.7/os.py", line 346, in execvp
_execvpe(file, args)
File "/home/ubuntu/anaconda2/lib/python2.7/os.py", line 382, in _execvpe
func(fullname, *argrest)
OSError: [Errno 2] No such file or directory
ubuntu@airflow_client:~$ airflow worker
[2016-06-13 04:08:43,573] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-06-13 04:08:43,935: ERROR/MainProcess] Unrecoverable error: ImportError('No module named postgresql',)
Traceback (most recent call last):
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/celery/worker/__init__.py", line 206, in start
self.blueprint.start(self)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/celery/bootsteps.py", line 119, in start
self.on_start()
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/celery/apps/worker.py", line 169, in on_start
string(self.colored.cyan(' \n', self.startup_info())),
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/celery/apps/worker.py", line 230, in startup_info
results=self.app.backend.as_uri(),
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/kombu/utils/__init__.py", line 325, in __get__
value = obj.__dict__[self.__name__] = self.__get(obj)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/celery/app/base.py", line 626, in backend
return self._get_backend()
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/celery/app/base.py", line 444, in _get_backend
self.loader)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/celery/backends/__init__.py", line 68, in get_backend_by_url
return get_backend_cls(backend, loader), url
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/celery/backends/__init__.py", line 49, in get_backend_cls
cls = symbol_by_name(backend, aliases)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/kombu/utils/__init__.py", line 96, in symbol_by_name
module = imp(module_name, package=package, **kwargs)
File "/home/ubuntu/anaconda2/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
ImportError: No module named postgresql
当cellery\u result\u backend
更改为默认db时+mysql://airflow:airflow@localhost:3306/airflow
并再次运行airflow worker
。结果是:
ubuntu@airflow_client:~/airflow$ airflow worker
[2016-06-13 04:17:32,387] {__init__.py:36} INFO - Using executor CeleryExecutor
-------------- celery@airflow_client2 v3.1.23 (Cipater)
---- **** -----
--- * *** * -- Linux-3.19.0-59-generic-x86_64-with-debian-jessie-sid
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: airflow.executors.celery_executor:0x7f5cb65cb510
- ** ---------- .> transport: amqp://username:**@192.168.1.2:5672//
- ** ---------- .> results: mysql://airflow:**@localhost:3306/airflow
- *** --- * --- .> concurrency: 16 (prefork)
-- ******* ----
--- ***** ----- [queues]
-------------- .> default exchange=default(direct) key=celery
[2016-06-13 04:17:33,385] {__init__.py:36} INFO - Using executor CeleryExecutor
Starting flask
[2016-06-13 04:17:33,737] {_internal.py:87} INFO - * Running on http://0.0.0.0:8793/ (Press CTRL+C to quit)
[2016-06-13 04:17:34,536: WARNING/MainProcess] celery@airflow_client2 ready.
我错过了什么?如何进一步诊断此问题?您需要确保安装芹菜花。也就是说,
pip install flower
导入错误:没有名为postgresql的模块错误是由于在celery\u result\u后端使用的前缀无效造成的。将数据库用作芹菜后端时,连接URL必须以db+
作为前缀。看见
因此,请替换:
celery_result_backend = postgresql+psycopg2://username:password@192.168.1.2:5432/airflow
比如:
celery_result_backend = db+postgresql://username:password@192.168.1.2:5432/airflow
代理URL是否正确?以两条斜线结束。。。exchange名称是什么?芹菜文档有两个斜杠,从配置中删除这两个斜杠不会更改输出。