Airflow 气流错误-值错误:无法配置处理程序';文件处理器';

Airflow 气流错误-值错误:无法配置处理程序';文件处理器';,airflow,aws-codebuild,airflow-scheduler,Airflow,Aws Codebuild,Airflow Scheduler,我使用Python3.6.5在codebuild容器中运行airflow 1.9.0,我们执行以下命令并得到错误消息 ValueError: Unable to configure handler 'file.processor': 'FileProcessorHandler' object has no attribute 'log' sudo sh脚本/setup.sh AIRFLOW_HOME="$(cd "$( dirname "${BASH_SOURCE[0]}" )" &am

我使用Python3.6.5在codebuild容器中运行airflow 1.9.0,我们执行以下命令并得到错误消息

ValueError: Unable to configure handler 'file.processor': 
  'FileProcessorHandler' object has no attribute 'log'
sudo sh脚本/setup.sh

AIRFLOW_HOME="$(cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
export PIPENV_VENV_IN_PROJECT=True

cat <<EOF > $AIRFLOW_HOME/.env
PIPENV_VENV_IN_PROJECT=True
AIRFLOW_HOME=$AIRFLOW_HOME
EOF

cat <<EOF > $AIRFLOW_HOME/airflow.cfg

[core]
airflow_home = $AIRFLOW_HOME
dags_folder = $AIRFLOW_HOME/dags
base_log_folder = $AIRFLOW_HOME/logs
remote_log_conn_id =
encrypt_s3_logs = False
logging_level = INFO
logging_config_class =
log_format = [%%(asctime)s] {%%(filename)s:%%(lineno)d} %%(levelname)s - %%(message)s
simple_log_format = %%(asctime)s %%(levelname)s - %%(message)s
executor = SequentialExecutor
sql_alchemy_conn = sqlite:///$AIRFLOW_HOME/airflow.db
sql_alchemy_pool_size = 5
sql_alchemy_pool_recycle = 3600
parallelism = 32
dag_concurrency = 16
dags_are_paused_at_creation = True
non_pooled_task_slot_count = 128
max_active_runs_per_dag = 16
load_examples = False
plugins_folder = $AIRFLOW_HOME/plugins
fernet_key = QfQ90TaCFkE9fSLJnoikJCRxj4cnqFX69D1_Lbnhinw=
donot_pickle = False
dagbag_import_timeout = 30
task_runner = BashTaskRunner
default_impersonation =
security =
unit_test_mode = False
task_log_reader = file.task
enable_xcom_pickling = True
killed_task_cleanup_time = 60

[cli]
api_client = airflow.api.client.local_client
endpoint_url = http://localhost:8080

[api]
auth_backend = airflow.api.auth.backend.default

[operators]
default_owner = Airflow
default_cpus = 1
default_ram = 512
default_disk = 512
default_gpus = 0


[webserver]
base_url = http://localhost:8080
web_server_host = 0.0.0.0
web_server_port = 8080
web_server_ssl_cert =
web_server_ssl_key =
web_server_worker_timeout = 120
worker_refresh_batch_size = 1
worker_refresh_interval = 30
secret_key = temporary_key
workers = 4
worker_class = sync
access_logfile = -
error_logfile = -
expose_config = False
authenticate = False
filter_by_owner = False
owner_mode = user
dag_default_view = tree
dag_orientation = LR
demo_mode = False
log_fetch_timeout_sec = 5
hide_paused_dags_by_default = False
page_size = 100

[email]
email_backend = airflow.utils.email.send_email_smtp

[dask]
cluster_address = 127.0.0.1:8786

[scheduler]
job_heartbeat_sec = 5
scheduler_heartbeat_sec = 5
run_duration = -1
min_file_process_interval = 0
dag_dir_list_interval = 300
print_stats_interval = 30
child_process_log_directory = $AIRFLOW_HOME/logs/scheduler
scheduler_zombie_task_threshold = 300
catchup_by_default = True
max_tis_per_query = 0
statsd_on = False
statsd_host = localhost
statsd_port = 8125
statsd_prefix = airflow
max_threads = 2
authenticate = False

[github_enterprise]
api_rev = v3

[admin]
hide_sensitive_variable_fields = True

EOF

pipenv --three install
. $(pwd)/.venv/bin/activate
之后,我有了我的buildspec.yml

  - sudo sh scripts/setup.sh
  - . $(pwd)/.venv/bin/activate
  - airflow initdb
pipenv --three install
. $(pwd)/.venv/bin/activate
当我试图执行任何气流指令时,我得到了这个错误,我没有找到任何解决这个问题的方法

[Container] 2018/07/31 19:04:07 Running command airflow initdb
Unable to load the config, contains a configuration error.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/logging/config.py", line 558, in configure
handler = self.configure_handler(handlers[name])
File "/usr/local/lib/python3.6/logging/config.py", line 731, in configure_handler
result = factory(**kwargs)
File "/codebuild/output/src571717869/src/.venv/lib/python3.6/site- 
packages/airflow/utils/log/file_processor_handler.py", line 50, in __init__
self._symlink_latest_log_directory()
File "/codebuild/output/src571717869/src/.venv/lib/python3.6/site- packages/airflow/utils/log/file_processor_handler.py", line 110, in _symlink_latest_log_directory
self.log.warning(
AttributeError: 'FileProcessorHandler' object has no attribute 'log'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/codebuild/output/src571717869/src/.venv/bin/airflow", line 16, in 
<module>
from airflow import configuration
File "/codebuild/output/src571717869/src/.venv/lib/python3.6/site- 
packages/airflow/__init__.py", line 31, in <module>
from airflow import settings
File "/codebuild/output/src571717869/src/.venv/lib/python3.6/site- 
packages/airflow/settings.py", line 148, in <module>
configure_logging()
File "/codebuild/output/src571717869/src/.venv/lib/python3.6/site- 
packages/airflow/logging_config.py", line 75, in configure_logging
raise e
File "/codebuild/output/src571717869/src/.venv/lib/python3.6/site- 
packages/airflow/logging_config.py", line 70, in configure_logging
dictConfig(logging_config)
File "/usr/local/lib/python3.6/logging/config.py", line 795, in dictConfig
dictConfigClass(config).configure()
File "/usr/local/lib/python3.6/logging/config.py", line 566, in configure
'%r: %s' % (name, e))
ValueError: Unable to configure handler 'file.processor': 'FileProcessorHandler' object has no attribute 'log'

[Container] 2018/07/31 19:04:07 Command did not exit successfully airflow initdb exit status 1
[Container]2018/07/31 19:04:07运行命令initdb
无法加载配置,包含配置错误。
回溯(最近一次呼叫最后一次):
文件“/usr/local/lib/python3.6/logging/config.py”,第558行,在configure中
handler=self.configure\u处理程序(处理程序[名称])
文件“/usr/local/lib/python3.6/logging/config.py”,第731行,在configure\u处理程序中
结果=工厂(**kwargs)
文件“/codebuild/output/src571717869/src/.venv/lib/python3.6/site-
packages/afflow/utils/log/file_processor_handler.py”,第50行,in_uuuinit__
self.\u符号链接\u最新\u日志\u目录()
文件“/codebuild/output/src571717869/src/.venv/lib/python3.6/site-packages/afflow/utils/log/File\u processor\u handler.py”,第110行,在最新的日志目录下
self.log.warning(
AttributeError:'FileProcessorHandler'对象没有属性'log'
在处理上述异常期间,发生了另一个异常:
回溯(最近一次呼叫最后一次):
文件“/codebuild/output/src571717869/src/.venv/bin/afflow”,第16行,in
从气流导入配置
文件“/codebuild/output/src571717869/src/.venv/lib/python3.6/site-
包装/气流/_初始_.py”,第31行,in
从气流导入设置
文件“/codebuild/output/src571717869/src/.venv/lib/python3.6/site-
包装/气流/设置.py”,第148行,英寸
配置日志记录()
文件“/codebuild/output/src571717869/src/.venv/lib/python3.6/site-
packages/airflow/logging_config.py”,第75行,在configure_logging中
提高e
文件“/codebuild/output/src571717869/src/.venv/lib/python3.6/site-
packages/airflow/logging_config.py”,第70行,在configure_logging中
dictConfig(日志记录配置)
文件“/usr/local/lib/python3.6/logging/config.py”,第795行,在dictConfig中
dictConfigClass(config).configure()
文件“/usr/local/lib/python3.6/logging/config.py”,第566行,在configure中
“%r:%s%”(名称,e))
ValueError:无法配置处理程序“file.processor”:“FileProcessorHandler”对象没有属性“log”
[Container]2018/07/31 19:04:07命令未成功退出initdb退出状态1

我更改了气流的执行方式,气流可以正常工作,还更改了有关配置的一些细节:

所以:Ubuntu 16.04 Python 3.6.5

我从setup.sh中删除以下行,并将其放入buildspec.yml

  - sudo sh scripts/setup.sh
  - . $(pwd)/.venv/bin/activate
  - airflow initdb
pipenv --three install
. $(pwd)/.venv/bin/activate
对于我的执行,最好使用
pipenv run
而不是
pipenv shell

()

Buildspec.yml
我怀疑这可能与配置中的
logging\u config\u class
为空有关,但我以前使用过此生成平台。@tobi6我目前在从airflow\u default.cfg复制的my airflow.cfg中有
logging\u config\u class
empty。是否有一个默认的开箱即用的值可以放在这个配置中,或者我们必须先配置一个日志类,然后在这里设置它的路径()?不用管我正在查看这里提到的日志更改:更新:一旦我设置了
logging\u config\u class
,这个错误对我来说就消失了。请注意,对于任何使用Airflow 1.10的人,您需要设置Kyle链接中提到的
logging\u config\u class