如何为Kubernetes Airflow worker pod启动KubernetesPodOperator创建kubeconfig

我正在Kubernetes引擎中设置气流，现在我有以下（运行）吊舱：

• postgres（带有装入的

  PersistentVolumeClaim

  ）
• 花
• web（气流仪表板）
• 兔子
• 调度程序
• 工人

从Airflow，我想运行一个启动pod的任务，在本例中，它从SFTP服务器下载一些文件。但是，应启动此新pod的气流中的

KubernetesPodOperator

无法运行，因为找不到kubeconfig

气流工人的配置如下所示。除了不同的

args

之外，其他气流吊舱完全相同

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: worker
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: airflow
        tier: worker
    spec:
      restartPolicy: Always
      containers:
        - name: worker
          image: my-gcp-project/kubernetes-airflow-in-container-registry:v1
          imagePullPolicy: IfNotPresent
          env:
            - name: AIRFLOW_HOME
              value: "/usr/local/airflow"
          args: ["worker"]

KubernetesPodOperator

配置如下：

maybe\u download=KubernetesPodOperator(
task_id='maybe_download_from_sftp'，
image='some/image:v1'，
namespace='default'，
name='maybe-download-from-sftp'，
参数=['sftp_下载']，
图像\u pull\u policy='IfNotPresent'，
dag=dag，
触发器\u rule='dummy'，
)

以下错误显示pod上没有kubeconfig

[2019-01-24 12:37:04,706] {models.py:1789} INFO - All retries failed; marking task as FAILED
[2019-01-24 12:37:04,722] {base_task_runner.py:101} INFO - Job 8: Subtask maybe_download_from_sftp Traceback (most recent call last):
[2019-01-24 12:37:04,722] {base_task_runner.py:101} INFO - Job 8: Subtask maybe_download_from_sftp   File "/usr/local/bin/airflow", line 32, in <module>
[2019-01-24 12:37:04,722] {base_task_runner.py:101} INFO - Job 8: Subtask maybe_download_from_sftp     args.func(args)
[2019-01-24 12:37:04,722] {base_task_runner.py:101} INFO - Job 8: Subtask maybe_download_from_sftp   File "/usr/local/lib/python3.6/site-packages/airflow/utils/cli.py", line 74, in wrapper
[2019-01-24 12:37:04,722] {base_task_runner.py:101} INFO - Job 8: Subtask maybe_download_from_sftp     return f(*args, **kwargs)
[2019-01-24 12:37:04,722] {base_task_runner.py:101} INFO - Job 8: Subtask maybe_download_from_sftp   File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py", line 490, in run
[2019-01-24 12:37:04,722] {base_task_runner.py:101} INFO - Job 8: Subtask maybe_download_from_sftp     _run(args, dag, ti)
[2019-01-24 12:37:04,722] {base_task_runner.py:101} INFO - Job 8: Subtask maybe_download_from_sftp   File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py", line 406, in _run
[2019-01-24 12:37:04,722] {base_task_runner.py:101} INFO - Job 8: Subtask maybe_download_from_sftp     pool=args.pool,
[2019-01-24 12:37:04,722] {base_task_runner.py:101} INFO - Job 8: Subtask maybe_download_from_sftp   File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 74, in wrapper
[2019-01-24 12:37:04,722] {base_task_runner.py:101} INFO - Job 8: Subtask maybe_download_from_sftp     return func(*args, **kwargs)
[2019-01-24 12:37:04,722] {base_task_runner.py:101} INFO - Job 8: Subtask maybe_download_from_sftp   File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 1659, in _run_raw_task
[2019-01-24 12:37:04,723] {base_task_runner.py:101} INFO - Job 8: Subtask maybe_download_from_sftp     result = task_copy.execute(context=context)
[2019-01-24 12:37:04,723] {base_task_runner.py:101} INFO - Job 8: Subtask maybe_download_from_sftp   File "/usr/local/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", line 90, in execute
[2019-01-24 12:37:04,723] {base_task_runner.py:101} INFO - Job 8: Subtask maybe_download_from_sftp     config_file=self.config_file)
[2019-01-24 12:37:04,723] {base_task_runner.py:101} INFO - Job 8: Subtask maybe_download_from_sftp   File "/usr/local/lib/python3.6/site-packages/airflow/contrib/kubernetes/kube_client.py", line 51, in get_kube_client
[2019-01-24 12:37:04,723] {base_task_runner.py:101} INFO - Job 8: Subtask maybe_download_from_sftp     return _load_kube_config(in_cluster, cluster_context, config_file)
[2019-01-24 12:37:04,723] {base_task_runner.py:101} INFO - Job 8: Subtask maybe_download_from_sftp   File "/usr/local/lib/python3.6/site-packages/airflow/contrib/kubernetes/kube_client.py", line 38, in _load_kube_config
[2019-01-24 12:37:04,723] {base_task_runner.py:101} INFO - Job 8: Subtask maybe_download_from_sftp     config.load_kube_config(config_file=config_file, context=cluster_context)
[2019-01-24 12:37:04,723] {base_task_runner.py:101} INFO - Job 8: Subtask maybe_download_from_sftp   File "/usr/local/airflow/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 537, inload_kube_config
[2019-01-24 12:37:04,723] {base_task_runner.py:101} INFO - Job 8: Subtask maybe_download_from_sftp     config_persister=config_persister)
[2019-01-24 12:37:04,723] {base_task_runner.py:101} INFO - Job 8: Subtask maybe_download_from_sftp   File "/usr/local/airflow/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 494, in_get_kube_config_loader_for_yaml_file
[2019-01-24 12:37:04,723] {base_task_runner.py:101} INFO - Job 8: Subtask maybe_download_from_sftp     with open(filename) as f:
[2019-01-24 12:37:04,723] {base_task_runner.py:101} INFO - Job 8: Subtask maybe_download_from_sftp FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/airflow/.kube/config'
[2019-01-24 12:37:08,300] {logging_mixin.py:95} INFO - [2019-01-24 12:37:08,299] {jobs.py:2627} INFO - Task exited with return code 1

[2019-01-24 12:37:04706]{models.py:1789}INFO-所有重试均失败；将任务标记为失败
[2019-01-2412:37:04722]{base_task_runner.py:101}信息-作业8：子任务可能从sftp回溯下载（最后一次调用）：
[2019-01-2412:37:04722]{base_task_runner.py:101}信息-作业8：子任务可能从文件“/usr/local/bin/afflow”下载，第32行，在
[2019-01-2412:37:04722]{base_task_runner.py:101}信息-作业8：子任务可能从_sftpargs.func（args）下载
[2019-01-24 12:37:04722]{base_task_runner.py:101}信息-作业8：子任务可能从包装器中的sftp文件“/usr/local/lib/python3.6/site packages/aiffair/utils/cli.py”第74行下载
[2019-01-2412:37:04722]{base_task_runner.py:101}信息-作业8：子任务可能从sftp返回f下载（*args，**kwargs）
[2019-01-24 12:37:04722]{base_task_runner.py:101}信息-作业8：子任务可能从sftp文件“/usr/local/lib/python3.6/site packages/afflow/bin/cli.py”下载，第490行，运行中
[2019-01-2412:37:04722]{base_task_runner.py:101}信息-作业8：子任务可能从sftp_run下载（args、dag、ti）
[2019-01-24 12:37:04722]{base_task_runner.py:101}信息-作业8：子任务可能从sftp文件“/usr/local/lib/python3.6/site packages/afflow/bin/cli.py”下载，第406行，正在运行
[2019-01-2412:37:04722]{base_task_runner.py:101}信息-作业8：子任务可能从\u sftp pool=args.pool下载，
[2019-01-24 12:37:04722]{base_task_runner.py:101}信息-作业8：子任务可能从包装器中的sftp文件“/usr/local/lib/python3.6/site packages/aiffair/utils/db.py”下载
[2019-01-2412:37:04722]{base_task_runner.py:101}信息-作业8：子任务可能从sftp return func下载（*args，**kwargs）
[2019-01-24 12:37:04722]{base_task_runner.py:101}信息-作业8：子任务可能从_sftp文件“/usr/local/lib/python3.6/site packages/aiffair/models.py”下载，第1659行，在_run_raw_task中
[2019-01-2412:37:04723]{base_task_runner.py:101}信息-作业8：子任务可能从任务下载结果=任务副本.execute（上下文=上下文）
[2019-01-24 12:37:04723]{base_task_runner.py:101}信息-作业8：子任务可能从_sftp文件“/usr/local/lib/python3.6/site packages/afflow/contrib/operators/kubernetes_pod_operator.py”下载，第90行，执行
[2019-01-2412:37:04723]{base\u task\u runner.py:101}信息-作业8：子任务可能\u从\u sftp config\u file=self.config\u文件下载\u）
[2019-01-24 12:37:04723]{base_task_runner.py:101}信息-作业8：子任务可能从_sftp文件“/usr/local/lib/python3.6/site packages/aiffair/contrib/kubernetes/kube_client.py”下载，第51行，在get_kube_____客户端中
[2019-01-24 12:37:04723]{base_task_runner.py:101}信息-作业8：子任务可能从sftp返回下载配置（在集群中，集群上下文，配置文件中）
[2019-01-24 12:37:04723]{base_task_runner.py:101}信息-作业8：子任务可能从_sftp文件/usr/local/lib/python3.6/site packages/afflow/contrib/kubernetes/kube_client.py”下载，第38行，在_load_kube_配置中
[2019-01-24 12:37:04723]{base_task_runner.py:101}信息-作业8：子任务可能从sftp config.load_kube_config下载（配置文件=配置文件，上下文=集群上下文）
[2019-01-24 12:37:04723]{base_task_runner.py:101}信息-作业8：子任务可能从_sftp文件“/usr/local/aiffair/.local/lib/python3.6/site packages/kubernetes/config/kube_config.py”下载，第537行，inload_kube__配置
[2019-01-2412:37:04723]{base\u task\u runner.py:101}信息-作业8：子任务可能\u从\u sftp config\u persister=config\u persister下载\u）
[2019-01-24 12:37:04723]{base_task_runner.py:101}信息-作业8：子任务可能从\u sftp文件/usr/local/aiffair/.local/lib/python3.6/site packages/kubernetes/config/kube\u config.py下载\u，第494行，位于\u-get\u kube\u-kube\u-config\u-loader\u yaml\u文件中
[2019-01-24 12:37:04723]{base_task_runner.py:101}信息-作业8：子任务可能从sftp下载，打开（文件名）为f:
[2019-01-24 12:37:04723]{base_task_runner.py:101}信息-作业8：子任务可能从_sftp FileNotFoundError:[Errno 2]没有这样的文件或目录：'/usr/local/afflow/.kube/config'
[2019-01-2412:37:08300]{logging_mixin.py:95}信息-[2019-01-2412:37:08299]{jobs.py:2627}信息-任务退出，返回代码为1

我希望pod启动并“自动”包含它所在的Kubernetes集群的上下文-如果这有意义的话。我觉得我错过了一些基本的东西。有人能帮忙吗？

如中所述，您需要

in_cluster=True

通知KPO它实际上在集群中

实际上，我建议使用Airflow归档一个bug，因为Airflow可以很容易地检测到它在集群内部运行的事实，并且应该有一个比y更理智的默认值