无法执行KuberneteSecutor

在中的项目之后，我正在尝试使用NFS服务器作为备份存储集成kubernetes executor。我有一个与NFS服务器链接的PV。Airflow Web服务器和调度器使用与Airflow pv绑定的PVC Airflow PVC。我已将我的dag文件放在NFS服务器/var/NFS/afflow/development/中。我还可以在Web服务器UI中看到新添加的DAG。但是，当我从UI执行DAG时，调度程序会为该任务触发一个新的POD，但新的工作POD无法运行

无法为pod tutorialprintdate-3E1A4443363E4C9F81FD6338CDB9873_development976b1e64-b46d-11e9-92af-025000000001装载卷：等待为pod开发/tutorialprintdate-3E1A4443363E4C9F81FD6338CDB9873附加或装载卷的超时。未安装卷的列表=[DAG]。未连接卷列表=[airflow dags airflow logs airflow config default token hjwth]

这是我的Web服务器和调度器部署文件

apiVersion: v1
kind: Service
metadata:
  name: airflow-webserver-svc
  namespace: development
spec:
  type: NodePort
  ports:
    - name: web
      protocol: TCP
      port: 8080
  selector:
    app: airflow-webserver-app
    namespace: development

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: airflow-webserver-dep
  namespace: development
spec:
  replicas: 1
  selector:
    matchLabels:
      app: airflow-webserver-app
      namespace: development
  template:
    metadata:
      labels:
        app: airflow-webserver-app
        namespace: development
    spec:
      restartPolicy: Always
      containers:
      - name: airflow-webserver-app
        image: airflow:externalConfigs
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
        args: ["-webserver"]
        env:
        - name: AIRFLOW_KUBE_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: AIRFLOW__CORE__FERNET_KEY
          valueFrom:
            secretKeyRef:
              name: airflow-secrets
              key: AIRFLOW__CORE__FERNET_KEY
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: airflow-secrets
              key: MYSQL_PASSWORD
        - name: MYSQL_PASSWORD
          valueFrom:
            secretKeyRef:
              name: airflow-secrets
              key: MYSQL_PASSWORD
        - name: DB_HOST
          value: mysql-svc.development.svc.cluster.local
        - name: DB_PORT
          value: "3306"
        - name: MYSQL_DATABASE
          value: airflow
        - name: MYSQL_USER
          value: airflow
        - name: MYSQL_PASSWORD
          value: airflow
        - name: AIRFLOW__CORE__EXECUTOR
          value: "KubernetesExecutor"
        volumeMounts:
        - name: airflow-config
          mountPath: /usr/local/airflow/airflow.cfg
          subPath: airflow.cfg
        - name: airflow-files
          mountPath: /usr/local/airflow/dags
          subPath: airflow/development/dags
        - name: airflow-files
          mountPath: /usr/local/airflow/plugins
          subPath: airflow/development/plugins
        - name: airflow-files
          mountPath: /usr/local/airflow/logs
          subPath: airflow/development/logs
        - name: airflow-files
          mountPath: /usr/local/airflow/temp
          subPath: airflow/development/temp
      volumes:
        - name: airflow-files
          persistentVolumeClaim:
            claimName: airflow-pvc
        - name: airflow-config
          configMap:
            name: airflow-config

调度程序yaml文件完全相同，只是容器args为args:[-scheduler]。这是我的airflow.cfg文件

apiVersion: v1
kind: ConfigMap
metadata:
  name: "airflow-config"
  namespace: development
data:
  airflow.cfg: |
    [core]
    airflow_home = /usr/local/airflow
    dags_folder = /usr/local/airflow/dags
    base_log_folder = /usr/local/airflow/logs
    executor = KubernetesExecutor
    plugins_folder = /usr/local/airflow/plugins

    load_examples = false

    [scheduler]
    child_process_log_directory = /usr/local/airflow/logs/scheduler

    [webserver]
    rbac = false

    [kubernetes]
    airflow_configmap =
    worker_container_repository = airflow
    worker_container_tag = externalConfigs
    worker_container_image_pull_policy = IfNotPresent
    delete_worker_pods = true
    dags_volume_claim = airflow-pvc
    dags_volume_subpath =
    logs_volume_claim = airflow-pvc
    logs_volume_subpath =

    env_from_configmap_ref = airflow-config
    env_from_secret_ref = airflow-secrets

    in_cluster = true
    namespace = development

    [kubernetes_node_selectors]
    # the key-value pairs to be given to worker pods.
    # the worker pods will be scheduled to the nodes of the specified key-value pairs.
    # should be supplied in the format: key = value

    [kubernetes_environment_variables]
    //the below configs gets overwritten by above [kubernetes] configs
    AIRFLOW__KUBERNETES__DAGS_VOLUME_CLAIM = airflow-pvc
    AIRFLOW__KUBERNETES__DAGS_VOLUME_SUBPATH = var/nfs/airflow/development/dags
    AIRFLOW__KUBERNETES__LOGS_VOLUME_CLAIM = airflow-pvc
    AIRFLOW__KUBERNETES__LOGS_VOLUME_SUBPATH = var/nfs/airflow/development/logs

    [kubernetes_secrets]
    AIRFLOW__CORE__SQL_ALCHEMY_CONN = airflow-secrets=AIRFLOW__CORE__SQL_ALCHEMY_CONN
    AIRFLOW_HOME = airflow-secrets=AIRFLOW_HOME

    [cli]
    api_client = airflow.api.client.json_client
    endpoint_url = https://airflow.crunchanalytics.cloud

    [api]
    auth_backend = airflow.api.auth.backend.default

    [admin]
    # ui to hide sensitive variable fields when set to true
    hide_sensitive_variable_fields = true

启动手动任务后，调度程序的日志告诉我KuberneteSecutorConfig执行时所有值都为None。好像它没有抓到配置？我几乎尝试了我所知道的一切，但都没能成功。谁能告诉我我错过了什么

[2019-08-01 14:44:22,944] {jobs.py:1341} INFO - Sending ('kubernetes_sample', 'run_this_first', datetime.datetime(2019, 8, 1, 13, 45, 51, 874679, tzinfo=<Timezone [UTC]>), 1) to executor with priority 3 and queue default
[2019-08-01 14:44:22,944] {base_executor.py:56} INFO - Adding to queue: airflow run kubernetes_sample run_this_first 2019-08-01T13:45:51.874679+00:00 --local -sd /usr/local/airflow/dags/airflow/development/dags/k8s_dag.py
[2019-08-01 14:44:22,948] {kubernetes_executor.py:629} INFO - Add task ('kubernetes_sample', 'run_this_first', datetime.datetime(2019, 8, 1, 13, 45, 51, 874679, tzinfo=<Timezone [UTC]>), 1) with command airflow run kubernetes_sample run_this_first 2019-08-01T13:45:51.874679+00:00 --local -sd /usr/local/airflow/dags/airflow/development/dags/k8s_dag.py with executor_config {}
[2019-08-01 14:44:22,949] {kubernetes_executor.py:379} INFO - Kubernetes job is (('kubernetes_sample', 'run_this_first', datetime.datetime(2019, 8, 1, 13, 45, 51, 874679, tzinfo=<Timezone [UTC]>), 1), 'airflow run kubernetes_sample run_this_first 2019-08-01T13:45:51.874679+00:00 --local -sd /usr/local/airflow/dags/airflow/development/dags/k8s_dag.py', KubernetesExecutorConfig(image=None, image_pull_policy=None, request_memory=None, request_cpu=None, limit_memory=None, limit_cpu=None, gcp_service_account_key=None, node_selectors=None, affinity=None, annotations={}, volumes=[], volume_mounts=[], tolerations=None))
[2019-08-01 14:44:23,042] {kubernetes_executor.py:292} INFO - Event: kubernetessamplerunthisfirst-7fe05ddb34aa4cb9a5604e420d5b60a3 had an event of type ADDED
[2019-08-01 14:44:23,046] {kubernetes_executor.py:324} INFO - Event: kubernetessamplerunthisfirst-7fe05ddb34aa4cb9a5604e420d5b60a3 Pending
[2019-08-01 14:44:23,049] {kubernetes_executor.py:292} INFO - Event: kubernetessamplerunthisfirst-7fe05ddb34aa4cb9a5604e420d5b60a3 had an event of type MODIFIED
[2019-08-01 14:44:23,049] {kubernetes_executor.py:324} INFO - Event: kubernetessamplerunthisfirst-7fe05ddb34aa4cb9a5604e420d5b60a3 Pending

---

使用气流版本：1.10.3

由于还没有答案，我将分享我目前的发现。在kubernetes部分下的my aiffort.conf中，我们要传递以下值

dags_volume_claim = airflow-pvc
dags_volume_subpath = airflow/development/dags
logs_volume_claim = airflow-pvc
logs_volume_subpath = airflow/development/logs

调度器如何从上述配置创建新pod的方式如下，仅提及卷和卷数量

"volumes": [
  {
    "name": "airflow-dags",
    "persistentVolumeClaim": {
      "claimName": "airflow-pvc"
    }
  },
  {
    "name": "airflow-logs",
    "persistentVolumeClaim": {
      "claimName": "airflow-pvc"
    }
  }],
"containers": [
    { ...
  "volumeMounts": [
      {
        "name": "airflow-dags",
        "readOnly": true,
        "mountPath": "/usr/local/airflow/dags",
        "subPath": "airflow/development/dags"
      },
      {
        "name": "airflow-logs",
        "mountPath": "/usr/local/airflow/logs",
        "subPath": "airflow/development/logs"
      }]
...}]

K8s不喜欢指向同一pvc气流的多个卷。为了解决这个问题，我需要为DAG和原木创建两个PVC和PV DAG\u volume\u claim=气流DAG PVC和原木\u volume\u claim=气流原木PVC，这两个都很好用

我不知道我正在使用的1.10.3的更新版本是否已经解决了这个问题。当ppl使用相同的PVC时，气流调度器应处理这种情况，然后创建一个具有单体积和两个体积数量的pod，参考该体积，例如

"volumes": [
  {
    "name": "airflow-dags-logs",    <--just an example name
    "persistentVolumeClaim": {
      "claimName": "airflow-pvc"
    }
  }
"containers": [
    { ...
  "volumeMounts": [
      {
        "name": "airflow-dags-logs",
        "readOnly": true,
        "mountPath": "/usr/local/airflow/dags",
        "subPath": "airflow/development/dags"     <--taken from configs
      },
      {
        "name": "airflow-dags-logs",
        "mountPath": "/usr/local/airflow/logs",
        "subPath": "airflow/development/logs"     <--taken from configs
      }]
...}]

---

我部署了一个具有上述配置的吊舱，它可以工作

你在Kubernetes日志中找到什么了吗，比如“异常”？Kubernetes日志。。。你是说节点日志？我在哪里可以看到它们？请运行thic命令：kubectl get pods-namespace kube system并查看是否有任何重新启动？您是否使用minikube？由于某些原因，airflow无法为DAG和日志安装相同的卷。如果我为DAG和日志分别创建单独的PV和PVC，这两个文件都指向单个NFS服务器，那么一切都正常。DAG文件如何复制到PV mounthPath/子路径？您是否有单独的CI/CD工作来下载DAG代码并将其复制到PV中？最初，我的做法与您所说的相同。但后来我的环境从图像中提取DAG。现在，管道使用新的/更新的DAG创建新的气流图像，并将airflow-configs.yaml dags_文件夹指向该图像下的指定位置。在我的例子中是/usr/local/aiffair/venv/lib/python3.6/site packages/my_dags。@AnumSheraz我正在尝试为Airfow配置KuberenetExecutor，并具有与您类似的配置，但我得到的服务令牌文件不存在错误。完整堆栈跟踪文件/usr/local/lib/python3.6/site-packages/kubernetes/config/incluster\u config.py，第64行，位于\u load\u config提升配置例外服务令牌文件中，不存在。kubernetes.config.config_exception.ConfigException:服务令牌文件不存在。您在气流1.10.3中遇到过类似的问题吗？在我的情况下，Airflow正在GKE上运行，并启用了工作负载标识。@pradeep您是如何修复错误服务令牌文件不存在的？@pradeep/@AnumSheraz我遇到了相同的错误-？服务令牌文件不存在。有什么帮助吗？

"volumes": [
  {
    "name": "airflow-dags-logs",    <--just an example name
    "persistentVolumeClaim": {
      "claimName": "airflow-pvc"
    }
  }
"containers": [
    { ...
  "volumeMounts": [
      {
        "name": "airflow-dags-logs",
        "readOnly": true,
        "mountPath": "/usr/local/airflow/dags",
        "subPath": "airflow/development/dags"     <--taken from configs
      },
      {
        "name": "airflow-dags-logs",
        "mountPath": "/usr/local/airflow/logs",
        "subPath": "airflow/development/logs"     <--taken from configs
      }]
...}]