Fatal编程技术网
  • postgresql/
  • Postgresql 气流+；库伯内特斯星系团+；Virtualbox:调度程序错误“；数据库连接无效。”；

Postgresql 气流+；库伯内特斯星系团+；Virtualbox:调度程序错误“；数据库连接无效。”；

postgresql kubernetes

Postgresql 气流+；库伯内特斯星系团+；Virtualbox:调度程序错误“；数据库连接无效。”；,postgresql,kubernetes,airflow-scheduler,Postgresql,Kubernetes,Airflow Scheduler,我正试图按照这个指南和官方存储库测试KubernetesPodOperator 我能够部署Kubernetes集群（./scripts/ci/Kubernetes/kube/deploy.sh-d persistent_mode），但调度程序和postgres容器之间似乎存在问题。计划程序无法从以下日志成功连接到postgres： $ kubectl logs airflow-698ff6b8cd-gdr7f scheduler `[2019-02-24 21:06:20,529] {s

我正试图按照这个指南和官方存储库测试KubernetesPodOperator

我能够部署Kubernetes集群（./scripts/ci/Kubernetes/kube/deploy.sh-d persistent_mode），但调度程序和postgres容器之间似乎存在问题。计划程序无法从以下日志成功连接到postgres：

    $ kubectl logs airflow-698ff6b8cd-gdr7f scheduler
`[2019-02-24 21:06:20,529] {settings.py:175} INFO - settings.configure_orm(): Usi
ng pool settings. pool_size=5, pool_recycle=1800, pid=1
[2019-02-24 21:06:20,830] {__init__.py:51} INFO - Using executor LocalExecutor
  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
[2019-02-24 21:06:21,317] {jobs.py:1490} INFO - Starting the scheduler
[2019-02-24 21:06:21,317] {jobs.py:1498} INFO - Processing each file at most -1 times
[2019-02-24 21:06:21,317] {jobs.py:1501} INFO - Searching for files in /root/airflow/dags
[2019-02-24 21:06:21,547] {jobs.py:1503} INFO - There are 22 files in /root/airflow/dags
[2019-02-24 21:06:21,688] {jobs.py:1548} INFO - Resetting orphaned tasks for active dag runs
[2019-02-24 21:06:22,059] {dag_processing.py:514} INFO - Launched DagFileProcessorManager with pid: 39
[2019-02-24 21:06:22,183] {settings.py:51} INFO - Configured default timezone <Timezone [UTC]>
[2019-02-24 21:06:22,200] {settings.py:175} INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800, pid=39
[2019-02-24 21:06:53,375] {sqlalchemy.py:81} WARNING - DB connection invalidated. Reconnecting...
[2019-02-24 21:07:04,396] {sqlalchemy.py:81} WARNING - DB connection invalidated. Reconnecting...
[2019-02-24 21:07:15,418] {sqlalchemy.py:81} WARNING - DB connection invalidated. Reconnecting...
[2019-02-24 21:07:26,448] {sqlalchemy.py:81} WARNING - DB connection invalidated. Reconnecting...
[2019-02-24 21:07:37,458] {sqlalchemy.py:81} WARNING - DB connection invalidated. Reconnecting...
[2019-02-24 21:07:48,472] {sqlalchemy.py:81} WARNING - DB connection invalidated. Reconnecting...`
postgres.yaml：

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: postgres-airflow
spec:
  replicas: 1
  template:
    metadata:
      labels:
        name: postgres-airflow
    spec:
      restartPolicy: Always
      containers:
        - name: postgres
          image: postgres
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 5432
              protocol: TCP
          volumeMounts:
            - name: dbvol
              mountPath: /var/lib/postgresql/data/pgdata
              subPath: pgdata
          env:
            - name: POSTGRES_USER
              value: root
            - name: POSTGRES_PASSWORD
              value: XXXX
            - name: POSTGRES_DB
              value: airflow
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata
            - name: POD_IP
              valueFrom: { fieldRef: { fieldPath: status.podIP } }
          livenessProbe:
            initialDelaySeconds: 60
            timeoutSeconds: 5
            failureThreshold: 5
            exec:
              command:
              - /bin/sh
              - -c
              - exec pg_isready --host $POD_IP ||  if [[ $(psql -qtAc --host $POD_IP 'SELECT pg_is_in_recovery') != "f" ]]; then  exit 0 else; exit 1; fi
          readinessProbe:
            initialDelaySeconds: 5
            timeoutSeconds: 5
            periodSeconds: 5
            exec:
              command:
              - /bin/sh
              - -c
              - exec pg_isready --host $POD_IP
          resources:
            requests:
              memory: .5Gi
              cpu: .5
      volumes:
        - name: dbvol
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: postgres-airflow
spec:
  clusterIP: None
  ports:
    - port: 5432
      targetPort: 5432
  selector:
    name: postgres-airflow
我是Kubernetes的新手，如果有任何帮助，我将不胜感激

编辑： 我能够检查initContainer日志，看起来POD之间的连接已经建立。此外，如果我打开Airflow web UI，我将看不到“最近的任务”、“Dag运行”，也看不到Dag的图形或树状视图，只看到加载圆图像

编辑：
非常感谢您的帮助，我在Web服务器上发现了几个错误的响应，比如static/dist/net:：ERR_ABORTED 404（未找到），所以我认为docker映像构建没有成功完成。我没有在./scripts/ci/kubernetes/docker/compile.sh上构建
python setup.py compile\u assets sdist-q
，而是在我的Dockerfile上添加了
RUN pip install--trusted host pypi.org--trusted host files.pythonhosted.org--trusted host pypi.python.org apache气流[芹菜，kubernetes，postgres，rabbitmq，ssh]

我认为这是因为您使用的是：LocalExecutor而不是kubernetesecutor

我只是按照同样的教程，无法重现您的问题：

[2019-03-01 16:07:22,053] {__init__.py:51} INFO - Using executor KubernetesExecutor
  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
[2019-03-01 16:07:29,564] {security.py:458} INFO - Start syncing user roles.
[2019-03-01 16:07:30,317] {security.py:196} INFO - Existing permissions for the role:Viewer within the database will persist.
[2019-03-01 16:07:31,281] {security.py:196} INFO - Existing permissions for the role:User within the database will persist.
[2019-03-01 16:07:31,709] {security.py:196} INFO - Existing permissions for the role:Op within the database will persist.
[2019-03-01 16:07:31,717] {security.py:374} INFO - Fetching a set of all permission, view_menu from FAB meta-table
[2019-03-01 16:07:33,357] {security.py:324} INFO - Cleaning faulty perms
[2019-03-01 16:07:37,878] {settings.py:175} INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800, pid=10
[2019-03-01 16:07:38 +0000] [10] [INFO] Starting gunicorn 19.9.0
[2019-03-01 16:07:38 +0000] [10] [INFO] Listening at: http://0.0.0.0:8080 (10)
[2019-03-01 16:07:38 +0000] [10] [INFO] Using worker: sync
[2019-03-01 16:07:38 +0000] [15] [INFO] Booting worker with pid: 15
[2019-03-01 16:07:38 +0000] [16] [INFO] Booting worker with pid: 16
[2019-03-01 16:07:38 +0000] [17] [INFO] Booting worker with pid: 17
[2019-03-01 16:07:38 +0000] [18] [INFO] Booting worker with pid: 18
Running the Gunicorn Server with:
Workers: 4 sync
Host: 0.0.0.0:8080
Timeout: 120
Logfiles: - -
=================================================================
[2019-03-01 16:07:57,078] {__init__.py:51} INFO - Using executor KubernetesExecutor
[2019-03-01 16:07:57,479] {__init__.py:51} INFO - Using executor KubernetesExecutor
[2019-03-01 16:07:57,490] {__init__.py:51} INFO - Using executor KubernetesExecutor
[2019-03-01 16:07:57,853] {__init__.py:51} INFO - Using executor KubernetesExecutor
[2019-03-01 16:08:02,411] {__init__.py:298} INFO - Filling up the DagBag from /root/airflow/dags
[2019-03-01 16:08:02,423] {__init__.py:298} INFO - Filling up the DagBag from /root/airflow/dags
[2019-03-01 16:08:02,420] {__init__.py:298} INFO - Filling up the DagBag from /root/airflow/dags
[2019-03-01 16:08:02,424] {__init__.py:298} INFO - Filling up the DagBag from /root/airflow/dags
[2019-03-01 16:08:08,480] {security.py:458} INFO - Start syncing user roles.
[2019-03-01 16:08:08,525] {security.py:458} INFO - Start syncing user roles.
[2019-03-01 16:08:09,250] {security.py:196} INFO - Existing permissions for the role:Viewer within the database will persist.
我的设置： minikube:v0.32.0
VirtualBox:5.2.22
Apache/Airflow构建于：b51712c

我认为这是因为您使用的是：LocalExecutor而不是KubernetesExecutor

我只是按照同样的教程，无法重现您的问题：

[2019-03-01 16:07:22,053] {__init__.py:51} INFO - Using executor KubernetesExecutor
  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
[2019-03-01 16:07:29,564] {security.py:458} INFO - Start syncing user roles.
[2019-03-01 16:07:30,317] {security.py:196} INFO - Existing permissions for the role:Viewer within the database will persist.
[2019-03-01 16:07:31,281] {security.py:196} INFO - Existing permissions for the role:User within the database will persist.
[2019-03-01 16:07:31,709] {security.py:196} INFO - Existing permissions for the role:Op within the database will persist.
[2019-03-01 16:07:31,717] {security.py:374} INFO - Fetching a set of all permission, view_menu from FAB meta-table
[2019-03-01 16:07:33,357] {security.py:324} INFO - Cleaning faulty perms
[2019-03-01 16:07:37,878] {settings.py:175} INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800, pid=10
[2019-03-01 16:07:38 +0000] [10] [INFO] Starting gunicorn 19.9.0
[2019-03-01 16:07:38 +0000] [10] [INFO] Listening at: http://0.0.0.0:8080 (10)
[2019-03-01 16:07:38 +0000] [10] [INFO] Using worker: sync
[2019-03-01 16:07:38 +0000] [15] [INFO] Booting worker with pid: 15
[2019-03-01 16:07:38 +0000] [16] [INFO] Booting worker with pid: 16
[2019-03-01 16:07:38 +0000] [17] [INFO] Booting worker with pid: 17
[2019-03-01 16:07:38 +0000] [18] [INFO] Booting worker with pid: 18
Running the Gunicorn Server with:
Workers: 4 sync
Host: 0.0.0.0:8080
Timeout: 120
Logfiles: - -
=================================================================
[2019-03-01 16:07:57,078] {__init__.py:51} INFO - Using executor KubernetesExecutor
[2019-03-01 16:07:57,479] {__init__.py:51} INFO - Using executor KubernetesExecutor
[2019-03-01 16:07:57,490] {__init__.py:51} INFO - Using executor KubernetesExecutor
[2019-03-01 16:07:57,853] {__init__.py:51} INFO - Using executor KubernetesExecutor
[2019-03-01 16:08:02,411] {__init__.py:298} INFO - Filling up the DagBag from /root/airflow/dags
[2019-03-01 16:08:02,423] {__init__.py:298} INFO - Filling up the DagBag from /root/airflow/dags
[2019-03-01 16:08:02,420] {__init__.py:298} INFO - Filling up the DagBag from /root/airflow/dags
[2019-03-01 16:08:02,424] {__init__.py:298} INFO - Filling up the DagBag from /root/airflow/dags
[2019-03-01 16:08:08,480] {security.py:458} INFO - Start syncing user roles.
[2019-03-01 16:08:08,525] {security.py:458} INFO - Start syncing user roles.
[2019-03-01 16:08:09,250] {security.py:196} INFO - Existing permissions for the role:Viewer within the database will persist.
我的设置： minikube:v0.32.0
VirtualBox:5.2.22
Apache/Airflow build from:b51712c

在我看来，很可能是您的configmap没有正确加载。你能试试“kubectl Descripte configmap airflow configmap”看看有没有什么东西吗？在我看来，很可能是你的configmap没有正确加载。你能试试“kubectl描述配置图气流配置图”看看有什么吗？