composer中的Python冲突会使整个kubernetes集群崩溃,如何修复它?

composer中的Python冲突会使整个kubernetes集群崩溃,如何修复它?,kubernetes,google-cloud-platform,google-kubernetes-engine,google-cloud-composer,Kubernetes,Google Cloud Platform,Google Kubernetes Engine,Google Cloud Composer,让我先说一句,我对库伯内特斯并不精通 昨天,为了Google Cloud Composer中的一个DAG,我们不得不安装/更新python依赖项。我不确定这是否是原因,但整个作曲家环境在这之后崩溃了 在浏览Logs Explorer时,我在计划程序和工作程序中发现以下错误: Traceback (most recent call last): File "/usr/local/bin/airflow", line 4, in <module> __im

让我先说一句,我对库伯内特斯并不精通

昨天,为了Google Cloud Composer中的一个DAG,我们不得不安装/更新python依赖项。我不确定这是否是原因,但整个作曲家环境在这之后崩溃了

在浏览Logs Explorer时,我在计划程序和工作程序中发现以下错误:

Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 4, in <module>
    __import__('pkg_resources').require('apache-airflow===1.10.2-composer')
  File "/opt/python3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3105, in <module>
    @_call_aside
  File "/opt/python3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3089, in _call_aside
    f(*args, **kwargs)
  File "/opt/python3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3118, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/opt/python3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 580, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/opt/python3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 593, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/opt/python3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 781, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'urllib3<1.25,>=1.21.1' distribution was not found and is required by requests
我得到:

Defaulting container name to airflow-worker.
Use 'kubectl describe pod/airflow-worker-6cdfc68fd4-4k4jm -n composer-1-7-5-airflow-1-10-2-2d974007' to see all of the containers in this pod.
error: unable to upgrade connection: container not found ("airflow-worker")
下面是kubectl get pods的一个结果——所有名称空间

NAMESPACE                                NAME                                                             READY   STATUS             RESTARTS   AGE
composer-1-7-5-airflow-1-10-2-2d974007   airflow-scheduler-574bcfbd47-gqnkp                               1/2     CrashLoopBackOff   234        19h
composer-1-7-5-airflow-1-10-2-2d974007   airflow-worker-6cdfc68fd4-4k4jm                                  1/2     CrashLoopBackOff   233        19h
composer-1-7-5-airflow-1-10-2-2d974007   airflow-worker-6cdfc68fd4-fwz5h                                  1/2     CrashLoopBackOff   232        19h
composer-1-7-5-airflow-1-10-2-2d974007   airflow-worker-6cdfc68fd4-vl25g                                  1/2     CrashLoopBackOff   233        19h
composer-1-7-5-airflow-1-10-2-2d974007   airflow-worker-75ff8dbb56-qxc7j                                  0/2     Evicted            0          21d
default                                  airflow-monitoring-5bd5f64896-g6q8v                              1/1     Running            0          21d
default                                  airflow-redis-0                                                  1/1     Running            0          21d
default                                  airflow-sqlproxy-577bbc7577-mxv5p                                1/1     Running            0          21d
default                                  composer-agent-7c388f77-840c-40c8-be09-66303d721742-xxqlf        0/1     Completed          0          66m
default                                  composer-agent-9fd51464-6ed2-4f3d-9714-762ea723cb61-5lc2s        0/1     Completed          0          19h
default                                  composer-fluentd-daemon-gm7vr                                    1/1     Running            0          21d
default                                  composer-fluentd-daemon-srw2h                                    1/1     Running            4          21d
default                                  composer-fluentd-daemon-swzgc                                    1/1     Running            0          21d
kube-system                              heapster-gke-7b4f99dd5f-8d2fx                                    3/3     Running            0          21d
kube-system                              kube-dns-5995c95f64-7hn2s                                        4/4     Running            0          21d
kube-system                              kube-dns-5995c95f64-dwlfv                                        4/4     Running            0          21d
kube-system                              kube-dns-autoscaler-8687c64fc-fpvm9                              1/1     Running            0          21d
kube-system                              kube-proxy-gke-europe-west1-pipelin-default-pool-a8d0baad-7zcs   1/1     Running            0          21d
kube-system                              kube-proxy-gke-europe-west1-pipelin-default-pool-a8d0baad-h2u4   1/1     Running            0          21d
kube-system                              kube-proxy-gke-europe-west1-pipelin-default-pool-a8d0baad-i3kz   1/1     Running            1          7d20h
kube-system                              l7-default-backend-fd59995cd-hkk6z                               1/1     Running            0          21d
kube-system                              metrics-server-v0.3.1-5c6fbf777-27hgk                            2/2     Running            0          21CrashLoopBackOff
kube-system                              prometheus-to-sd-5z9sw                                           2/2     Running            0          21d
kube-system                              prometheus-to-sd-8dsr8                                           2/2     Running            2          21d
kube-system                              prometheus-to-sd-f55cl                                           2/2     Running            0          21d
在谷歌上搜索一下,我发现在Kubernetes中,
CrashLoopBackOff
错误可能很难诊断/解决。由于我对这项技术不太熟悉,我请求你在这件事上给予帮助

  • 我如何连接到工人
  • 如何安装/更新从这个worker执行的python环境的库?在GoogleCloudComposer中,这是解决python依赖性问题的正确方法吗

  • 如果你能帮忙的话,能得到尽可能多的细节就太好了。谢谢。

    能否将Cloud Composer环境更新为最新版本的Composer-1.12.5-airflow-1.10.10?它随附urllib3==1.25.8版本。@Ines我已经考虑过了,但它太危险了,因为我不想在我的错误环境中丢失一些我不知道的东西(例如连接、变量、可能因墨菲定律而消失的文件)。。。因此,我现在做的是创建一个新的环境,并尝试将我的错误环境中的内容一点一点地导入到我的新环境中
    NAMESPACE                                NAME                                                             READY   STATUS             RESTARTS   AGE
    composer-1-7-5-airflow-1-10-2-2d974007   airflow-scheduler-574bcfbd47-gqnkp                               1/2     CrashLoopBackOff   234        19h
    composer-1-7-5-airflow-1-10-2-2d974007   airflow-worker-6cdfc68fd4-4k4jm                                  1/2     CrashLoopBackOff   233        19h
    composer-1-7-5-airflow-1-10-2-2d974007   airflow-worker-6cdfc68fd4-fwz5h                                  1/2     CrashLoopBackOff   232        19h
    composer-1-7-5-airflow-1-10-2-2d974007   airflow-worker-6cdfc68fd4-vl25g                                  1/2     CrashLoopBackOff   233        19h
    composer-1-7-5-airflow-1-10-2-2d974007   airflow-worker-75ff8dbb56-qxc7j                                  0/2     Evicted            0          21d
    default                                  airflow-monitoring-5bd5f64896-g6q8v                              1/1     Running            0          21d
    default                                  airflow-redis-0                                                  1/1     Running            0          21d
    default                                  airflow-sqlproxy-577bbc7577-mxv5p                                1/1     Running            0          21d
    default                                  composer-agent-7c388f77-840c-40c8-be09-66303d721742-xxqlf        0/1     Completed          0          66m
    default                                  composer-agent-9fd51464-6ed2-4f3d-9714-762ea723cb61-5lc2s        0/1     Completed          0          19h
    default                                  composer-fluentd-daemon-gm7vr                                    1/1     Running            0          21d
    default                                  composer-fluentd-daemon-srw2h                                    1/1     Running            4          21d
    default                                  composer-fluentd-daemon-swzgc                                    1/1     Running            0          21d
    kube-system                              heapster-gke-7b4f99dd5f-8d2fx                                    3/3     Running            0          21d
    kube-system                              kube-dns-5995c95f64-7hn2s                                        4/4     Running            0          21d
    kube-system                              kube-dns-5995c95f64-dwlfv                                        4/4     Running            0          21d
    kube-system                              kube-dns-autoscaler-8687c64fc-fpvm9                              1/1     Running            0          21d
    kube-system                              kube-proxy-gke-europe-west1-pipelin-default-pool-a8d0baad-7zcs   1/1     Running            0          21d
    kube-system                              kube-proxy-gke-europe-west1-pipelin-default-pool-a8d0baad-h2u4   1/1     Running            0          21d
    kube-system                              kube-proxy-gke-europe-west1-pipelin-default-pool-a8d0baad-i3kz   1/1     Running            1          7d20h
    kube-system                              l7-default-backend-fd59995cd-hkk6z                               1/1     Running            0          21d
    kube-system                              metrics-server-v0.3.1-5c6fbf777-27hgk                            2/2     Running            0          21CrashLoopBackOff
    kube-system                              prometheus-to-sd-5z9sw                                           2/2     Running            0          21d
    kube-system                              prometheus-to-sd-8dsr8                                           2/2     Running            2          21d
    kube-system                              prometheus-to-sd-f55cl                                           2/2     Running            0          21d