无法在Ubuntu容器中使用DockerOperator从Airflow容器执行命令

我对Docker+Airflow非常陌生。下面是我正在努力实现的目标

我有4个服务，如下面的合成文件所示。3个与气流有关，一个作为测试实例。与气流相关的容器：

气流数据库

，

气流Web服务器

，

气流调度器

能够相互通信，并且我能够运行示例DAG。 现在我添加了第四个服务（ubuntu），我正试图使用DockerOperator从DAG发送一个简单的命令“/bin/sleep 10”（下面是DAG文件）。但由于某些原因，我收到了“权限被拒绝”消息（也附上了DAG错误文件）

如果我从localhost运行气流，而不是从docker容器内部运行气流，它就会工作 不知道我错过了什么。以下是我尝试过的一些方法：

• 在docker_url中：

已替换<代码>unix://var/run/docker.sock与

tcp://172.20.0.1

认为它可以通过docker主机ip解决

已使用gateway.host.internal

甚至删除了运营商的docker_url选项，但意识到它无论如何会被默认为unix://var/run/docker.sock

尝试了一系列的组合，tcp://172.20.0.1:2376, tcp://172.20.0.1:2375

将主机端口映射到Ubuntu，如8085:8085等

• 可能是airflow web服务器的airflow用户被Ubuntu开除了
• 因此，在Ubuntu容器中创建了一个组，并向其中添加了一个用户——不，也不起作用
• api_版本：选项“自动”也不起作用，并不断给出版本未找到错误。因此，我必须使用1.41硬编码，因为我在

  docker version

  命令中发现了这一点。不确定这是不是应该的

提前感谢您在其他方面给予我的帮助，我可以尝试使这项工作顺利进行：）

docker compose.yml

version: '3.2'

services:
# Ubuntu Container  
  ubuntu:
    image: ubuntu
    networks:
      - mynetwork

# Airflow Database
  airflow-database:
    image: postgres:12
    env_file:
      - .env
    ports:
      - 5432:5432
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./airflow/database/data:/var/lib/postgresql/data/pgdata
      - ./airflow/database/logs:/var/lib/postgresql/data/log
    command: >
     postgres
       -c listen_addresses=*
       -c logging_collector=on
       -c log_destination=stderr
       -c max_connections=200
    networks:
      - mynetwork

# Airflow DB Init
  initdb:
      image: apache/airflow:2.0.0-python3.8
      env_file:
        - .env
      depends_on:
        - airflow-database
      volumes:
        - /var/run/docker.sock:/var/run/docker.sock
        - ./airflow/metadata-airflow/dags:/opt/airflow/dags
        - ./airflow/logs:/opt/airflow/logs
      entrypoint: /bin/bash
      command: -c "airflow db init && airflow users create --firstname admin --lastname admin --email admin@admin.com --password admin --username admin --role Admin"
      networks:
        - mynetwork

# Airflow Webserver
  airflow-webserver:
    image: apache/airflow:2.0.0-python3.8
    env_file:
      - .env
    depends_on:
      - airflow-database
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./airflow/metadata-airflow/dags:/opt/airflow/dags
      - ./airflow/logs:/opt/airflow/logs
    ports:
      - 8080:8080
    deploy:
      restart_policy:
        condition: on-failure
        delay: 8s
        max_attempts: 3
    command: webserver
    healthcheck:
      test: ["CMD-SHELL", "[ -f /opt/airflow/airflow-webserver.pid ]"]
      interval: 30s
      timeout: 30s
      retries: 3
    networks:
      - mynetwork

# Airflow Scheduler
  airflow-scheduler:
    image: apache/airflow:2.0.0-python3.8
    env_file:
      - .env
    depends_on:
      - airflow-database
      - airflow-webserver
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./airflow/metadata-airflow/dags:/opt/airflow/dags
      - ./airflow/logs:/opt/airflow/logs
    deploy:
      restart_policy:
        condition: on-failure
        delay: 8s
        max_attempts: 3
    command: scheduler
    networks:
      - mynetwork

networks:
  mynetwork:

DAG文件

from datetime import timedelta
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.providers.docker.operators.docker import DockerOperator
from airflow.utils.dates import days_ago


default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'email': ['airflow@example.com'],
    'email_on_failure': False,
    'email_on_retry': False,
}

dag = DAG(
    'docker_sample',
    default_args=default_args,
    schedule_interval=None,
    start_date=days_ago(2),
)

t1 = DockerOperator(
    task_id='docker_op_tester',
    api_version='auto', 
    image='ubuntu',
    docker_url='unix://var/run/docker.sock',
    auto_remove=True,
    command=[
        "/bin/bash",
        "-c",
        "/bin/sleep 30; "],
    network_mode='bridge',
    dag=dag,
)


t1

DAG错误日志

*** Reading local file: /opt/airflow/logs/docker_sample/docker_op_tester/2021-01-09T05:16:17.174981+00:00/1.log
[2021-01-09 05:16:26,726] {taskinstance.py:826} INFO - Dependencies all met for <TaskInstance: docker_sample.docker_op_tester 2021-01-09T05:16:17.174981+00:00 [queued]>
[2021-01-09 05:16:26,774] {taskinstance.py:826} INFO - Dependencies all met for <TaskInstance: docker_sample.docker_op_tester 2021-01-09T05:16:17.174981+00:00 [queued]>
[2021-01-09 05:16:26,775] {taskinstance.py:1017} INFO - 
--------------------------------------------------------------------------------
[2021-01-09 05:16:26,776] {taskinstance.py:1018} INFO - Starting attempt 1 of 1
[2021-01-09 05:16:26,776] {taskinstance.py:1019} INFO - 
--------------------------------------------------------------------------------
[2021-01-09 05:16:26,790] {taskinstance.py:1038} INFO - Executing <Task(DockerOperator): docker_op_tester> on 2021-01-09T05:16:17.174981+00:00
[2021-01-09 05:16:26,794] {standard_task_runner.py:51} INFO - Started process 1057 to run task
[2021-01-09 05:16:26,817] {standard_task_runner.py:75} INFO - Running: ['airflow', 'tasks', 'run', 'docker_sample', 'docker_op_tester', '2021-01-09T05:16:17.174981+00:00', '--job-id', '360', '--pool', 'default_pool', '--raw', '--subdir', 'DAGS_FOLDER/example_docker.py', '--cfg-path', '/tmp/tmp4phq52dv']
[2021-01-09 05:16:26,821] {standard_task_runner.py:76} INFO - Job 360: Subtask docker_op_tester
[2021-01-09 05:16:26,932] {logging_mixin.py:103} INFO - Running <TaskInstance: docker_sample.docker_op_tester 2021-01-09T05:16:17.174981+00:00 [running]> on host 367f0fc7d092
[2021-01-09 05:16:27,036] {taskinstance.py:1230} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_EMAIL=airflow@example.com
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=docker_sample
AIRFLOW_CTX_TASK_ID=docker_op_tester
AIRFLOW_CTX_EXECUTION_DATE=2021-01-09T05:16:17.174981+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2021-01-09T05:16:17.174981+00:00
[2021-01-09 05:16:27,054] {taskinstance.py:1396} ERROR - ('Connection aborted.', PermissionError(13, 'Permission denied'))
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
PermissionError: [Errno 13] Permission denied

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
    retries = retries.increment(
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/util/retry.py", line 410, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', PermissionError(13, 'Permission denied'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1086, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1260, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1300, in _execute_task
    result = task_copy.execute(context=context)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/docker/operators/docker.py", line 286, in execute
    if self.force_pull or not self.cli.images(name=self.image):
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/image.py", line 89, in images
    res = self._result(self._get(self._url("/images/json"), params=params),
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 230, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/home/airflow/.local/lib/python3.8/site-packages/requests/sessions.py", line 543, in get
    return self.request('GET', url, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', PermissionError(13, 'Permission denied'))
[2021-01-09 05:16:27,073] {taskinstance.py:1433} INFO - Marking task as FAILED. dag_id=docker_sample, task_id=docker_op_tester, execution_date=20210109T051617, start_date=20210109T051626, end_date=20210109T051627
[2021-01-09 05:16:27,136] {local_task_job.py:118} INFO - Task exited with return code 1

***正在读取本地文件：/opt/aiffort/logs/docker_sample/docker_op_tester/2021-01-09T05:16:17.174981+00:00/1.log
[2021-01-09 05:16:26726]{taskinstance.py:826}信息-所有依赖项都满足以下条件：
[2021-01-09 05:16:26774]{taskinstance.py:826}信息-所有依赖项都满足以下条件：
[2021-01-09 05:16:26775]{taskinstance.py:1017}信息-
--------------------------------------------------------------------------------
[2021-01-09 05:16:26776]{taskinstance.py:1018}信息-开始尝试1次中的第1次
[2021-01-09 05:16:26776]{taskinstance.py:1019}信息-
--------------------------------------------------------------------------------
[2021-01-09 05:16:26790]{taskinstance.py:1038}INFO-在2021-01-09T05:16:17.174981+00:00执行
[2021-01-09 05:16:26794]{standard_task_runner.py:51}INFO-已启动进程1057以运行任务
[2021-01-09 05:16:26817]{standard_task_runner.py:75}运行信息：['aiffort'、'tasks'、'run'、'docker_sample'、'docker_op_tester'、'2021-01-09T05:16:17.174981+00:00'、'--job id'、'360'、'-pool'、'default_pool'、'-raw'、'-subdir'、'DAGS'、'DAGS_文件夹/example_docker.py'、'cfg'、'cfg path'、'tmp/tmp4; qdv']
[2021-01-09 05:16:26821]{standard_task_runner.py:76}信息-作业360：子任务docker_op_tester
[2021-01-09 05:16:26932]{logging_mixin.py:103}信息-在主机367f0fc7d092上运行
[2021-01-09 05:16:27036]{taskinstance.py:1230}信息-导出以下环境变量：
气流\u CTX\u DAG\u电子邮件=airflow@example.com
气流\u CTX\u DAG\u所有者=气流
气流\u CTX\u DAG\u ID=docker\u样本
气流\u CTX\u任务\u ID=docker\u op\u测试仪
空气流量执行日期=2021-01-09T05:16:17.174981+00:00
气流（CTX）流量（DAG）运行ID=手动（2021-01-09T05:16:17.174981+00:00）
[2021-01-09 05:16:27054]{taskinstance.py:1396}错误-（'Connection aborted'，PermissionError（13，'Permission denied'））
回溯（最近一次呼叫最后一次）：
urlopen中第670行的文件“/home/aiffort/.local/lib/python3.8/site packages/urlib3/connectionpool.py”
httplib\u response=self.\u发出请求(
文件“/home/afflow/.local/lib/python3.8/site packages/urllib3/connectionpool.py”，第392行，在请求中
conn.request（方法，url，**httplib\u request\u kw）
请求中的文件“/usr/local/lib/python3.8/http/client.py”，第1255行
self.\u发送\u请求（方法、url、正文、标题、编码\u分块）
文件“/usr/local/lib/python3.8/http/client.py”，第1301行，在发送请求中
self.endheaders（body，encode\u chunked=encode\u chunked）
文件“/usr/local/lib/python3.8/http/client.py”，第1250行，在endheaders中
self.\u发送\u输出（消息体，encode\u chunked=encode\u chunked）
文件“/usr/local/lib/python3.8/http/client.py”，第1010行，在发送输出中
self.send（msg）
文件“/usr/local/lib/python3.8/http/client.py”，第950行，在send中
self.connect（）
文件“/home/afflow/.local/lib/python3.8/site packages/docker/transport/unixconn.py”，第43行，在connect中
sock.connect（self.unix\u套接字）
PermissionError:[Errno 13]权限被拒绝
在处理上述异常期间，发生了另一个异常：
回溯（最近一次呼叫最后一次）：
文件“/home/afflow/.local/lib/python3.8/site packages/requests/adapters.py”，第439行，在send中
resp=conn.urlopen(
urlopen中的文件“/home/afflow/.local/lib/python3.8/site packages/urlib3/connectionpool.py”，第726行
重试次数=重试次数。增量(
文件“/home/afflow/.local/lib/python3.8/site packages/urllib3/util/retry.py”，第410行，增量
升起六个。重新升起（类型（错误），错误，_stacktrace）
文件“/home/afflow/.local/lib/python3.8/site packages/urllib3/packages/six.py”，第734行，在reraise中
通过_回溯（tb）提升值
urlopen中第670行的文件“/home/aiffort/.local/lib/python3.8/site packages/urlib3/connectionpool.py”
httplib\u response=self.\u发出请求(
文件“/home/afflow/.local/lib/python3.8/site packages/urllib3/connectionpool.py”，第392行，在请求中
conn.request（方法，url，**httpl
 image: apache/airflow:2.0.0-python3.8
 group_add:
   - 1001

AIRFLOW__CORE__ENABLE_XCOM_PICKLING=True