如何使用airflow检查长时间运行的http任务的状态？

我的用例是使用气流控制跨微服务的许多预定作业。我尝试的解决方案是将airflow用作集中式作业调度器，并通过http调用触发作业。其中一些作业将运行很长时间，例如超过10分钟或最多1小时

---

我如何定期检查这些作业的状态？如果远程任务已完成，但不知道作业是否成功，该怎么办？我是否可以将作业完成事件发布到kafka，并使airflow在kafka上侦听以获取作业状态？

使用airflow和您的微服务，您可以通过多种方式实现这一点。通常，您需要使用一个传感器，这是类似于此的合适气流对象。首先，请查看并四处走动。在气流中，传感器的使用方式与操作员的使用方式相同。因此，您可以创建如下作业：

http_post_task -> http_sensor_task -> success_task

from airflow.operators.sensors import BaseSensorOperator
from airflow.utils.decorators import apply_defaults
from time import sleep
import requests

class HTTPSensorOperator(BaseSensorOperator): 
    """
    Pokes a URL until it returns 200
    """
    ui_color = '#000000'
    @apply_defaults
    def __init__( self, url, *args, **kwargs):
        super(HTTPSensorOperator, self).__init__(*args, **kwargs)
        self.url = url


    def poke(self, context):
        """
        GET request url and return True if response is 200, False otherwise
        """
        r = requests.post(self.url)
        if r.status_code == 200:
            return True
        else:
            return False

    def execute(self, context):
        """
        Check the url and wait for it to return 200.
        """
        started_at = datetime.utcnow()
        while not self.poke(context):
            if (datetime.utcnow() - started_at).total_seconds() > self.timeout:
                if self.soft_fail:
                    raise AirflowSkipException("Exporting {0}/{1} took to long.".format(self.project, self.instance))
                else:
                    raise AirflowSkipException("Exporting {0}/{1} took to long.".format(self.project, self.instance))
            sleep(self.poke_interval)
        self.log.info("Success criteria met. Exiting.")

在http_post_任务将触发作业的情况下，http_sensor_任务将定期检查作业是否完成，例如获取微服务请求并检查200，可能是？，并且成功_任务将在http_sensor_任务成功后执行

您的http_sensor_任务需要是您自己的自定义传感器。下面是一些sudo代码，可以帮助您创建此传感器。记住，传感器就像操作员一样使用。考虑到向微服务发出请求，然后发出另一个请求来检查任务获取请求的状态并检查200，您将扩展BaseSeServer操作符类：

http_post_task -> http_sensor_task -> success_task

from airflow.operators.sensors import BaseSensorOperator
from airflow.utils.decorators import apply_defaults
from time import sleep
import requests

class HTTPSensorOperator(BaseSensorOperator): 
    """
    Pokes a URL until it returns 200
    """
    ui_color = '#000000'
    @apply_defaults
    def __init__( self, url, *args, **kwargs):
        super(HTTPSensorOperator, self).__init__(*args, **kwargs)
        self.url = url


    def poke(self, context):
        """
        GET request url and return True if response is 200, False otherwise
        """
        r = requests.post(self.url)
        if r.status_code == 200:
            return True
        else:
            return False

    def execute(self, context):
        """
        Check the url and wait for it to return 200.
        """
        started_at = datetime.utcnow()
        while not self.poke(context):
            if (datetime.utcnow() - started_at).total_seconds() > self.timeout:
                if self.soft_fail:
                    raise AirflowSkipException("Exporting {0}/{1} took to long.".format(self.project, self.instance))
                else:
                    raise AirflowSkipException("Exporting {0}/{1} took to long.".format(self.project, self.instance))
            sleep(self.poke_interval)
        self.log.info("Success criteria met. Exiting.")

然后使用如下运算符：

http_sensor_task = HTTPSensorOperator(
      task_id="http_sensor_task",
      url="http://localhost/check_job?job_id=1",
      timeout=3600, # 1 hour
      dag=dag
   )

---

因此，您必须决定微服务将如何与气流进行通信。就在我的脑海里，我想你会提出一个请求来触发一项工作，然后再提出一个请求，也许是10秒钟来检查一项工作。祝你好运

有很多方法可以通过气流和微服务实现这一点。通常，您需要使用一个传感器，这是类似于此的合适气流对象。首先，请查看并四处走动。在气流中，传感器的使用方式与操作员的使用方式相同。因此，您可以创建如下作业：

http_post_task -> http_sensor_task -> success_task

from airflow.operators.sensors import BaseSensorOperator
from airflow.utils.decorators import apply_defaults
from time import sleep
import requests

class HTTPSensorOperator(BaseSensorOperator): 
    """
    Pokes a URL until it returns 200
    """
    ui_color = '#000000'
    @apply_defaults
    def __init__( self, url, *args, **kwargs):
        super(HTTPSensorOperator, self).__init__(*args, **kwargs)
        self.url = url


    def poke(self, context):
        """
        GET request url and return True if response is 200, False otherwise
        """
        r = requests.post(self.url)
        if r.status_code == 200:
            return True
        else:
            return False

    def execute(self, context):
        """
        Check the url and wait for it to return 200.
        """
        started_at = datetime.utcnow()
        while not self.poke(context):
            if (datetime.utcnow() - started_at).total_seconds() > self.timeout:
                if self.soft_fail:
                    raise AirflowSkipException("Exporting {0}/{1} took to long.".format(self.project, self.instance))
                else:
                    raise AirflowSkipException("Exporting {0}/{1} took to long.".format(self.project, self.instance))
            sleep(self.poke_interval)
        self.log.info("Success criteria met. Exiting.")

在http_post_任务将触发作业的情况下，http_sensor_任务将定期检查作业是否完成，例如获取微服务请求并检查200，可能是？，并且成功_任务将在http_sensor_任务成功后执行

您的http_sensor_任务需要是您自己的自定义传感器。下面是一些sudo代码，可以帮助您创建此传感器。记住，传感器就像操作员一样使用。考虑到向微服务发出请求，然后发出另一个请求来检查任务获取请求的状态并检查200，您将扩展BaseSeServer操作符类：

http_post_task -> http_sensor_task -> success_task

from airflow.operators.sensors import BaseSensorOperator
from airflow.utils.decorators import apply_defaults
from time import sleep
import requests

class HTTPSensorOperator(BaseSensorOperator): 
    """
    Pokes a URL until it returns 200
    """
    ui_color = '#000000'
    @apply_defaults
    def __init__( self, url, *args, **kwargs):
        super(HTTPSensorOperator, self).__init__(*args, **kwargs)
        self.url = url


    def poke(self, context):
        """
        GET request url and return True if response is 200, False otherwise
        """
        r = requests.post(self.url)
        if r.status_code == 200:
            return True
        else:
            return False

    def execute(self, context):
        """
        Check the url and wait for it to return 200.
        """
        started_at = datetime.utcnow()
        while not self.poke(context):
            if (datetime.utcnow() - started_at).total_seconds() > self.timeout:
                if self.soft_fail:
                    raise AirflowSkipException("Exporting {0}/{1} took to long.".format(self.project, self.instance))
                else:
                    raise AirflowSkipException("Exporting {0}/{1} took to long.".format(self.project, self.instance))
            sleep(self.poke_interval)
        self.log.info("Success criteria met. Exiting.")

然后使用如下运算符：

http_sensor_task = HTTPSensorOperator(
      task_id="http_sensor_task",
      url="http://localhost/check_job?job_id=1",
      timeout=3600, # 1 hour
      dag=dag
   )

因此，您必须决定微服务将如何与气流进行通信。就在我的脑海里，我想你会提出一个请求来触发一项工作，然后再提出一个请求，也许是10秒钟来检查一项工作。祝你好运