Airflow 如何在dag中将pythonoperator任务的参数传递给SimpleHttOperator任务?
我想触发一个simplehttpoperator,如下所示: 气流触发器测试触发器--conf'{“name”:“something”} 然后我使用pythonoperator python_callable通过使用kwargs['dag_run'].conf接受参数,我想将['dag_run'].conf传递给simplehttpoperator,我该如何做?有人能帮忙吗Airflow 如何在dag中将pythonoperator任务的参数传递给SimpleHttOperator任务?,airflow,apache-airflow,airflow-scheduler,Airflow,Apache Airflow,Airflow Scheduler,我想触发一个simplehttpoperator,如下所示: 气流触发器测试触发器--conf'{“name”:“something”} 然后我使用pythonoperator python_callable通过使用kwargs['dag_run'].conf接受参数,我想将['dag_run'].conf传递给simplehttpoperator,我该如何做?有人能帮忙吗 cc_ = {} def run_this_func(ds, **kwargs): cc_ = kwargs[
cc_ = {}
def run_this_func(ds, **kwargs):
cc_ = kwargs['dag_run'].conf
logging.info(cc_)
return cc_
run_this = PythonOperator(
task_id='run_this',
provide_context=True,
python_callable=run_this_func,
dag=dag)
http_task = SimpleHttpOperator(
task_id='http_task',
http_conn_id='test_http',
method='POST',
endpoint='/api/v1/function',
data=cc_,
headers={"Authorization": "Basic YWRtaW46MTIzNDU2", "Accept": "application/json, text/plain, */*"},
response_check=lambda response: True if "10000" in response.content else False,
dag=dag)
http_task.set_upstream(run_this)
对于任务之间的通信,您可能需要检查XCOM *****更新******
(感谢Daniel提供更多详细信息) 下面是一些代码,您可以尝试一下,在SimpleHttpOperator中,您可以通过XCOM获得返回值:
http_task = SimpleHttpOperator(
task_id='http_task',
http_conn_id='test_http',
method='POST',
endpoint='/api/v1/function',
data=json.loads("{{ task_instance.xcom_pull(task_ids='run_this', key='return_value') }}"),
headers={"Authorization": "Basic YWRtaW46MTIzNDU2", "Accept": "application/json, text/plain, */*"},
response_check=lambda response: True if "10000" in response.content else False,
dag=dag)
感谢@Chengzhi和@Daniel。 最后,我在Jinja2/filter.py中编写了一个自定义过滤器“tojson”,因为默认的Jinja2版本是2.8.1,而Jinja2直到2.9版本才包含名为“tojson”的内置过滤器
def do_tojson(value):
value = json.JSONEncoder().encode(value)
return value
在dag文件中,代码如下所示。它起作用了
def run_this_func(ds, **kwargs):
cc_ = kwargs['dag_run'].conf
return cc_
run_this = PythonOperator(
task_id='run_this',
provide_context=True,
python_callable=run_this_func,
dag=dag)
http_task = SimpleHttpOperator(
task_id='http_task',
http_conn_id='test_http',
method='POST',
endpoint='/api/v1/task',
data="{{ task_instance.xcom_pull(task_ids='run_this') |tojson}}",
headers={"Authorization": "Basic YWRtaW46MTIzNDU2", "Accept": "application/json, text/plain, */*",
"Content-Type": "application/json"},
response_check=lambda response: True if "10000" in response.content else False,
dag=dag)
http_task.set_upstream(run_this)
但是如何在simplehttpoperator中使用XCOM呢?你能给出任何案例代码吗?@pyfroggogogo,我更新了一些代码示例,如果必须将此工作模板作为字符串传递,请尝试。可以使用
data=json.loads({{…| tojson}}}')
在呈现后将其返回到dict类型。