Airflow 将xcom从Python虚拟操作符推送到Python虚拟操作符

Airflow 将xcom从Python虚拟操作符推送到Python虚拟操作符,airflow,apache-airflow-xcom,Airflow,Apache Airflow Xcom,我有以下DAG: def load_data(): 数据=加载_数据() n_rows=data.shape[0] 返回n_行 def get_n_行(**kwargs): ti=kwargs[“ti”] n\u rows=ti.xcom\u pull(任务id=“加载数据”) 如果n_行>10: 返回“多行任务” 返回“不是很多行任务” 与达格( "测试",, 默认参数={“提供上下文”:True}, description='A test', 计划间隔=时间增量(天数=1), 开始日期=日

我有以下DAG:


def load_data():
数据=加载_数据()
n_rows=data.shape[0]
返回n_行
def get_n_行(**kwargs):
ti=kwargs[“ti”]
n\u rows=ti.xcom\u pull(任务id=“加载数据”)
如果n_行>10:
返回“多行任务”
返回“不是很多行任务”
与达格(
"测试",,
默认参数={“提供上下文”:True},
description='A test',
计划间隔=时间增量(天数=1),
开始日期=日前(1),
标记=['test']
)作为dag:
t1=PythonVirtualenvOperator(
task_id=“加载数据”,
要求=[],
python\u callable=加载\u数据
)
t2=分支操作器(
task_id=“branch”,
python\u callable=获取\u n\u行
问题是
get\u n\u行中的
ti.xcom\u pull(“加载数据”)
没有返回任何值,即
load\u data
的返回值似乎没有被推送到xcom

如果我随后尝试使用
xcom\u push
而不是只返回
load\u data
中的值,那么当我添加
**kwargs
作为
load\u data
的输入时,它将失败

def load_data(**kwargs):
    ti = kwargs["ti"]
    ti.xcom_push(value=5)
(即使我只有一个空的身体,但输入中有
**kwargs

投掷

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1138, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/usr/local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1311, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/usr/local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1341, in _execute_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python3.6/site-packages/airflow/operators/python.py", line 493, in execute
    super().execute(context=serializable_context)
  File "/usr/local/lib/python3.6/site-packages/airflow/operators/python.py", line 117, in execute
    return_value = self.execute_callable()
  File "/usr/local/lib/python3.6/site-packages/airflow/operators/python.py", line 512, in execute_callable
    self._write_args(input_filename)
  File "/usr/local/lib/python3.6/site-packages/airflow/operators/python.py", line 540, in _write_args
    self.pickling_library.dump({'args': self.op_args, 'kwargs': self.op_kwargs}, file)
TypeError: can't pickle module objects
在使用
PythonVirtualenvOperator
时,是否有特殊的方法来处理XCom