Google bigquery 气流的bigqueryoperator不使用udf
我正在尝试使用Google的Composer任务在Airflow中运行一个基本的bigquery操作符,该任务使用用户定义的函数UDF 该示例来自BigQuery,并在BigQuery中完美运行 然而,当我上传到composer时,我找不到函数:multiplyInputs。。。请参见下面的python脚本 BigQueryOperator的udf_config字段需要一个列表,因此我将我的udf定义为一个包含一个字符串的列表-不确定这是否正确,因为它显然没有注册为udf 任何帮助都将不胜感激Google bigquery 气流的bigqueryoperator不使用udf,google-bigquery,airflow,google-cloud-composer,Google Bigquery,Airflow,Google Cloud Composer,我正在尝试使用Google的Composer任务在Airflow中运行一个基本的bigquery操作符,该任务使用用户定义的函数UDF 该示例来自BigQuery,并在BigQuery中完美运行 然而,当我上传到composer时,我找不到函数:multiplyInputs。。。请参见下面的python脚本 BigQueryOperator的udf_config字段需要一个列表,因此我将我的udf定义为一个包含一个字符串的列表-不确定这是否正确,因为它显然没有注册为udf 任何帮助都将不胜感激
import datetime
from airflow import models
from airflow.contrib.operators import bigquery_operator
yesterday = datetime.datetime.combine(datetime.datetime.today() -
datetime.timedelta(1),
datetime.datetime.min.time())
default_dag_args = {
# Setting start date as yesterday starts the DAG
immediately when it is
# detected in the Cloud Storage bucket.
'start_date': yesterday,
# To email on failure or retry set 'email' arg to your
email and enable
# emailing here.
'email_on_failure': False,
'email_on_retry': False,
# If a task fails, retry it once after waiting at least
5 minutes
'retries': 1,
'retry_delay': datetime.timedelta(minutes=5),
'project_id': 'vital-platform-791'
}
with models.DAG('udf_example',
schedule_interval=datetime.timedelta(days=1),
default_args=default_dag_args) as dag:
table = 'udf_table'
# flatten fe table
task_id = table + '_fe'
udf_config = ["""CREATE TEMPORARY FUNCTION multiplyInputs(x
FLOAT64, y FLOAT64)
RETURNS FLOAT64
LANGUAGE js AS \"""
return x*y;
\""";
"""]
print udf_config
query = """WITH numbers AS
(SELECT 1 AS x, 5 as y
UNION ALL
SELECT 2 AS x, 10 as y
UNION ALL
SELECT 3 as x, 15 as y)
SELECT x, y, multiplyInputs(x, y) as product
FROM numbers"""
print query
query = query
destination_table = 'vital-platform-791.alpha_factors.
{table}_fe'.format(table=table)
t_fe = bigquery_operator.BigQueryOperator(task_id=task_id,
bql=query,
destination_dataset_table=destination_table,
use_legacy_sql=False,
write_disposition='WRITE_TRUNCATE',
udf_config=udf_config)
我对这个例子有点困惑。看起来您只需要合并udf_配置和查询:
将您的UDF函数上传到Google云存储中,并将其传递给UDF_配置参数 例如: 您的UDF函数位于gs://testbucket/testfolder/UDF.js中 然后在dag中使用: udf_gcs_path=gs://testbucket/testfolder/udf.js BigQueryOperator.BigQueryOperator任务id=任务id, bql=查询, 目的地\数据集\表格=目的地\表格, 使用\u legacy\u sql=False, write_disposition='write_TRUNCATE', udf_config=[{resourceUri:udf_gcs_path}] 参考资料:
谢谢,这个很好用。BigQueryOperator的显式udf_config字段在某种程度上误导了我,因为我认为在使用udf时必须填充它。我认为问题是针对标准SQL的,因此应该接受这个答案,而不是我的答案。干得好@ElliottbrossardThank,看看文档,这似乎是传统sql的方法,而不是标准sql?
query = ""CREATE TEMPORARY FUNCTION multiplyInputs(x
FLOAT64, y FLOAT64)
RETURNS FLOAT64
LANGUAGE js AS \"""
return x*y;
\""";
WITH numbers AS
(SELECT 1 AS x, 5 as y
UNION ALL
SELECT 2 AS x, 10 as y
UNION ALL
SELECT 3 as x, 15 as y)
SELECT x, y, multiplyInputs(x, y) as product
FROM numbers;"""