Google cloud dataflow 无法使用Dataflow REST API传递作业参数
我试图通过调用数据流模板,但是,当在正文中指定参数时,将抛出无效的\u参数错误。删除参数字段后,它工作正常。我尝试了Javascript和Python,但得到了完全相同的错误Google cloud dataflow 无法使用Dataflow REST API传递作业参数,google-cloud-dataflow,apache-beam,dataflow,Google Cloud Dataflow,Apache Beam,Dataflow,我试图通过调用数据流模板,但是,当在正文中指定参数时,将抛出无效的\u参数错误。删除参数字段后,它工作正常。我尝试了Javascript和Python,但得到了完全相同的错误 from googleapiclient.discovery import build from oauth2client.client import GoogleCredentials def invoke_dataflow_job(): credentials = GoogleCredentials.get_
from googleapiclient.discovery import build
from oauth2client.client import GoogleCredentials
def invoke_dataflow_job():
credentials = GoogleCredentials.get_application_default()
service = build('dataflow', 'v1b3', credentials=credentials)
TEMPLATE_LOCATION = 'gs://xxx';
PROJECT_ID = 'xxx';
JOB_NAME = 'xxx';
LOCATION = 'europe-west1';
ZONE = 'europe-west1-b';
TEMP_LOCATION = 'xxx';
BODY = {
'jobName': JOB_NAME,
'gcsPath': TEMPLATE_LOCATION,
'parameters': {
'messageToPrint': 'Blue sky'
},
'environment': {
'tempLocation': TEMP_LOCATION,
'zone': ZONE,
}
}
request = service.projects().locations().templates().create(projectId=PROJECT_ID, location=LOCATION, body=BODY)
response = request.execute()
return response
RESTAPI支持传入参数吗?看看代码,我认为应该是这样的
有什么建议吗
错误消息:
The workflow could not be created. Causes: (4adc3dbfd180d8a2): Found unexpected parameters: ['messageToPrint' (perhaps you meant 'userAgent')]
更新1:
样本DoFn:
public static class BuildPathFn extends DoFn<String, Void> {
private static final long serialVersionUID = -2815123956194177539L;
private final ValueProvider<String> baseDir;
private final ValueProvider<String> year;
private final ValueProvider<String> month;
private final ValueProvider<String> day;
private final ValueProvider<String> hour;
private final ValueProvider<String> filePattern;
public BuildPathFn (ValueProvider<String> baseDir, ValueProvider<String> year,
ValueProvider<String> month, ValueProvider<String> day,
ValueProvider<String> hour, ValueProvider<String> filePattern) {
this.baseDir = baseDir;
this.year = year;
this.month = month;
this.day = day;
this.hour = hour;
this.filePattern = filePattern;
}
@ProcessElement
public void processElement(@Element String element, OutputReceiver<Void> receiver) {
try {
String path = baseDir.get() + "/" + year.get() + "/" + month.get() + "/" + day.get() + "/" + hour.get() + "/" + filePattern.get();
LOG.info("BuildPathFn >> Path: {}", path);
} catch (Exception e) {
LOG.error("BuildPathFn >> Exception: {}", e.getMessage());
}
}
}
用于构建模板的mvn命令的一部分:
".... --baseDir --year --month --day --hour --filePattern"
代码很好,这是传递参数的正确方法。请记住,该选项需要在模板中定义为一个选项。当模板未传递任何要在运行时解析的参数时。然后,当模板的描述中已经有代码时(即,在请求正文中使用
参数),应评估选项值。刚刚测试了这一点,只要在模板中定义了参数,它就对我有效。你能分享完整的错误信息吗?非常感谢你花时间测试它。参数肯定在模板中,但是,在编译模板时,我没有将其作为参数包含在mvn编译的命令行中。使用添加的参数重新编译后,效果非常好。非常感谢。请添加您的评论作为答案,以便我可以接受。谢谢您的确认。但是,在执行此操作之前,我想确认在调用模板时是否能够使用其他参数。因为我认为,如果您在mvn命令中指定它,它可能总是解析原始参数,因为对我来说,您的怀疑是正确的。我确实看了您提到的另一个问题,但是,我无法读取传入参数值的值。运行模板时不会引发任何错误。请参阅上面的更新1。已修复。我想我没有正确地阅读你在另一个线程中的说明。在模板生成时,需要从mvn命令中完全删除参数。您现在可以将其添加为答案。再次感谢。
".... --baseDir --year --month --day --hour --filePattern"