Google cloud platform 如何使用谷歌提供的模板[pubsub to Datastore]?;
我想使用谷歌提供的模板,将数据从pubsub流式传输到数据存储。 我遵循编写本文档的步骤。 我通过这一步Google cloud platform 如何使用谷歌提供的模板[pubsub to Datastore]?;,google-cloud-platform,google-cloud-datastore,google-cloud-dataflow,google-cloud-pubsub,Google Cloud Platform,Google Cloud Datastore,Google Cloud Dataflow,Google Cloud Pubsub,我想使用谷歌提供的模板,将数据从pubsub流式传输到数据存储。 我遵循编写本文档的步骤。 我通过这一步 mvn clean && mvn compile 但下一步,出现了错误 [INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ google-cloud-teleport-java --- 2018-08-17 13:36:19 INFO DataflowRunner:266 - Pipeline
mvn clean && mvn compile
但下一步,出现了错误
[INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ google-cloud-teleport-java ---
2018-08-17 13:36:19 INFO DataflowRunner:266 - PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 117 files. Enable logging at DEBUG level to see which files wi
ll be staged.
[WARNING]
java.lang.IllegalStateException: Missing required properties: errorTag
at com.google.cloud.teleport.templates.common.AutoValue_DatastoreConverters_WriteJsonEntities$Builder.build(AutoValue_DatastoreConverters_WriteJsonEntities.java:89)
at com.google.cloud.teleport.templates.PubsubToDatastore.main(PubsubToDatastore.java:65)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:282)
at java.lang.Thread.run(Thread.java:748)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 35.348 s
[INFO] Finished at: 2018-08-17T13:36:20+09:00
[INFO] Final Memory: 59M/146M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project google-cloud-teleport-java: An exception occured while executing the Java class. Missing required propert
ies: errorTag -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
然后,我尝试了datastoretosubsub模板和GSCTextToDatastore模板,这些都是成功的
所以,我不明白问题出在哪里。
哪里错了
请给我一些建议…注意。看起来您在特定的DataflowTemplate中发现了一个bug,管道没有配置错误路径,即使在编写JSON实体时需要路径。修复程序相对简单,应该很快提交给master。同时,您可以对
pubsubsubtodatastore
管道代码进行两次更改,从而使管道正常工作
首先修改代码,以便datastoreoptions
扩展ErrorWriteOptions
接口。您的新选项声明应类似于以下内容:
接口数据存储选项
延伸
管道选项,
PubsubReadOptions,
JavascriptTextTransformerOptions,
数据存储写入选项,
ErrorWriteOptions{}
然后修改main方法中的代码,以便管道配置错误TupleTag
,并将任何错误消息路由到LogErrors
转换。这将确保任何未能输出到数据存储的数据被捕获并存储在地面军事系统上。新的main方法应类似于以下内容:
TupleTag errorTag=new TupleTag(){};
Pipeline=Pipeline.create(选项);
管道
.apply(publisubio.readStrings()
.fromTopic(options.getPubsubReadTopic()))
.apply(TransformTextViaJavascript.newBuilder()
.setfilesystemspath(options.getJavascriptTextTransformGcsPath())
.setFunctionName(options.getJavascriptTextTransformFunctionName())
.build())
.apply(WriteJsonEntities.newBuilder()
.setProjectId(options.getDatastoreWriteProjectId())
.setErrorTag(errorTag)
.build())
.apply(LogErrors.newBuilder()
.setErrorWritePath(options.getErrorWritePath())
.setErrorTag(errorTag)
.build());
谢谢您的回答!!我会试试你的建议,然后再跑。非常感谢你!仍然未推送:(java.lang.IllegalStateException:在没有触发器的情况下,GroupByKey无法应用于GlobalWindow中的非绑定PCollection。请在GroupByKey之前使用Window.into或Window.triggering转换。