Google cloud platform 数据流侧输入刷新
我们试图在流数据流作业中以一定的间隔刷新sideinput。 跟踪此链接 但不能在语法上实现它。有人能为下面的代码片段提供实现sideinput刷新的正确方法吗Google cloud platform 数据流侧输入刷新,google-cloud-platform,google-cloud-dataflow,dataflow,Google Cloud Platform,Google Cloud Dataflow,Dataflow,我们试图在流数据流作业中以一定的间隔刷新sideinput。 跟踪此链接 但不能在语法上实现它。有人能为下面的代码片段提供实现sideinput刷新的正确方法吗 PCollection<KV<String, String>> updateVariable = pipeline.apply(JdbcIO.<KV<String, String>>read() .withDataSourceConfiguration(JdbcIO
PCollection<KV<String, String>> updateVariable = pipeline.apply(JdbcIO.<KV<String, String>>read()
.withDataSourceConfiguration(JdbcIO.DataSourceConfiguration
.create("com.mysql.jdbc.Driver", "jdbc:mysql://" + sqlIp + "/" + sqlDb).withUsername(sqlUser)
.withPassword(sqlPwd))
.withQuery(
"select * from OBD_LOOKUP")
.withCoder(KvCoder.of(StringUtf8Coder.of(), StringUtf8Coder.of()))
.withRowMapper(new JdbcIO.RowMapper<KV<String, String>>() {
@Override
public KV<String, String> mapRow(java.sql.ResultSet resultSet) throws Exception {
/ TODO Auto-generated method stub
return KV.of(resultSet.getString(1), resultSet.getString(2));
}
}));
final PCollectionView<Map<String, String>> lookupCollection = updateVariable
.apply("Assign into Global Window",
Window.<KV<String, String>>into(new GlobalWindows())
.triggering(Repeatedly.forever(AfterProcessingTime.pastFirstElementInPane()))
.accumulatingFiredPanes())
.apply("SideInputViewFormed", View.<String, String>asMap());
PCollection<String> resultnew = result
.apply(ParDo.of(new ObdLookUpSideInput(lookupCollection)).withSideInputs(lookupCollection));
PCollection updateVariable=pipeline.apply(JdbcIO.read())
.withDataSourceConfiguration(JdbcIO.DataSourceConfiguration
.create(“com.mysql.jdbc.Driver”,“jdbc:mysql://”+sqlIp+“/”+sqlDb)。withUsername(sqlUser)
.withPassword(sqlPwd))
.withQuery(
“从OBD_查找中选择*)
.withCoder(KvCoder.of(StringUtf8Coder.of(),StringUtf8Coder.of()))
.withRowMapper(新的JdbcIO.RowMapper(){
@凌驾
公共映射行(java.sql.ResultSet-ResultSet)引发异常{
/TODO自动生成的方法存根
返回KV.of(resultSet.getString(1),resultSet.getString(2));
}
}));
final PCollectionView lookupCollection=updateVariable
.apply(“分配到全局窗口”,
Window.into(新的GlobalWindows())
.触发(重复.forever(在ProcessingTime.pastFirstElementInPane()之后)
.累积FiredPanes())
.apply(“SideInputViewFormed”,View.asMap());
PCollection resultnew=结果
.应用(新对象的第4部分(lookupCollection))。使用SideInputs(lookupCollection));
该模式利用
View.asSinglton()
View.asMap
本身就是一个聚合,它将继承触发并累积其输入。在此特定模式中,在此转换之前将管道设置为累积窗格模式将导致重复的键错误
有关此问题的更多详细信息,请参见此问题的答案。您在指南中尝试了哪一项?生成序列还是周期性脉冲?