Google cloud platform 数据流侧输入刷新

Google cloud platform 数据流侧输入刷新,google-cloud-platform,google-cloud-dataflow,dataflow,Google Cloud Platform,Google Cloud Dataflow,Dataflow,我们试图在流数据流作业中以一定的间隔刷新sideinput。 跟踪此链接 但不能在语法上实现它。有人能为下面的代码片段提供实现sideinput刷新的正确方法吗 PCollection<KV<String, String>> updateVariable = pipeline.apply(JdbcIO.<KV<String, String>>read() .withDataSourceConfiguration(JdbcIO

我们试图在流数据流作业中以一定的间隔刷新sideinput。 跟踪此链接 但不能在语法上实现它。有人能为下面的代码片段提供实现sideinput刷新的正确方法吗

PCollection<KV<String, String>> updateVariable = pipeline.apply(JdbcIO.<KV<String, String>>read()
            .withDataSourceConfiguration(JdbcIO.DataSourceConfiguration
                    .create("com.mysql.jdbc.Driver", "jdbc:mysql://" + sqlIp + "/" + sqlDb).withUsername(sqlUser)
                    .withPassword(sqlPwd))
            .withQuery(
                    "select * from OBD_LOOKUP")
            .withCoder(KvCoder.of(StringUtf8Coder.of(), StringUtf8Coder.of()))
            .withRowMapper(new JdbcIO.RowMapper<KV<String, String>>() {
                @Override
                public KV<String, String> mapRow(java.sql.ResultSet resultSet) throws Exception {
                    / TODO Auto-generated method stub
                    return KV.of(resultSet.getString(1), resultSet.getString(2));
                }
            }));

   final PCollectionView<Map<String, String>> lookupCollection = updateVariable             
            .apply("Assign into Global Window",
            Window.<KV<String, String>>into(new GlobalWindows())
                    .triggering(Repeatedly.forever(AfterProcessingTime.pastFirstElementInPane()))
                    .accumulatingFiredPanes())
            .apply("SideInputViewFormed", View.<String, String>asMap());

    PCollection<String> resultnew = result
            .apply(ParDo.of(new ObdLookUpSideInput(lookupCollection)).withSideInputs(lookupCollection));
PCollection updateVariable=pipeline.apply(JdbcIO.read())
.withDataSourceConfiguration(JdbcIO.DataSourceConfiguration
.create(“com.mysql.jdbc.Driver”,“jdbc:mysql://”+sqlIp+“/”+sqlDb)。withUsername(sqlUser)
.withPassword(sqlPwd))
.withQuery(
“从OBD_查找中选择*)
.withCoder(KvCoder.of(StringUtf8Coder.of(),StringUtf8Coder.of()))
.withRowMapper(新的JdbcIO.RowMapper(){
@凌驾
公共映射行(java.sql.ResultSet-ResultSet)引发异常{
/TODO自动生成的方法存根
返回KV.of(resultSet.getString(1),resultSet.getString(2));
}
}));
final PCollectionView lookupCollection=updateVariable
.apply(“分配到全局窗口”,
Window.into(新的GlobalWindows())
.触发(重复.forever(在ProcessingTime.pastFirstElementInPane()之后)
.累积FiredPanes())
.apply(“SideInputViewFormed”,View.asMap());
PCollection resultnew=结果
.应用(新对象的第4部分(lookupCollection))。使用SideInputs(lookupCollection));

该模式利用

View.asSinglton()

View.asMap
本身就是一个聚合,它将继承触发并累积其输入。在此特定模式中,在此转换之前将管道设置为累积窗格模式将导致重复的键错误

有关此问题的更多详细信息,请参见此问题的答案。

您在指南中尝试了哪一项?生成序列还是周期性脉冲?