Google cloud dataflow 使用Beam和DataFlow将数据从云SQL移动到弹性搜索

Google cloud dataflow 使用Beam和DataFlow将数据从云SQL移动到弹性搜索,google-cloud-dataflow,google-cloud-sql,apache-beam,Google Cloud Dataflow,Google Cloud Sql,Apache Beam,我是beam和Google dataflow的新手,我创建了一个简单的类,使用批处理将数据从云sql迁移到弹性搜索,方法如下: package com.abc; class DataFlowTest{ public static void main(String[] args) { DataflowPipelineOptions options = PipelineOptionsFactory.as(DataflowPipelineOptions.class);

我是beam和Google dataflow的新手,我创建了一个简单的类,使用批处理将数据从云sql迁移到弹性搜索,方法如下:

package com.abc;

class DataFlowTest{

    public static void main(String[] args) {

    DataflowPipelineOptions options = PipelineOptionsFactory.as(DataflowPipelineOptions.class);
        options.setProject("staging");  options.setTempLocation("gs://csv_to_sql_staging/temp");    options.setStagingLocation("gs://csv_to_sql_staging/staging");  options.setRunner(DataflowRunner.class);    options.setGcpTempLocation("gs://csv_to_sql_staging/temp");
            Pipeline p = Pipeline.create(options);


      p.begin();

      p.apply(JdbcIO.read().withQuery("select * from user_table").withDataSourceConfiguration(JdbcIO.DataSourceConfiguration.create("com.mysql.jdbc.Driver", "jdbc:mysql://"+EnvironmentVariable.getDatabaseIp()+"/" + EnvironmentVariable.getDatabaseName()+ "&socketFactory=com.google.cloud.sql.mysql.SocketFactory&user="+Credentials.getDatabaseUsername()+"&password="+Credentials.getDatabasePassword()+"&useSSL=false")));


  Write w = ElasticsearchIO.write().withConnectionConfiguration(
            ElasticsearchIO.ConnectionConfiguration.create(new String [] {"host"}, "user-temp", "String").withUsername("elastic").withPassword("password")
            );
   p.apply(w);
p、 run().waitUntilFinish();}}

并在pom.xml中找到我的依赖项
org.apache.beam
beam SDK java核心
2.19.0
org.apache.beam
谷歌云数据流java
2.19.0
com.google.api-client
谷歌api客户端
com.google.http-client
谷歌http客户端
com.google.http-client
google-http-client-jackson2
org.apache.beam
beam SDK java io elasticsearch
2.19.0
org.apache.beam
beam SDK java io jdbc
2.19.0
org.apache.beam
beam SDK java io谷歌云平台
2.19.0
io.grpc
grpc核心
现在的问题是一个编译错误,它说:

类型管道中的方法apply(ptTransform)不适用于参数(ElasticsearchIO.Write) 在该行:p.apply(w)

有人能帮上忙吗?
我在pom文件中做了一些排除,以修复一些依赖项冲突

无法直接将ElasticSearchIO.write应用于管道对象。首先创建一个PCollection,然后将ElasticsearchIO应用于PCollection。请参考下面的代码

PCollection<String> sqlResult1 = p.apply(
                JdbcIO.<String>read().withDataSourceConfiguration(config).withQuery("select * from test_table")
                        .withCoder(StringUtf8Coder.of()).withRowMapper(new JdbcIO.RowMapper<String>() {

                            private static final long serialVersionUID = 1L;

                            public String mapRow(ResultSet resultSet) throws Exception {
                                StringBuilder val = new StringBuilder();
                                return val.append(resultSet.getString(0)).append(resultSet.getString(1)).toString();
                                // return KV.of(resultSet.getString(1), resultSet.getString(2));
                            }
                        }));

        sqlResult1.apply(ElasticsearchIO.write().withConnectionConfiguration(ElasticsearchIO.ConnectionConfiguration
                .create(new String[] { "https://host:9243" }, "user-temp", "String").withUsername("").withPassword("")));
PCollection sqlResult1=p.apply(
JdbcIO.read()
.withCoder(StringUtf8Coder.of()).withRowMapper(新的JdbcIO.RowMapper()){
私有静态最终长serialVersionUID=1L;
公共字符串mapRow(ResultSet ResultSet)引发异常{
StringBuilder val=新的StringBuilder();
返回val.append(resultSet.getString(0)).append(resultSet.getString(1)).toString();
//返回KV.of(resultSet.getString(1),resultSet.getString(2));
}
}));
sqlResult1.apply(ElasticsearchIO.write())withConnectionConfiguration(ElasticsearchIO.ConnectionConfiguration
.create(新字符串[]{)https://host:9243},“用户临时”、“字符串”)。使用用户名(“”)。使用密码(“”);

我认为上面的代码应该适用于您的用例

您是否尝试过调用
p转换
Write w=ElasticsearchIO.Write()…p.apply(w)
替换为
p.apply(ElasticsearchIO.Write())
?是的,同样的结果仍然是我正在尝试理解您的用例,如果我的理解有误,请告诉我。所以您正在尝试从cloudSql读取数据,并将结果数据写入弹性搜索。正确的?如果是这样的话,我可以帮你。是的,请允许我正是这么做的,我已经通过了这个错误,我现在面临另一个问题;'我在这里发布了@Tamer Saleh,如果你已经解决了当前的问题,那么写下答案是一个好习惯,以帮助下一个贡献者研究类似问题的解决方案。
PCollection<String> sqlResult1 = p.apply(
                JdbcIO.<String>read().withDataSourceConfiguration(config).withQuery("select * from test_table")
                        .withCoder(StringUtf8Coder.of()).withRowMapper(new JdbcIO.RowMapper<String>() {

                            private static final long serialVersionUID = 1L;

                            public String mapRow(ResultSet resultSet) throws Exception {
                                StringBuilder val = new StringBuilder();
                                return val.append(resultSet.getString(0)).append(resultSet.getString(1)).toString();
                                // return KV.of(resultSet.getString(1), resultSet.getString(2));
                            }
                        }));

        sqlResult1.apply(ElasticsearchIO.write().withConnectionConfiguration(ElasticsearchIO.ConnectionConfiguration
                .create(new String[] { "https://host:9243" }, "user-temp", "String").withUsername("").withPassword("")));