ApacheBeam-将BigQuery TableRow写入Cassandra
我正在尝试使用TableRow从BigQuery读取数据,并将输出写入Cassandra。怎么做 这是我试过的。这项工作:ApacheBeam-将BigQuery TableRow写入Cassandra,cassandra,google-bigquery,google-cloud-dataflow,apache-beam,dataflow,Cassandra,Google Bigquery,Google Cloud Dataflow,Apache Beam,Dataflow,我正在尝试使用TableRow从BigQuery读取数据,并将输出写入Cassandra。怎么做 这是我试过的。这项工作: /* Read BQ */ PCollection<CxCpmMapProfile> data = p.apply(BigQueryIO.read(new SerializableFunction<SchemaAndRecord, CxCpmMapProfile>() { public CxCpmMapProfile apply(Schem
/* Read BQ */
PCollection<CxCpmMapProfile> data = p.apply(BigQueryIO.read(new SerializableFunction<SchemaAndRecord, CxCpmMapProfile>() {
public CxCpmMapProfile apply(SchemaAndRecord record) {
GenericRecord r = record.getRecord();
return new CxCpmMapProfile((String) r.get("channel_no").toString(), (String) r.get("channel_name").toString());
}
}).fromQuery("SELECT channel_no, channel_name FROM `dataset_name.table_name`").usingStandardSql().withoutValidation());
/* Write to Cassandra */
data.apply(CassandraIO.<CxCpmMapProfile>write()
.withHosts(Arrays.asList("<IP addr1>", "<IP addr2>"))
.withPort(9042)
.withUsername("cassandra_user").withPassword("cassandra_password").withKeyspace("cassandra_keyspace")
.withEntity(CxCpmMapProfile.class));
但是,当我使用TableRow这样更改Read BQ part时:
/* Read from BQ using readTableRow */
PCollection<TableRow> data = p.apply(BigQueryIO.readTableRows()
.fromQuery("SELECT channel_no, channel_name FROM `dataset_name.table_name`")
.usingStandardSql().withoutValidation());
在写给卡桑德拉的信中,我发现了以下错误
方法ApplyTransform错误是由包含TableRow元素的输入PCollection引起的,而CassandraIO read需要CxCpmMacProfile元素。您需要将BigQuery中的元素作为CxCpmMacProfile元素读取。有一个从表中读取行并通过该方法将其解析为自定义类型的示例。谢谢,因此在这种情况下,似乎唯一的选项是自定义类型,而不是TableRow