Google bigquery 将数据流式传输到Google BigQuery表:使用InsertId消除重复记录的问题
我们正在使用Camel BigQueryAPI(版本2.20)将记录从ActiveMQ服务器(版本5.14.3)上的消息队列流式传输到Google BigQuery表中 我们已经在主站点上运行的Spring框架中实现并部署了作为XML路由定义的流机制,它似乎工作得很好Google bigquery 将数据流式传输到Google BigQuery表:使用InsertId消除重复记录的问题,google-bigquery,apache-camel,spring-jms,spring-camel,Google Bigquery,Apache Camel,Spring Jms,Spring Camel,我们正在使用Camel BigQueryAPI(版本2.20)将记录从ActiveMQ服务器(版本5.14.3)上的消息队列流式传输到Google BigQuery表中 我们已经在主站点上运行的Spring框架中实现并部署了作为XML路由定义的流机制,它似乎工作得很好 <?xml version="1.0" encoding="UTF-8"?> <beans xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
<?xml version="1.0" encoding="UTF-8"?>
<beans
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://www.springframework.org/schema/beans"
xmlns:beans="http://www.springframework.org/schema/beans"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
./spring-beans.xsd
http://camel.apache.org/schema/spring
./camel-spring.xsd">
<!--
# ==========================================================================
# ActiveMQ JMS Bean Definition
# ==========================================================================
-->
<bean id="jms" class="org.apache.camel.component.jms.JmsComponent">
<property name="connectionFactory">
<bean class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="nio://192.168.10.10:61616?jms.useAsyncSend=true" />
<property name="userName" value="MyAmqUserName" />
<property name="password" value="MyAmqPassword" />
</bean>
</property>
</bean>
<!--
# ==========================================================================
# GoogleBigQueryComponent
# https://github.com/apache/camel/tree/master/components/camel-google-bigquery
# ==========================================================================
-->
<bean id="gcp" class="org.apache.camel.component.google.bigquery.GoogleBigQueryComponent">
<property name="connectionFactory">
<bean class="org.apache.camel.component.google.bigquery.GoogleBigQueryConnectionFactory">
<property name="credentialsFileLocation" value="MyDir/MyGcpKeyFile.json" />
</bean>
</property>
</bean>
<!--
# ==========================================================================
# Main Context Bean Definition
# ==========================================================================
-->
<camelContext id="camelContext" xmlns="http://camel.apache.org/schema/spring" >
<!--
# ==================================================================
# Message Route :
# 1. consume messages from my AMQ queue
# 2. set the InsertId / INSERT_ID (it is not clear which is the correct one)
# 3. write message to Google BigQuery table
# see https://github.com/apache/camel/blob/master/components/camel-google-bigquery/src/main/docs/google-bigquery-component.adoc
# ==================================================================
<log message="${headers} | ${body}" />
-->
<route>
<from uri="jms:my.amq.queue.of.output.data.for.gcp?acknowledgementModeName=DUPS_OK_ACKNOWLEDGE&concurrentConsumers=20" />
<setHeader headerName="CamelGoogleBigQuery.InsertId">
<simple>${header.KeyValuePreviouslyGenerated}</simple>
</setHeader>
<setHeader headerName="GoogleBigQueryConstants.INSERT_ID">
<simple>${header.KeyValuePreviouslyGenerated}</simple>
</setHeader>
<to uri="gcp:my_gcp_project:my_bq_data_set:my_bq_table" />
</route>
</camelContext>
</beans>
${header.KeyValuePreviouslyGenerated}
${header.KeyValuePreviouslyGenerated}
为了实现高(er)可用性,我们现在已将相同的实现部署到备份站点,流式传输到相同的目标BigQuery表。正如预期的那样,当相同的记录从两个站点流到同一个表中时,会出现重复的记录。为了消除记录重复,我们尝试遵循以下指南:
消息头部分建议使用合适的运行时键值设置名为camelgebigquery.InsertId的消息头
但是,在同一页下方的“确保数据一致性”部分建议设置GoogleBigQueryConstants.INSERT_ID
我们已经检查了主服务器和备份服务器是否在同一时区(UTC)内运行,并且正在生成我们认为合适的运行时唯一键:一个包含UNIX时间的字符串,精确到秒
上面的代码示例显示,我们已经尝试了这两种方法,但是对目标BigQuery表中的数据的检查表明这两种方法似乎都不起作用,即我们仍然有重复的记录
问题
GoogleBigQueryConstants.INSERT\u ID
是字符串常量,值为CamelGoogleBigQueryInsertId
像这样使用它:
<setHeader headerName="CamelGoogleBigQueryInsertId">
<simple>${header.KeyValuePreviouslyGenerated}</simple>
</setHeader>
${header.KeyValuePreviouslyGenerated}
下面是演示此行为的单元测试:
关于这些标题的文档有点过时,我已经修复了它,可以在中找到正确的版本。在网站上,它将很快发布。重新测试和审查结果的时间比预期的要长一些,但非常感谢,“CamelGoogleBigQueryInsertId”工作得很好-这说明文档中的一个小点是如何决定事情的!再次感谢。