Google bigquery 插入到BigQuery时发生Apache Beam错误

使用谷歌云数据流的ApacheBeam2.9.0

我在尝试插入BigQuery时遇到一个错误，该错误似乎与架构有关：

RuntimeError: Could not successfully insert rows to BigQuery table [project:mlpipeline.twitter_posts]. Errors: [<InsertErrorsValueListEntry errors: [<ErrorProto debugInfo: u'' location: u'text' message: u'Invalid NUMERIC value: RT @Deep_In_Depth: 3 Advanced Python Functions for Data Scientists #DeepLearning #MachineLearning #ArtificialIntell\u2026' reason: u'invalid'>] index: 0>, <InsertErrorsValueListEntry errors: [<ErrorProto debugInfo: u'' location: u'text' message: u'Invalid NUMERIC value: What Is Deep Learning? #DeepLearning #MachineLearning #ArtificialIntelligence #DataScience #DL #ML #DS #AI #DNN #NeuralNetworks #NLP #GPU #TensorFlow #Keras #Pytorch #Python #HPC #Automation #AutonomousCar #Quant' reason: u'invalid'>] index: 1>] [while running 'generatedPtransform-20667']
at _flush_batch (/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/bigquery.py:1380)
at finish_bundle (/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/bigquery.py:1368)
at apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle (common.py:365)
at apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle (common.py:361)
at apache_beam.runners.common.DoFnRunner._invoke_bundle_method (common.py:697)
at apache_beam.runners.common.DoFnRunner._reraise_augmented (common.py:724)
at apache_beam.runners.common.DoFnRunner._invoke_bundle_method (common.py:699)
at apache_beam.runners.common.DoFnRunner.finish (common.py:705)
at apache_beam.runners.worker.operations.DoOperation.finish (operations.py:508)
at apache_beam.runners.worker.operations.DoOperation.finish (operations.py:507)
at apache_beam.runners.worker.operations.DoOperation.finish (operations.py:506)
at process_bundle (/usr/local/lib/python2.7/dist-packages/apache_beam/runners/worker/bundle_processor.py:441)
at process_bundle (/usr/local/lib/python2.7/dist-packages/apache_beam/runners/worker/sdk_worker.py:237)
at do_instruction (/usr/local/lib/python2.7/dist-packages/apache_beam/runners/worker/sdk_worker.py:221)
at <lambda> (/usr/local/lib/python2.7/dist-packages/apache_beam/runners/worker/sdk_worker.py:170)
at _execute (/usr/local/lib/python2.7/dist-packages/apache_beam/runners/worker/sdk_worker.py:135)

我已将

bigqueryschema_json

中的文本字段更改为

STRING

和

NUMERIC

，但问题相同

完整代码：

---

任何有关如何调试此错误的详细信息？

文本

字段不应是错误状态下的数值（

位置：u'text'消息：u'Invalid NUMERIC value…

），而应是字符串。您能提供目标表的模式吗？它可能是用

文本

作为

数字

创建的。如果是这种情况，您必须使用正确的类型重新创建表。也就是说，表最初是用错误的数字字段创建的，但由于某些原因，我的控制台没有显示新创建的表。删除表后，使用字符串字段all是好的。谢谢

# Make explicit BQ schema for output tables
    bigqueryschema_json = '{"fields": [' \
                          '{"name":"id","type":"STRING"},' \
                          '{"name":"text","type":"NUMERIC"},' \
                          '{"name":"user_id","type":"STRING"},' \
                          '{"name":"sentiment","type":"FLOAT"},' \
                          '{"name":"posted_at","type":"TIMESTAMP"}' \
                          ']}'
    bigqueryschema = parse_table_schema_from_json(bigqueryschema_json)