Warning: file_get_contents(/data/phpspider/zhask/data//catemap/2/python/323.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
PythonSDK流式数据流运行程序_Python_Google Bigquery_Google Cloud Dataflow_Apache Beam - Fatal编程技术网

PythonSDK流式数据流运行程序

PythonSDK流式数据流运行程序,python,google-bigquery,google-cloud-dataflow,apache-beam,Python,Google Bigquery,Google Cloud Dataflow,Apache Beam,我有一个用Python编写的数据流,我正试图在GCP上运行它。数据流继续以以下方式终止: 工作流失败。原因:未知消息代码 我的数据流管道中的主要代码是: schema='Member\u ID:INTEGER,First\u Name:STRING,Last\u Name:STRING,Gender:STRING,Age:INTEGER,Height:STRING,weight:INTEGER,Hours\u Sleep:INTEGER,carries\u consumered:INTEGER,

我有一个用Python编写的数据流,我正试图在GCP上运行它。数据流继续以以下方式终止:

工作流失败。原因:未知消息代码

我的数据流管道中的主要代码是:

schema='Member\u ID:INTEGER,First\u Name:STRING,Last\u Name:STRING,Gender:STRING,Age:INTEGER,Height:STRING,weight:INTEGER,Hours\u Sleep:INTEGER,carries\u consumered:INTEGER,carries\u Burned:INTEGER,Evt\u Date:Date,Height\u incheight:INTEGER,Min\u Sleep\u Hours:INTEGER,Max\u Sleep\u Hours:INTEGER,ough\u Sleep:BOOL'
#读取、转换和本地源数据
p=梁管道(选项=选项)
#从PubSub读入PCollection。
事件=(p |'Read PubSub'>>beam.io.ReadFromPubSub(topic='projects/prefab-embasic-220213/topics/health_event'))
|'Parse CSV'>>beam.ParDo(getCSVFields())
|“转换类型”>>beam.ParDo(ConvDataTypes())
|“转换高度”>>beam.ParDo(ConvHeight())
|'Join CDC Sleep'>>beam.ParDo(cdcseleepjoin(),cdcsleep)
|“创建行”>>beam.ParDo(CreateRow())
|'Write to BQ'>>beam.io.Write(beam.io.bigQueryLink(
'prefab-embasic-220213:nhcdata.nhcevents',schema=schema,
write_disposition=beam.io.BigQueryDisposition.write_TRUNCATE,
create_disposition=beam.io.BigQueryDisposition.create_(如果需要))
)
结果=p.运行()
结果。等待直到完成()
如果我删除

|“写入BQ”>>beam.io.Write(beam.io.bigQueryLink(
'prefab-embasic-220213:nhcdata.nhcevents',schema=schema,
write_disposition=beam.io.BigQueryDisposition.write_TRUNCATE,
create_disposition=beam.io.BigQueryDisposition.create_(如果需要))
然后数据流正常启动。

beam.io.Write(beam.io.BigQuerySink())是本机数据流接收器,仅适用于批处理作业。对于流式管道,应改为使用。用法:

beam.io.WriteToBigQuery(table='table\u name',dataset='dataset',project='project\u id')
你可以在这方面找到一个很好的例子