Google cloud platform 超出Apache beam PubSubIO写入负载大小

Google cloud platform 超出Apache beam PubSubIO写入负载大小,google-cloud-platform,google-cloud-dataflow,apache-beam,Google Cloud Platform,Google Cloud Dataflow,Apache Beam,我试图使用beam进行重播进程,从BigQuery表读取数据,然后将数据写入PubSub主题,但我的数据流作业一直失败,出现了一个未捕获的异常。从日志中,我可以看到此错误: exception: "com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request { "code" : 400, "errors" : [ { "domain" : "global", "message" : "

我试图使用beam进行重播进程,从BigQuery表读取数据,然后将数据写入PubSub主题,但我的数据流作业一直失败,出现了一个未捕获的异常。从日志中,我可以看到此错误:

exception:  
"com.google.api.client.googleapis.json.GoogleJsonResponseException: 
400 Bad Request
{
"code" : 400,
"errors" : [ {
"domain" : "global",
"message" : "Request payload size exceeds the limit: 10485760 bytes.",
"reason" : "badRequest"
} ],
"message" : "Request payload size exceeds the limit: 10485760 
bytes.",
"status" : "INVALID_ARGUMENT"
}
我正在使用ApacheBeam 2.1.0的PubsubIO.writeMessages()和一个PubsubMessages的PCollection,我已经创建了一个xml负载和一个包含一个属性的属性映射。这些消息的大小远远低于10MB,并且在它们到达原始BQ表之前已经通过Pubsub发送

例如:

PCollection<TableRow> input = pipeline.apply(BigQueryIO.read().fromQuery(queryBuilder.toString()).usingStandardSql());
        PCollection<PubsubMessage> replayMessages = input.apply("Create pubsub messages" ,ParDo.of(new DoFn<TableRow, PubsubMessage>() {
            private static final long serialVersionUID = -123;
            @DoFn.ProcessElement
            public void processElement(ProcessContext c) throws Exception {
                PubsubMessage pubsubMessage = new PubsubMessage(c.element().get("rawXml").toString().getBytes(),
                                    ImmutableMap.<String, String>of("myMessageId", c.element().get("myMessageId").toString()));
                c.output(pubsubMessage);
            }
        }));
        replayMessages.apply("Write messages to topic",PubsubIO.writeMessages().to(topicName));
PCollection input=pipeline.apply(BigQueryIO.read().fromQuery(queryBuilder.toString()).usingStandardSql());
PCollection replayMessages=input.apply(“创建pubsub消息”),ParDo.of(new DoFn(){
私有静态最终长serialVersionUID=-123;
@DoFn.ProcessElement
public void processElement(ProcessContext c)引发异常{
PubsubMessage PubsubMessage=新的PubsubMessage(c.element().get(“rawXml”).toString().getBytes(),
ImmutableMap.of(“myMessageId”,c.element().get(“myMessageId”).toString());
c、 输出(pubsubMessage);
}
}));
replayMessages.apply(“将消息写入主题”,PubsubIO.writeMessages().to(topicName));
这似乎只适用于95条及以下的邮件。任何时候我试图发布超过95个,我都会得到负载大小超出限制的错误。这是横梁的极限吗