Java-向大查询重复插入一行数据问题

我正在创建一个将被第三方攻击的系统 我预计每分钟25k请求的点击量大约为1000万次

这是密码我的密码

public class MyCustomClass extends HttpServlet {
    private static final Logger LOGGER = LoggerFactory.getLogger(MyCustomClass.class);

    public MyCustomClass() throws IOException, InterruptedException {
    }

    @Override
    protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
        String body = req.getReader().lines().collect(Collectors.joining(System.lineSeparator()));
        ObjectMapper mapper = new ObjectMapper();
        ThirdPartySpec spec = mapper.readValue(body, ThirdPartySpec.class);
        insertDetail(spec);
        resp.setContentType("application/json");
        resp.setStatus(200);
    }
    
    private boolean insertDetail(ThirdPartySpec spec) {
        try {
            Map<String, Object> rowContent = new HashMap<>();
            rowContent.put("field1", spec.getData1());
            rowContent.put("field2", spec.getData2());
            insertToBigQuery(bigquery, "myprojectid", "mydatasetid", rowContent);
        } catch (Exception e) {
            LOGGER.error("error aa " + new Gson().toJson(spec), e);
            return false;
        }
        return true;
    }
    private void insertToBigQuery(BigQuery bigQuery, String datasetName, String tableName, Map<String, Object> rowContent) throws Exception {
        TableId tableId = TableId.of(datasetName, tableName);
        InsertAllResponse response =
                bigQuery.insertAll(
                        InsertAllRequest.newBuilder(tableId)
                                .addRow(rowContent)
                                .build());

        if (response.hasErrors()) {
            for (Map.Entry<Long, List<BigQueryError>> entry : response.getInsertErrors().entrySet()) {
                throw new Exception(entry.getValue().toString());
            }
        }
    }

公共类MyCustomClass扩展了HttpServlet{
私有静态最终记录器Logger=LoggerFactory.getLogger（MyCustomClass.class）；
public MyCustomClass（）引发IOException，InterruptedException{
}
@凌驾
受保护的void doPost（HttpServletRequest-req，HttpServletResponse-resp）引发ServletException，IOException{
String body=req.getReader（）.lines（）.collect（collector.joining（System.lineSeparator（））；
ObjectMapper mapper=新的ObjectMapper（）；
ThirdPartySpec spec=mapper.readValue（body，ThirdPartySpec.class）；
插入细节（规范）；
分别为setContentType（“应用程序/json”）；
分别为setStatus（200）；
}
私有布尔插入细节（第三方规范）{
试一试{
Map rowContent=newhashmap（）；
rowContent.put（“field1”，spec.getData1（））；
rowContent.put（“field2”，spec.getData2（））；
insertToBigQuery（bigquery，“myprojectid”，“mydatasetid”，rowContent）；
}捕获（例外e）{
LOGGER.error（“error aa”+new Gson（）.toJson（spec），e）；
返回false；
}
返回true；
}
私有void insertToBigQuery（BigQuery BigQuery、String datasetName、String tableName、Map rowContent）引发异常{
TableId TableId=TableId.of（datasetName，tableName）；
插入应答=
bigQuery.insertAll(
InsertAllRequest.newBuilder（表ID）
.addRow（行内容）
.build（））；
if（response.hasErrors（））{
for（Map.Entry:response.getInsertErrors（）.entrySet（））{
抛出新异常（entry.getValue（）.toString（））；
}
}
}

我的当前配置

  <instance-class>F4</instance-class>
  <automatic-scaling>
    <target-cpu-utilization>0.8</target-cpu-utilization>
    <max-instances>10</max-instances>
    <min-instances>1</min-instances>
    <max-concurrent-requests>100</max-concurrent-requests>
  </automatic-scaling>

F4
那是个错误。
服务器遇到临时错误，无法完成您的请求。请在30秒后重试。我们只知道这些。
原因：com.google.api.client.googleapis.json.GoogleJsonResponseException:502网关错误
邮递https://www.googleapis.com/bigquery/v2/projects/myprojectname/datasets/mydatasetname/tables/mytablename/insertAll?prettyPrint=false
在等待太长时间无法尝试服务您的请求后，请求被中止。

我已经做了一些研究，但仍然没有线索

是否有任何修复或其他方法可用于在bigquery上存储数据

供参考：

每行由第三方发送数据（这是最终的，不能更改）

---

在我看来，函数insertDetail（）所使用的数据似乎有问题，因为它每次都会触发捕获日志，这会导致大查询超时，您会收到这些错误。我的假设是，当另一端已关闭连接时，您正试图写入连接。

com.traveloka.accessor.mycustomclass insertDetail: error aa {"UserId":"123","name":"Superman"}
com.google.cloud.bigquery.BigQueryException: Broken pipe (Write failed)
    at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)
    at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.insertAll(HttpBigQueryRpc.java:494)
    at com.google.cloud.bigquery.BigQueryImpl.insertAll(BigQueryImpl.java:1036)
Caused by: java.net.SocketException: Broken pipe (Write failed)
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:155)

com.traveloka.accessor.mycustomclass insertDetail: error aa {"UserId":"123","name":"Superman"}
com.google.cloud.bigquery.BigQueryException: Remote host closed connection during handshake
    at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)
    at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.insertAll(HttpBigQueryRpc.java:494)
    at com.google.cloud.bigquery.BigQueryImpl.insertAll(BigQueryImpl.java:1036)
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1004)
    at sun.security.ssl.SSLSocketImpl.waitForClose(SSLSocketImpl.java:1783)
    at sun.security.ssl.HandshakeOutStream.flush(HandshakeOutStream.java:124)
    at sun.security.ssl.Handshaker.sendChangeCipherSpec(Handshaker.java:1156)

com.traveloka.accessor.mycustomclass insertDetail: error aa {"UserId":"123","name":"Superman"}
com.google.cloud.bigquery.BigQueryException: Read timed out
    at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)
    at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.insertAll(HttpBigQueryRpc.java:494)
    at com.google.cloud.bigquery.BigQueryImpl.insertAll(BigQueryImpl.java:1036)
Caused by: java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:171)

com.traveloka.accessor.mycustomclass insertDetail: error aa {"UserId":"123","name":"Superman"}
com.google.cloud.bigquery.BigQueryException: 502 Bad Gateway
POST https://www.googleapis.com/bigquery/v2/projects/myprojectname/datasets/mydatasetname/tables/mytablename/insertAll?prettyPrint=false
<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 502 (Server Error)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>502.</b> <ins>That’s an error.</ins>
  <p>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.  <ins>That’s all we know.</ins>
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 502 Bad Gateway
POST https://www.googleapis.com/bigquery/v2/projects/myprojectname/datasets/mydatasetname/tables/mytablename/insertAll?prettyPrint=false

Request was aborted after waiting too long to attempt to service your request.