Java Amazon Kinesis getRecord()不匹配结果

Java Amazon Kinesis getRecord()不匹配结果,java,amazon-web-services,amazon-kinesis,Java,Amazon Web Services,Amazon Kinesis,我刚开始使用Kinesis,它的API是可用的 我用它将100条记录推送到动觉 for (int j = 0; j < 100; j++) { PutRecordRequest putRecordRequest = new PutRecordRequest(); putRecordRequest.setStreamName(myStreamName); putRecordRequest.setData(ByteBuffer.wrap(data

我刚开始使用Kinesis,它的API是可用的

我用它将100条记录推送到动觉

for (int j = 0; j < 100; j++) {
        PutRecordRequest putRecordRequest = new PutRecordRequest();
        putRecordRequest.setStreamName(myStreamName);
        putRecordRequest.setData(ByteBuffer.wrap(data.getBytes()));   
        putRecordRequest.setPartitionKey(String.format("partitionKey-%d", j));   
        PutRecordResult putRecordResult = kinesisClient.putRecord(putRecordRequest);
        System.out.println("Successfully putrecord, partition key : " + putRecordRequest.getPartitionKey()
                + ", ShardID : " + putRecordResult.getShardId() + ", Sequence No : "+ putRecordResult.getSequenceNumber());
}
这是我的函数getDataCount(shardId,myStreamName)

publicstaticintgetdatacount(字符串shardId,字符串streamName){
int dataCount=0;
串切碎器;
GetShardIteratorRequest GetShardIteratorRequest=新建GetShardIteratorRequest();
getShardIteratorRequest.setStreamName(流名称);
getShardIteratorRequest.setShardId(shardId);
getShardIteratorRequest.setShardIteratorType(ShardIteratorType.TRIM_HORIZON);
GetShardIteratorResult GetShardIteratorResult=kinisClient.getShardIterator(getShardIteratorRequest);
shardIterator=getShardIteratorResult.getShardIterator();
GetRecordsRequest GetRecordsRequest=新建GetRecordsRequest();
getRecordsRequest.setShare迭代器(shardIterator);
getRecordsRequest.setLimit(1000);
GetRecordsResult GetRecordsResult=kinisClient.getRecords(getRecordsRequest);
列表记录=getRecordsResult.getRecords();
如果(!records.isEmpty()&&records.size()>0){
dataCount=records.size();
迭代器迭代器=records.Iterator();
while(iterator.hasNext()){
Record=iterator.next();
byte[]bytes=record.getData().array();
字符串记录数据=新字符串(字节);
System.out.println(“碎片Id:“+shardId+”序号为:“+”记录数据:“+recordData”);
}
}
返回数据计数;
}
但是每次我运行这段代码时,这段代码都会给出不匹配的结果,比如有时它会显示81或91


请说明这一点:)

推送到kinesis的记录有限制(数据/秒),因此如果超过此限制(速度取决于使用的碎片数),记录可能会失败。 现在,您可以使用kinesis API获取失败记录计数,并将其再次推送到kinesis

List<CompletionStage<PutRecordsResponse>> putRecordResponseCompletionStage = <PutRecordRequest call with Stream Name , Partition key and Data>;

AtomicInteger failedCount = new AtomicInteger();
      AtomicInteger recordsCount = new AtomicInteger();

     int loopCount = 0;
      for (CompletionStage<PutRecordsResponse> stage : putRecordResponseCompletionStage) {
        loopCount++;
        try {

          PutRecordsResponse response = stage.toCompletableFuture().get();
          failedCount.addAndGet(response.failedRecordCount());
          recordsCount.addAndGet(response.records().size());
        } catch (InterruptedException e) {
          e.printStackTrace();
        } catch (ExecutionException e) {
          e.printStackTrace();
        }

      }


Check example [here](https://docs.aws.amazon.com/streams/latest/dev/developing-producers-with-sdk.html)
List putRecordResponseCompletionStage=;
AtomicInteger failedCount=新的AtomicInteger();
AtomicInteger recordsCount=新的AtomicInteger();
int loopCount=0;
for(CompletionStage阶段:putRecordResponseCompletionStage){
loopCount++;
试一试{
PutRecordsResponse response=stage.toCompletableFuture().get();
failedCount.addAndGet(response.failedRecordCount());
recordsCount.addAndGet(response.records().size());
}捕捉(中断异常e){
e、 printStackTrace();
}捕获(执行例外){
e、 printStackTrace();
}
}
检查示例[此处](https://docs.aws.amazon.com/streams/latest/dev/developing-producers-with-sdk.html)

能否尝试无限循环碎片以获取记录,直到获得100条记录。每个GetRecords请求可能不会提供您放入该碎片中的所有记录。您是否尝试过替换
List records=getRecordsResult.GetRecords()带有
列表记录=getRecordsResult.getRecords()为了避免类型歧义?根据,在插入流和读取记录的可用性之间有3秒的延迟,是否最后的记录还不可用?你在插入和读取之间等待了吗?我知道现在已经过了两年了。。。研究一下如何使用——基本上是一个小型的库,它可以使从磁带中读取记录变得更容易/更简单,这将是非常有用的
.....
while (shardIterator.hasNext()) {
        Shard shard = shardIterator.next();
        String shardId = shard.getShardId();
        int datacount = getDataCount(shardId, myStreamName);
        totalStreamDataCount+= datacount;
        System.out.println("Data Count for Shard " + shardId + " is : " + datacount); 
}
.....
 public static int getDataCount(String shardId, String streamName) {
        int dataCount = 0;
        String shardIterator;
      GetShardIteratorRequest getShardIteratorRequest = new GetShardIteratorRequest();
      getShardIteratorRequest.setStreamName(streamName);
      getShardIteratorRequest.setShardId(shardId);
      getShardIteratorRequest.setShardIteratorType(ShardIteratorType.TRIM_HORIZON);


      GetShardIteratorResult getShardIteratorResult = kinesisClient.getShardIterator(getShardIteratorRequest);
      shardIterator = getShardIteratorResult.getShardIterator();
      GetRecordsRequest getRecordsRequest = new GetRecordsRequest();
      getRecordsRequest.setShardIterator(shardIterator);
      getRecordsRequest.setLimit(1000);

      GetRecordsResult getRecordsResult = kinesisClient.getRecords(getRecordsRequest);
      List<Record> records = getRecordsResult.getRecords();
      if(!records.isEmpty() && records.size() > 0){
          dataCount = records.size();
          Iterator<Record> iterator = records.iterator();
          while(iterator.hasNext()) {
              Record record = iterator.next();
              byte[] bytes = record.getData().array();
              String recordData = new String(bytes);
              System.out.println("Shard Id. :"+shardId+"Seq. No. is : "+"  Record data :"+recordData);
          }
      }


    return dataCount;
}
List<CompletionStage<PutRecordsResponse>> putRecordResponseCompletionStage = <PutRecordRequest call with Stream Name , Partition key and Data>;

AtomicInteger failedCount = new AtomicInteger();
      AtomicInteger recordsCount = new AtomicInteger();

     int loopCount = 0;
      for (CompletionStage<PutRecordsResponse> stage : putRecordResponseCompletionStage) {
        loopCount++;
        try {

          PutRecordsResponse response = stage.toCompletableFuture().get();
          failedCount.addAndGet(response.failedRecordCount());
          recordsCount.addAndGet(response.records().size());
        } catch (InterruptedException e) {
          e.printStackTrace();
        } catch (ExecutionException e) {
          e.printStackTrace();
        }

      }


Check example [here](https://docs.aws.amazon.com/streams/latest/dev/developing-producers-with-sdk.html)