Java 在合并多个ByteBuff时,在非OLEDBYTEBuffAllocator和在netty中池中进行选择

Java 在合并多个ByteBuff时,在非OLEDBYTEBuffAllocator和在netty中池中进行选择,java,netty,asynchttpclient,Java,Netty,Asynchttpclient,我使用图书馆下载文件。 例如,我需要从一台主机下载500个文件,每个文件大小约为10MB,然后将这些文件写入磁盘。对于本例,我将提供以下代码: 声明异步HttpClient。我使用ResponseBodyPartFactory.LAZY来避免复制到java堆中 final AsyncHttpClient asyncHttpClient = Dsl.asyncHttpClient(Dsl.config() .setFollowRedirect(true)

我使用图书馆下载文件。 例如,我需要从一台主机下载500个文件,每个文件大小约为10MB,然后将这些文件写入磁盘。对于本例,我将提供以下代码:

  • 声明异步HttpClient。我使用ResponseBodyPartFactory.LAZY来避免复制到java堆中

    final AsyncHttpClient asyncHttpClient = Dsl.asyncHttpClient(Dsl.config()
                 .setFollowRedirect(true)
                 .setResponseBodyPartFactory(AsyncHttpClientConfig.ResponseBodyPartFactory.LAZY)
                 .setUseOpenSsl(true)
                 .setRequestTimeout(6000000)
                 .setUseNativeTransport(true)
                 .build());
    
  • 声明ExecutorService,它将用于写入文件。使用固定大小,因为默认执行器服务可以创建1000多个线程(他使用cachedThreadPool)

  • 文件下载代码。我使用localhost是因为我想获得最大的下载速度,然后将这些文件写回磁盘。主要部分在方法onBodyPartReceived

     for (int i = 0; i < downloadCount; i++) {
     asyncHttpClient.prepareGet("http://localhost/Pizigani_1367_Chart_10MB.jpg").execute(new AsyncHandler<Void>() {
         private long currentFilePosition;
         private AsynchronousFileChannel fileChannel;
         private CompositeByteBuf compositeByteBuf;
    
    
         @Override
         public AsyncHandler.State onStatusReceived(HttpResponseStatus status) throws IOException {
             if (status.getStatusCode() != 200) {
                 return State.ABORT;
             }
             this.fileChannel = AsynchronousFileChannel.open(Paths.get("/tmp/" + UUID.randomUUID().toString() + "file.jpg"), EnumSet.of(StandardOpenOption.WRITE, StandardOpenOption.TRUNCATE_EXISTING, StandardOpenOption.CREATE), executorService);
             this.compositeByteBuf = Unpooled.compositeBuffer(16);
             return State.CONTINUE;
         }
    
         @Override
         public State onHeadersReceived(HttpHeaders headers) {
             return State.CONTINUE;
         }
    
         @Override
         public void onThrowable(Throwable t) {
         }
    
         @Override
         public Void onCompleted() {
             return null;
         }
    
         @Override
         public State onBodyPartReceived(HttpResponseBodyPart content) {
             final ByteBuf responseByteBuf = ((LazyResponseBodyPart) content).getBuf().retain();
             final CompositeByteBuf compositeByteBuf = this.compositeByteBuf.addComponent(true, responseByteBuf);
             if (compositeByteBuf.numComponents() == compositeByteBuf.maxNumComponents() || content.isLast()) {
                 final ByteBuf mergedDirectBuffer = UnpooledByteBufAllocator.DEFAULT.directBuffer(compositeByteBuf.readableBytes()); //or Pooled?
                 for (ByteBuffer b : compositeByteBuf.nioBuffers()) {
                     mergedDirectBuffer.writeBytes(b);
                 }
                 compositeByteBuf.release();
                 ByteBuffer asyncBuffer = mergedDirectBuffer.nioBuffer().rewind();
    
                 long currentFilePosition= this.currentFilePosition;
                 this.currentFilePosition += mergedDirectBuffer.readableBytes();
                 this.compositeByteBuf = Unpooled.compositeBuffer(16);
                 fileChannel.write(asyncBuffer, currentFilePosition, mergedDirectBuffer, new CompletionHandler<>() {
                     @Override
                     public void completed(Integer result, ByteBuf attachment) {
                         attachment.release();
                         if (content.isLast()) {
                             try {
                                 fileChannel.close();
                             } catch (IOException e) {
                                 throw new UncheckedIOException(e);
                             }
                         }
                     }
    
                     @Override
                     public void failed(Throwable exc, ByteBuf attachment) {
                         attachment.release();
                         try {
                             fileChannel.close();
                         } catch (IOException e) {
                             throw new UncheckedIOException(e);
                         }
                     }
                 });
    
             }
    
             return State.CONTINUE;
         }
     });
    
  • 所以我提出了一个问题。 在将缓冲区合并到新的直接缓冲区时,我应该选择池式还是非池式?只是我的一些测量结果表明,未经冷却的速度更快,而共用的通常不会释放内存

    将组合缓冲区写入磁盘的方法正确吗?也许有更好的方法? 我不太了解内蒂

     for (int i = 0; i < downloadCount; i++) {
     asyncHttpClient.prepareGet("http://localhost/Pizigani_1367_Chart_10MB.jpg").execute(new AsyncHandler<Void>() {
         private long currentFilePosition;
         private AsynchronousFileChannel fileChannel;
         private CompositeByteBuf compositeByteBuf;
    
    
         @Override
         public AsyncHandler.State onStatusReceived(HttpResponseStatus status) throws IOException {
             if (status.getStatusCode() != 200) {
                 return State.ABORT;
             }
             this.fileChannel = AsynchronousFileChannel.open(Paths.get("/tmp/" + UUID.randomUUID().toString() + "file.jpg"), EnumSet.of(StandardOpenOption.WRITE, StandardOpenOption.TRUNCATE_EXISTING, StandardOpenOption.CREATE), executorService);
             this.compositeByteBuf = Unpooled.compositeBuffer(16);
             return State.CONTINUE;
         }
    
         @Override
         public State onHeadersReceived(HttpHeaders headers) {
             return State.CONTINUE;
         }
    
         @Override
         public void onThrowable(Throwable t) {
         }
    
         @Override
         public Void onCompleted() {
             return null;
         }
    
         @Override
         public State onBodyPartReceived(HttpResponseBodyPart content) {
             final ByteBuf responseByteBuf = ((LazyResponseBodyPart) content).getBuf().retain();
             final CompositeByteBuf compositeByteBuf = this.compositeByteBuf.addComponent(true, responseByteBuf);
             if (compositeByteBuf.numComponents() == compositeByteBuf.maxNumComponents() || content.isLast()) {
                 final ByteBuf mergedDirectBuffer = UnpooledByteBufAllocator.DEFAULT.directBuffer(compositeByteBuf.readableBytes()); //or Pooled?
                 for (ByteBuffer b : compositeByteBuf.nioBuffers()) {
                     mergedDirectBuffer.writeBytes(b);
                 }
                 compositeByteBuf.release();
                 ByteBuffer asyncBuffer = mergedDirectBuffer.nioBuffer().rewind();
    
                 long currentFilePosition= this.currentFilePosition;
                 this.currentFilePosition += mergedDirectBuffer.readableBytes();
                 this.compositeByteBuf = Unpooled.compositeBuffer(16);
                 fileChannel.write(asyncBuffer, currentFilePosition, mergedDirectBuffer, new CompletionHandler<>() {
                     @Override
                     public void completed(Integer result, ByteBuf attachment) {
                         attachment.release();
                         if (content.isLast()) {
                             try {
                                 fileChannel.close();
                             } catch (IOException e) {
                                 throw new UncheckedIOException(e);
                             }
                         }
                     }
    
                     @Override
                     public void failed(Throwable exc, ByteBuf attachment) {
                         attachment.release();
                         try {
                             fileChannel.close();
                         } catch (IOException e) {
                             throw new UncheckedIOException(e);
                         }
                     }
                 });
    
             }
    
             return State.CONTINUE;
         }
     });
    
            if (compositeByteBuf.numComponents() == compositeByteBuf.maxNumComponents() || content.isLast()) {
                final ByteBuf mergedDirectBuffer = UnpooledByteBufAllocator.DEFAULT.directBuffer(compositeByteBuf.readableBytes()); //or Pooled?
                for (ByteBuffer b : compositeByteBuf.nioBuffers()) {
                    mergedDirectBuffer.writeBytes(b);
                }
                compositeByteBuf.release();
                ByteBuffer asyncBuffer = mergedDirectBuffer.nioBuffer().rewind();
    
                long currentFilePosition= this.currentFilePosition;
                this.currentFilePosition += mergedDirectBuffer.readableBytes();
                this.compositeByteBuf = Unpooled.compositeBuffer(16);
                fileChannel.write(asyncBuffer, currentFilePosition, mergedDirectBuffer, new CompletionHandler<>() {