Java 如何实现非';t字节数组输入流?
我经常从网上下载很多(可能很大)文件 在我的特殊情况下,我需要将这些下载URL的字节输入流发送到另一个服务进行解析 一种天真的方法是这样做:Java 如何实现非';t字节数组输入流?,java,asynchttpclient,Java,Asynchttpclient,我经常从网上下载很多(可能很大)文件 在我的特殊情况下,我需要将这些下载URL的字节输入流发送到另一个服务进行解析 一种天真的方法是这样做: AsyncHttpClient asyncHttpClient = Dsl.asyncHttpClient(Dsl.config() .setMaxConnectionsPerHost(-1) .setMaxConnections(-1) .setPooledConnectionIdleTimeout(60 * 10 * 1000)
AsyncHttpClient asyncHttpClient = Dsl.asyncHttpClient(Dsl.config()
.setMaxConnectionsPerHost(-1)
.setMaxConnections(-1)
.setPooledConnectionIdleTimeout(60 * 10 * 1000)
.setConnectionTtl(6 * 60 * 1000)
.setConnectTimeout(5 * 1000)
.setRequestTimeout(5 * 60 * 1000)
.setFollowRedirect(true)
.setRealm(new Realm.Builder(username, password)
.setNtlmDomain(domain)
.setScheme(Realm.AuthScheme.NTLM)
.build())
Response httpGetResponse = asyncHttpClient.prepareGet(url).execute().get();
return httpGetResponse.getResponseBodyAsStream();
但我们了解到,与HTTP组件HTTP客户端不同,异步HTTP客户端将把整个文件下载到内存中
在我的情况下,这将很快导致OOM
因此,另一种选择是:
Response httpGetResponse = asyncHttpClient.prepareGet(url).execute(new AsyncHandler<Response>() {
private final Response.ResponseBuilder builder = new Response.ResponseBuilder();
@Override
public State onBodyPartReceived(HttpResponseBodyPart bodyPart) throws Exception {
bodyPart.getBodyByteBuffer(); // Each chunk of bytes will be fed into this method.
// I need to write these bytes to the resuting input stream
// without streaming them all into memory.
return State.CONTINUE;
}
@Override
public State onHeadersReceived(HttpHeaders headers) throws Exception {
builder.accumulate(headers);
return State.CONTINUE;
}
@Override
public State onStatusReceived(HttpResponseStatus responseStatus) throws Exception {
builder.accumulate(responseStatus);
return State.CONTINUE;
}
@Override
public Response onCompleted() throws Exception {
return builder.build();
}
@Override
public void onThrowable(Throwable t) {
}
}).get();
Response-httpGetResponse=asyncHttpClient.prepareGet(url).execute(new-asynchHandler()){
private final Response.ResponseBuilder builder=new Response.ResponseBuilder();
@凌驾
onBodyPartReceived的公共状态(HttpResponseBodyPart bodyPart)引发异常{
bodyPart.getBodyByteBuffer();//每个字节块都将被送入此方法。
//我需要将这些字节写入正在恢复的输入流
//不需要把它们全部存入内存。
返回状态。继续;
}
@凌驾
公共状态onHeadersReceived(HttpHeaders)引发异常{
累计(表头);
返回状态。继续;
}
@凌驾
公共状态onStatusReceived(HttpResponseStatus responseStatus)引发异常{
建造商累积(responseStatus);
返回状态。继续;
}
@凌驾
公共响应onCompleted()引发异常{
返回builder.build();
}
@凌驾
可滚动的公共空间(可丢弃的t){
}
}).get();
当这些字节进入输入流时,获取它们的最简单、最干净的方法是什么
我有两个想法:
1) 将输入写入文件,然后对文件进行流式处理
或
2) 立即返回一个管道输入流,接收到的字节将写入管道输入流
有没有人可以与我们分享一个有效的例子 我正确地认为有人已经这样做了。事实上,在搜索“异步http客户端”和“管道输入流”之后,我在项目本身中发现了以下内容: 用法:
PipedInputStream pipedInputStream = new PipedInputStream();
PipedOutputStream pipedOutputStream = new PipedOutputStream(pipedInputStream);
BodyDeferringAsyncHandler bodyDeferringAsyncHandler = new BodyDeferringAsyncHandler(pipedOutputStream);
Future<Response> futureResponse = asyncHttpClient.prepareGet(url).execute(bodyDeferringAsyncHandler);
Response response = bodyDeferringAsyncHandler.getResponse();
if (response.getStatusCode() == 200) {
return new BodyDeferringAsyncHandler.BodyDeferringInputStream(futureResponse,
bodyDeferringAsyncHandler,
pipedInputStream);
} else {
return null;
}
PipedInputStream PipedInputStream=new PipedInputStream();
PipedOutputStream PipedOutputStream=新的PipedOutputStream(pipedInputStream);
BodyDeferringAsyncHandler BodyDeferringAsyncHandler=新的BodyDeferringAsyncHandler(PipedOutStream);
Future futureResponse=asyncHttpClient.prepareGet(url).execute(bodyDeferringAsyncHandler);
Response-Response=bodyDeferringAsyncHandler.getResponse();
if(response.getStatusCode()==200){
返回新的BodyDeferringAsyncHandler.BodyDeferringInputStream(futureResponse,
BodyDeleringAsyncHandler,
pipedInputStream);
}否则{
返回null;
}