C++ AWS CPP TransferManager vs GetObjectRequest流到文件fstream OOM
我使用AWS CPP SDK()在一个小型Linux系统(只有32 MB RAM)上从S3下载一个文件。我使用的是GetObjectRequest类,如下所示。它工作得很好,将文件下载到我的系统上的FStream,这样就不会消耗太多内存 现在,我想将下载代码转换为TransferManager方法,以便获得进度回调。我已经重写了代码的这一部分,如下所示。它一开始很好,打印下载的百分比,但当它达到~14MB RAM(大约是下载时Linux中可用的内存量)时,它会因为使用了太多RAM而被内核杀死 我创建了一个文件流,就像我为GetObjectRequest所做的一样。我做错了什么?我怎样才能解决这个问题?谢谢 不使用所有RAM的旧方法:C++ AWS CPP TransferManager vs GetObjectRequest流到文件fstream OOM,c++,amazon-web-services,aws-sdk,C++,Amazon Web Services,Aws Sdk,我使用AWS CPP SDK()在一个小型Linux系统(只有32 MB RAM)上从S3下载一个文件。我使用的是GetObjectRequest类,如下所示。它工作得很好,将文件下载到我的系统上的FStream,这样就不会消耗太多内存 现在,我想将下载代码转换为TransferManager方法,以便获得进度回调。我已经重写了代码的这一部分,如下所示。它一开始很好,打印下载的百分比,但当它达到~14MB RAM(大约是下载时Linux中可用的内存量)时,它会因为使用了太多RAM而被内核杀死 我
// Old way
GetObjectRequest getObjectRequest;
getObjectRequest.SetBucket(bucket.c_str());
getObjectRequest.SetKey(keyName.c_str());
getObjectRequest.SetResponseStreamFactory([&destination](){
return Aws::New<Aws::FStream>(
"s3file", destination, std::ios_base::out); });
GetObjectOutcome getObjectOutcome = SessionClient->GetObject(getObjectRequest);
if(getObjectOutcome.IsSuccess())
{
std::cout << "<AWS DOWNLOAD> Get FW success!" << std::endl;
}
else
{
std::cout << "<AWS DOWNLOAD> Get FW failed: " << getObjectOutcome.GetError().GetMessage() << std::endl;
exit(1);
}
//老路
GetObjectRequest GetObjectRequest;
getObjectRequest.SetBucket(bucket.c_str());
getObjectRequest.SetKey(keyName.c_str());
getObjectRequest.SetResponseStreamFactory([&destination](){
返回Aws::新建(
“s3file”,目标,std::ios_base::out);});
GetObjectOutput GetObjectOutput=SessionClient->GetObject(getObjectRequest);
if(GetObjectOutput.IsSuccess())
{
std::coutTransferManager只会在您位于10mb或更大的范围内并且希望利用并行化时使事情变得更容易。它将预先分配最大堆大小,并且不会使堆的增长超过该大小。考虑到您的RAM限制,我不会使用TransferManager。您仍然可以收到进度通知。C查看AmazonWebServiceRequest类中的回调机制
// New way
Aws::Transfer::TransferManagerConfiguration transferConfig;
transferConfig.s3Client = SessionClient;
std::shared_ptr<Aws::Transfer::TransferHandle> requestPtr(nullptr);
transferConfig.downloadProgressCallback =
[](const Aws::Transfer::TransferManager*, const Aws::Transfer::TransferHandle& handle)
{
std::cout << "\r" << "<AWS DOWNLOAD> Download Progress: " << static_cast<int>(handle.GetBytesTransferred() * 100.0 / handle.GetBytesTotalSize()) << " Percent " << handle.GetBytesTransferred() << " bytes\n";
};
Aws::Transfer::TransferManager transferManager(transferConfig);
requestPtr = transferManager.DownloadFile(bucket.c_str(), keyName.c_str(), [&destination](){
Aws::FStream *stream = Aws::New<Aws::FStream>("s3file", destination, std::ios_base::out);
stream->rdbuf()->pubsetbuf(NULL, 0);
return stream; });
requestPtr->WaitUntilFinished();
size_t retries = 0;
//just make sure we don't fail because a download part failed. (e.g. network problems or interuptions)
while (requestPtr->GetStatus() == Aws::Transfer::TransferStatus::FAILED && retries++ < 5)
{
std::cout << "<AWS DOWNLOAD> FW Download trying download again!" << std::endl;
transferManager.RetryDownload(requestPtr);
requestPtr->WaitUntilFinished();
}
// Check status
if ( requestPtr->GetStatus() == Aws::Transfer::TransferStatus::COMPLETED ) {
if ( requestPtr->GetBytesTotalSize() == requestPtr->GetBytesTransferred() ) {
std::cout << "<AWS DOWNLOAD> Get FW success!" << std::endl;
exit(0);
}
else {
std::cout << "<AWS DOWNLOAD> Get FW failed - Bytes downloaded did not equal requested number of bytes: " << requestPtr->GetBytesTotalSize() << requestPtr->GetBytesTransferred() << std::endl;
exit(1);
}
}
else {
std::cout << "<AWS DOWNLOAD> Get FW failed - download was never completed even after retries" << std::endl;
exit(1);
}