C++ AWS CPP TransferManager vs GetObjectRequest流到文件fstream OOM

C++ AWS CPP TransferManager vs GetObjectRequest流到文件fstream OOM,c++,amazon-web-services,aws-sdk,C++,Amazon Web Services,Aws Sdk,我使用AWS CPP SDK()在一个小型Linux系统(只有32 MB RAM)上从S3下载一个文件。我使用的是GetObjectRequest类,如下所示。它工作得很好,将文件下载到我的系统上的FStream,这样就不会消耗太多内存 现在,我想将下载代码转换为TransferManager方法,以便获得进度回调。我已经重写了代码的这一部分,如下所示。它一开始很好,打印下载的百分比,但当它达到~14MB RAM(大约是下载时Linux中可用的内存量)时,它会因为使用了太多RAM而被内核杀死 我

我使用AWS CPP SDK()在一个小型Linux系统(只有32 MB RAM)上从S3下载一个文件。我使用的是GetObjectRequest类,如下所示。它工作得很好,将文件下载到我的系统上的FStream,这样就不会消耗太多内存

现在,我想将下载代码转换为TransferManager方法,以便获得进度回调。我已经重写了代码的这一部分,如下所示。它一开始很好,打印下载的百分比,但当它达到~14MB RAM(大约是下载时Linux中可用的内存量)时,它会因为使用了太多RAM而被内核杀死

我创建了一个文件流,就像我为GetObjectRequest所做的一样。我做错了什么?我怎样才能解决这个问题?谢谢

不使用所有RAM的旧方法:

    // Old way
    GetObjectRequest getObjectRequest;
    getObjectRequest.SetBucket(bucket.c_str());
    getObjectRequest.SetKey(keyName.c_str());
    getObjectRequest.SetResponseStreamFactory([&destination](){
     return Aws::New<Aws::FStream>(
     "s3file", destination, std::ios_base::out); });

    GetObjectOutcome getObjectOutcome = SessionClient->GetObject(getObjectRequest);
    if(getObjectOutcome.IsSuccess())
    {
        std::cout << "<AWS DOWNLOAD> Get FW success!" << std::endl;
    }
    else
    {
        std::cout << "<AWS DOWNLOAD> Get FW failed: " << getObjectOutcome.GetError().GetMessage() << std::endl;
        exit(1);
    }
//老路
GetObjectRequest GetObjectRequest;
getObjectRequest.SetBucket(bucket.c_str());
getObjectRequest.SetKey(keyName.c_str());
getObjectRequest.SetResponseStreamFactory([&destination](){
返回Aws::新建(
“s3file”,目标,std::ios_base::out);});
GetObjectOutput GetObjectOutput=SessionClient->GetObject(getObjectRequest);
if(GetObjectOutput.IsSuccess())
{

std::coutTransferManager只会在您位于10mb或更大的范围内并且希望利用并行化时使事情变得更容易。它将预先分配最大堆大小,并且不会使堆的增长超过该大小。考虑到您的RAM限制,我不会使用TransferManager。您仍然可以收到进度通知。C查看AmazonWebServiceRequest类中的回调机制

// New way
Aws::Transfer::TransferManagerConfiguration transferConfig;
transferConfig.s3Client = SessionClient;

std::shared_ptr<Aws::Transfer::TransferHandle> requestPtr(nullptr);

transferConfig.downloadProgressCallback =
        [](const Aws::Transfer::TransferManager*, const Aws::Transfer::TransferHandle& handle)
{
    std::cout << "\r" << "<AWS DOWNLOAD> Download Progress: " << static_cast<int>(handle.GetBytesTransferred() * 100.0 / handle.GetBytesTotalSize()) << " Percent " << handle.GetBytesTransferred() << " bytes\n";
};

Aws::Transfer::TransferManager transferManager(transferConfig);

requestPtr = transferManager.DownloadFile(bucket.c_str(), keyName.c_str(), [&destination](){

    Aws::FStream *stream = Aws::New<Aws::FStream>("s3file", destination, std::ios_base::out);
    stream->rdbuf()->pubsetbuf(NULL, 0);

    return stream; });

requestPtr->WaitUntilFinished();

size_t retries = 0;
//just make sure we don't fail because a download part failed. (e.g. network problems or interuptions)
while (requestPtr->GetStatus() == Aws::Transfer::TransferStatus::FAILED && retries++ < 5)
{
    std::cout << "<AWS DOWNLOAD> FW Download trying download again!" << std::endl;
    transferManager.RetryDownload(requestPtr);
    requestPtr->WaitUntilFinished();
}

// Check status
if ( requestPtr->GetStatus() == Aws::Transfer::TransferStatus::COMPLETED ) {
    if ( requestPtr->GetBytesTotalSize() == requestPtr->GetBytesTransferred() ) {
        std::cout << "<AWS DOWNLOAD> Get FW success!" << std::endl;
        exit(0);
    }
    else {
        std::cout << "<AWS DOWNLOAD> Get FW failed - Bytes downloaded did not equal requested number of bytes: " << requestPtr->GetBytesTotalSize() << requestPtr->GetBytesTransferred() << std::endl;
        exit(1);
    }
}
else {
    std::cout << "<AWS DOWNLOAD> Get FW failed - download was never completed even after retries" << std::endl;
    exit(1);
}