C++ 为什么HDF5输出Caffe层写入看似不正确尺寸的数据?
我对将给定Caffe层的输出写入文件感兴趣。我希望对多个图像执行此操作,因此我对HDF5输出层代码进行了一些修改,以便为每个图像创建一个文件,其中包含每个图像的功能。以下是修改后的SaveBlobs函数:C++ 为什么HDF5输出Caffe层写入看似不正确尺寸的数据?,c++,machine-learning,neural-network,deep-learning,caffe,C++,Machine Learning,Neural Network,Deep Learning,Caffe,我对将给定Caffe层的输出写入文件感兴趣。我希望对多个图像执行此操作,因此我对HDF5输出层代码进行了一些修改,以便为每个图像创建一个文件,其中包含每个图像的功能。以下是修改后的SaveBlobs函数: template <typename Dtype> void HDF5OutputLayer<Dtype>::SaveBlobs() { LOG(INFO) << "Saving HDF5 f
template <typename Dtype> void HDF5OutputLayer<Dtype>::SaveBlobs() {
LOG(INFO) << "Saving HDF5 file " << file_name_ << "ds: " << ds_iter_;
CHECK_EQ(data_blob_.num(), label_blob_.num()) <<
"data blob and label blob must have the same batch size";
// Open hdf5 file to write this blob
file_name_ = this->layer_param_.hdf5_output_param().file_name();
ostringstream appender;
appender << "_" << ds_iter_ << ".h5";
file_name_.append(appender.str());
file_id_ = H5Fcreate(file_name_.c_str(), H5F_ACC_TRUNC, H5P_DEFAULT,
H5P_DEFAULT);
CHECK_GE(file_id_, 0) << "Failed to open HDF5 file" << file_name_;
// Write the data and label
hdf5_save_nd_dataset(file_id_, HDF5_DATA_DATASET_NAME, data_blob_);
hdf5_save_nd_dataset(file_id_, HDF5_DATA_LABEL_NAME, label_blob_);
LOG(INFO) << "Successfully saved " << data_blob_.num() << " rows";
LOG(INFO) << "SAVEBLOB - Data size is: " << data_blob_.shape_string();
LOG(INFO) << "SAVEBLOB - Label size is: " << label_blob_.shape_string();
// Close the file
herr_t status = H5Fclose(file_id_);
CHECK_GE(status, 0) << "Failed to close HDF5 file " << file_name_;
// Update iterator for next image
ds_iter_++;
}
我之所以认为保存的数据不正确,是因为当我观察网络设置时,conv5_3顶部的尺寸如下所示:
I0206 23:07:44.815330 7630 layer_factory.hpp:77] Creating layer conv5_3_relu5_3_0_split
I0206 23:07:44.815343 7630 net.cpp:106] Creating Layer conv5_3_relu5_3_0_split
I0206 23:07:44.815348 7630 net.cpp:454] conv5_3_relu5_3_0_split <- conv5_3
I0206 23:07:44.815356 7630 net.cpp:411] conv5_3_relu5_3_0_split -> conv5_3_relu5_3_0_split_0
I0206 23:07:44.815366 7630 net.cpp:411] conv5_3_relu5_3_0_split -> conv5_3_relu5_3_0_split_1
I0206 23:07:44.815372 7630 net.cpp:411] conv5_3_relu5_3_0_split -> conv5_3_relu5_3_0_split_2
I0206 23:07:44.815382 7630 net.cpp:411] conv5_3_relu5_3_0_split -> conv5_3_relu5_3_0_split_3
I0206 23:07:44.815459 7630 net.cpp:150] Setting up conv5_3_relu5_3_0_split
I0206 23:07:44.815467 7630 net.cpp:157] Top shape: 1 512 14 14 (100352)
I0206 23:07:44.815474 7630 net.cpp:157] Top shape: 1 512 14 14 (100352)
I0206 23:07:44.815479 7630 net.cpp:157] Top shape: 1 512 14 14 (100352)
I0206 23:07:44.815484 7630 net.cpp:157] Top shape: 1 512 14 14 (100352)
I0206 23:07:44.815495 7630 net.cpp:165] Memory required for data: 116006912
I0206 23:07:44.815500 7630 layer_factory.hpp:77] Creating layer hdf5output
I0206 23:07:44.815511 7630 net.cpp:106] Creating Layer hdf5output
I0206 23:07:44.815515 7630 net.cpp:454] hdf5output <- conv5_3_relu5_3_0_split_0
I0206 23:07:44.815521 7630 net.cpp:454] hdf5output <- conv5_3_relu5_3_0_split_1
I0206 23:07:44.815527 7630 net.cpp:150] Setting up hdf5output
I0206 23:07:44.815531 7630 net.cpp:165] Memory required for data: 116006912
这表明不仅输出的尺寸不正确,而且它们在迭代(图像)之间也不同!日志中显示的维度与写入h5文件的数据维度相匹配,因此日志准确地描述了代码的行为。我的问题是为什么会这样?似乎我已经正确地设置了所有内容,但一定有什么我遗漏了…正如@hbaderts帮助我发现的,结果证明HDF5层是正确的,并且输出了正确维度的数据。我对维度的混淆是因为网络原型测试版本中输入维度的定义似乎是静态的。事实证明,由于我使用pycaffe函数net.forward(**forward_kwargs)加载数据,卷积层会自行缩放以满足输入图像的不同输入维度。这解释了大于预期的特征尺寸以及它们在图像之间变化的事实。正如@hbaderts帮助我发现的,结果证明HDF5层是正确的,并且输出了正确尺寸的数据。我对维度的混淆是因为网络原型测试版本中输入维度的定义似乎是静态的。事实证明,由于我使用pycaffe函数net.forward(**forward_kwargs)加载数据,卷积层会自行缩放以满足输入图像的不同输入维度。这就解释了比预期尺寸更大的特征尺寸以及它们在图像之间的差异。您确定所有输入到网络中的图像大小相同吗?谢谢您的回复!JPEG图像的大小不同,但输入数据层指定为1 x 3 x 224 x 224。因此,就网络而言,所有图像的大小都与按输入尺寸缩放的图像大小相同。让我猜猜:您的第一个图像是864x608像素,第二个是896x608像素,第三个是608x800像素?你能展示一下你是如何将图像加载到网络中的吗?即,您有什么数据层,以及如何运行caffe?我认为您修改的HDF5层很好,加载数据是个问题。您确定所有输入到网络中的图像大小都相同吗?谢谢您的回复!JPEG图像的大小不同,但输入数据层指定为1 x 3 x 224 x 224。因此,就网络而言,所有图像的大小都与按输入尺寸缩放的图像大小相同。让我猜猜:您的第一个图像是864x608像素,第二个是896x608像素,第三个是608x800像素?你能展示一下你是如何将图像加载到网络中的吗?即,您有什么数据层,以及如何运行caffe?我认为修改后的HDF5层很好,加载数据就是问题所在。
I0206 23:07:44.815330 7630 layer_factory.hpp:77] Creating layer conv5_3_relu5_3_0_split
I0206 23:07:44.815343 7630 net.cpp:106] Creating Layer conv5_3_relu5_3_0_split
I0206 23:07:44.815348 7630 net.cpp:454] conv5_3_relu5_3_0_split <- conv5_3
I0206 23:07:44.815356 7630 net.cpp:411] conv5_3_relu5_3_0_split -> conv5_3_relu5_3_0_split_0
I0206 23:07:44.815366 7630 net.cpp:411] conv5_3_relu5_3_0_split -> conv5_3_relu5_3_0_split_1
I0206 23:07:44.815372 7630 net.cpp:411] conv5_3_relu5_3_0_split -> conv5_3_relu5_3_0_split_2
I0206 23:07:44.815382 7630 net.cpp:411] conv5_3_relu5_3_0_split -> conv5_3_relu5_3_0_split_3
I0206 23:07:44.815459 7630 net.cpp:150] Setting up conv5_3_relu5_3_0_split
I0206 23:07:44.815467 7630 net.cpp:157] Top shape: 1 512 14 14 (100352)
I0206 23:07:44.815474 7630 net.cpp:157] Top shape: 1 512 14 14 (100352)
I0206 23:07:44.815479 7630 net.cpp:157] Top shape: 1 512 14 14 (100352)
I0206 23:07:44.815484 7630 net.cpp:157] Top shape: 1 512 14 14 (100352)
I0206 23:07:44.815495 7630 net.cpp:165] Memory required for data: 116006912
I0206 23:07:44.815500 7630 layer_factory.hpp:77] Creating layer hdf5output
I0206 23:07:44.815511 7630 net.cpp:106] Creating Layer hdf5output
I0206 23:07:44.815515 7630 net.cpp:454] hdf5output <- conv5_3_relu5_3_0_split_0
I0206 23:07:44.815521 7630 net.cpp:454] hdf5output <- conv5_3_relu5_3_0_split_1
I0206 23:07:44.815527 7630 net.cpp:150] Setting up hdf5output
I0206 23:07:44.815531 7630 net.cpp:165] Memory required for data: 116006912
I0206 23:07:46.108660 7630 hdf5_output_layer.cpp:31] Saving HDF5 file ds: 0
I0206 23:07:46.115536 7630 hdf5_output_layer.cpp:48] Successfully saved 1 rows
I0206 23:07:46.115557 7630 hdf5_output_layer.cpp:49] SAVEBLOB - Data size is: 1 512 54 38 (1050624)
I0206 23:07:46.115566 7630 hdf5_output_layer.cpp:50] SAVEBLOB - Label size is: 1 512 54 38 (1050624)
I0206 23:07:46.316557 7630 hdf5_output_layer.cpp:31] Saving HDF5 file ./test_features/image_0.h5ds: 1
I0206 23:07:46.322437 7630 hdf5_output_layer.cpp:48] Successfully saved 1 rows
I0206 23:07:46.322456 7630 hdf5_output_layer.cpp:49] SAVEBLOB - Data size is: 1 512 56 38 (1089536)
I0206 23:07:46.322463 7630 hdf5_output_layer.cpp:50] SAVEBLOB - Label size is: 1 512 56 38 (1089536)
I0206 23:07:46.457828 7630 hdf5_output_layer.cpp:31] Saving HDF5 file ./test_features/image_1.h5ds: 2
I0206 23:07:46.463618 7630 hdf5_output_layer.cpp:48] Successfully saved 1 rows
I0206 23:07:46.463636 7630 hdf5_output_layer.cpp:49] SAVEBLOB - Data size is: 1 512 38 50 (972800)
I0206 23:07:46.463644 7630 hdf5_output_layer.cpp:50] SAVEBLOB - Label size is: 1 512 38 50 (972800)
I0206 23:07:46.594746 7630 hdf5_output_layer.cpp:31] Saving HDF5 file ./test_features/image_2.h5ds: 3