Mxnet 如何理解SageMaker图像分类示例日志中的速度数

我正在玩加州理工学院图像分类笔记本的SageMaker示例：。我按照笔记本中的步骤操作，但将资源部分改为使用

ml.p3.16xlarge

，它有8个V100 GPU，如下所示：

"ResourceConfig": {
    "InstanceCount": 1,
    "InstanceType": "ml.p3.16xlarge",
    "VolumeSizeInGB": 50
}

当我在训练后查看日志文件时，我发现速度仅为

895图像/s

，这与使用单个GPU（p3.2xlarge）非常相似。我猜速度仅适用于单个GPU，而我使用8个GPU时的实际速度应该是

895*8=7160

。有人能证实这一点吗？或者我错了

完整日志见下文：

Docker entrypoint called with argument(s): train
[10/02/2018 21:40:21 INFO 139764860892992] Reading default configuration from /opt/amazon/lib/python2.7/site-packages/image_classification/default-input.json: {u'beta_1': 0.9, u'gamma': 0.9, u'beta_2': 0.999, u'optimizer': u'sgd', u'use_pretrained_model': 0, u'eps': 1e-08, u'epochs': 30, u'lr_scheduler_factor': 0.1, u'num_layers': 152, u'image_shape': u'3,224,224', u'precision_dtype': u'float32', u'mini_batch_size': 32, u'weight_decay': 0.0001, u'learning_rate': 0.1, u'momentum': 0}
[10/02/2018 21:40:21 INFO 139764860892992] Reading provided configuration from /opt/ml/input/config/hyperparameters.json: {u'learning_rate': u'0.01', u'use_pretrained_model': u'1', u'epochs': u'2', u'num_training_samples': u'15420', u'num_layers': u'18', u'mini_batch_size': u'512', u'image_shape': u'3,224,224', u'num_classes': u'257'}
[10/02/2018 21:40:21 INFO 139764860892992] Final configuration: {u'optimizer': u'sgd', u'learning_rate': u'0.01', u'epochs': u'2', u'lr_scheduler_factor': 0.1, u'num_layers': u'18', u'precision_dtype': u'float32', u'mini_batch_size': u'512', u'num_classes': u'257', u'beta_1': 0.9, u'beta_2': 0.999, u'use_pretrained_model': u'1', u'eps': 1e-08, u'weight_decay': 0.0001, u'momentum': 0, u'image_shape': u'3,224,224', u'gamma': 0.9, u'num_training_samples': u'15420'}
[10/02/2018 21:40:21 INFO 139764860892992] Using pretrained model for initalizing weights
[10/02/2018 21:40:21 INFO 139764860892992] ---- Parameters ----
[10/02/2018 21:40:21 INFO 139764860892992] num_layers: 18
[10/02/2018 21:40:21 INFO 139764860892992] data type: <type 'numpy.float32'>
[10/02/2018 21:40:21 INFO 139764860892992] epochs: 2
[10/02/2018 21:40:21 INFO 139764860892992] optimizer: sgd
[10/02/2018 21:40:21 INFO 139764860892992] momentum: 0.900000
[10/02/2018 21:40:21 INFO 139764860892992] weight_decay: 0.000100
[10/02/2018 21:40:21 INFO 139764860892992] learning_rate: 0.010000
[10/02/2018 21:40:21 INFO 139764860892992] lr_scheduler_step defined without lr_scheduler_factor, will be ignored...
[10/02/2018 21:40:21 INFO 139764860892992] mini_batch_size: 512
[10/02/2018 21:40:21 INFO 139764860892992] image_shape: 3,224,224
[10/02/2018 21:40:21 INFO 139764860892992] num_classes: 257
[10/02/2018 21:40:21 INFO 139764860892992] num_training_samples: 15420
[10/02/2018 21:40:21 INFO 139764860892992] augmentation_type: None
[10/02/2018 21:40:21 INFO 139764860892992] kv_store: device
[10/02/2018 21:40:21 INFO 139764860892992] checkpoint_frequency: 2
[10/02/2018 21:40:21 INFO 139764860892992] multi_label: 0
[10/02/2018 21:40:21 INFO 139764860892992] --------------------
[21:40:21] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.2.x.288.0/RHEL5_64/generic-flavor/src/src/nnvm/legacy_json_util.cc:209: Loading symbol saved by previous version v0.8.0. Attempting to upgrade...
[21:40:21] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.2.x.288.0/RHEL5_64/generic-flavor/src/src/nnvm/legacy_json_util.cc:217: Symbol successfully upgraded!
[10/02/2018 21:40:21 INFO 139764860892992] Setting number of threads: 63
[21:41:02] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.2.x.288.0/RHEL5_64/generic-flavor/src/src/kvstore/././comm.h:634: only 32 out of 56 GPU pairs are enabled direct access. It may affect the performance. You can set MXNET_ENABLE_GPU_P2P=0 to turn it off
[21:41:02] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.2.x.288.0/RHEL5_64/generic-flavor/src/src/kvstore/././comm.h:643: .vvvv...
[21:41:02] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.2.x.288.0/RHEL5_64/generic-flavor/src/src/kvstore/././comm.h:643: v.vv.v..
[21:41:02] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.2.x.288.0/RHEL5_64/generic-flavor/src/src/kvstore/././comm.h:643: vv.v..v.
[21:41:02] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.2.x.288.0/RHEL5_64/generic-flavor/src/src/kvstore/././comm.h:643: vvv....v
[21:41:02] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.2.x.288.0/RHEL5_64/generic-flavor/src/src/kvstore/././comm.h:643: v....vvv
[21:41:02] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.2.x.288.0/RHEL5_64/generic-flavor/src/src/kvstore/././comm.h:643: .v..v.vv
[21:41:02] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.2.x.288.0/RHEL5_64/generic-flavor/src/src/kvstore/././comm.h:643: ..v.vv.v
[21:41:02] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.2.x.288.0/RHEL5_64/generic-flavor/src/src/kvstore/././comm.h:643: ...vvvv.
[21:41:03] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.2.x.288.0/RHEL5_64/generic-flavor/src/src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:107: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
[10/02/2018 21:41:18 INFO 139764860892992] Epoch[0] Batch [20]#011Speed: 903.34 samples/sec#011accuracy=0.020554
[10/02/2018 21:41:23 INFO 139764860892992] Epoch[0] Train-accuracy=0.055990
[10/02/2018 21:41:23 INFO 139764860892992] Epoch[0] Time cost=21.168
[10/02/2018 21:41:30 INFO 139764860892992] Epoch[0] Validation-accuracy=0.257747
[10/02/2018 21:41:42 INFO 139764860892992] Epoch[1] Batch [20]#011Speed: 895.73 samples/sec#011accuracy=0.393694
[10/02/2018 21:41:47 INFO 139764860892992] Epoch[1] Train-accuracy=0.439128
[10/02/2018 21:41:47 INFO 139764860892992] Epoch[1] Time cost=17.307
[10/02/2018 21:41:48 INFO 139764860892992] Saved checkpoint to "/opt/ml/model/image-classification-0002.params"
[10/02/2018 21:41:53 INFO 139764860892992] Epoch[1] Validation-accuracy=0.561719

使用参数调用Docker入口点：train
[10/02/2018 21:40:21信息139764860892992]从/opt/amazon/lib/python2.7/site-packages/image_classification/default-input.json读取默认配置：{u'beta_1'：0.9，u'gamma'：0.9，u'beta_2'：0.999，u'optimizer'：u'sgd'，u'use_pretrained_model'：0，u'eps'：1e-08，u'epochs'：30，u'lr_调度程序_factor'：0.1，u'num_layers'：152，u'image_形状：u'32242; 24224;'，u'precision_dtype'：u'float32'，u'mini_批量大小：32，u'weight(重量：0.0001，u'learning rate，u'momentum'0.0001，u'
[10/02/2018 21:40:21 INFO 139764860892992]从/opt/ml/input/config/hyperparameters.json阅读提供的配置：{u'learning_rate'：u'0.01'，u'use_pretrained_model'：u'1'，u'epochs'：u'2'，u'num training_samples'：u'15420'，u'num layers'：u'18'，u'mini_batch_size'：u'512'，u'image_'u'shape'：u'32242; u'num classes'：u'257'}
[10/02/2018 21:40:21信息139764860892992]最终配置：{u'优化器：u'sgd'，u'learning_rate'：u'0.01'，u'epochs'：u'2'，u'lr_scheduler_factor'：0.1，u'num_layers'：u'18'，u'precision_dtype'：u'float32'，u'mini__批量大小：u'512'，u'num__类：u'257'，u'beta_1'：0.9，u'beta_2'：0.999，u'use_-rained_模型：u'1'，u'1'，u'eps-08衰减，u'weight"u'22'，u'u'22，u'momentum'：u'4'，u'gamma'：0.9，u'num_训练样本：u'15420'}
[10/02/2018 21:40:21信息139764860892992]使用预训练模型初始化权重
[10/02/2018 21:40:21信息139764860892992]——参数----
[10/02/2018 21:40:21信息139764860892992]层数：18
[10/02/2018 21:40:21信息139764860892992]数据类型：
[10/02/2018 21:40:21信息139764860892992]时代：2
[10/02/2018 21:40:21信息139764860892992]优化器：新加坡元
[10/02/2018 21:40:21信息139764860892992]动量：0.900000
[10/02/2018 21:40:21信息139764860892992]重量衰减：0.000100
[10/02/2018 21:40:21信息139764860892992]学习率：0.010000
[10/02/2018 21:40:21信息139764860892992]定义的lr_调度程序_步骤没有lr_调度程序_因子，将被忽略。。。
[10/02/2018 21:40:21信息139764860892992]小批量尺寸：512
[10/02/2018 21:40:21信息139764860892992]图像形状：3224224
[10/02/2018 21:40:21信息139764860892992]数量类别：257
[10/02/2018 21:40:21信息139764860892992]培训样本数量：15420
[10/02/2018 21:40:21信息139764860892992]增强类型：无
[10/02/2018 21:40:21信息139764860892992]千伏存储：设备
[10/02/2018 21:40:21信息139764860892992]检查点频率：2
[10/02/2018 21:40:21信息139764860892992]多重标签：0
[10/02/2018 21:40:21信息139764860892992]--------------------
[21:40:21]/opt/brazic pkg cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.2.x.288.0/RHEL5_64/generic flavor/src/src/nnvm/legacy_json_util.cc:209:加载以前版本v0.8.0保存的符号。正在尝试升级。。。
[21:40:21]/opt/brazic pkg cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.2.x.288.0/RHEL5_64/generic flavor/src/src/nnvm/legacy_json_util.cc:217:Symbol成功升级！
[10/02/2018 21:40:21信息139764860892992]设置线程数：63
[21:41:02]/opt/brazic pkg cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.2.x.288.0/RHEL5_64/generic flavor/src/src/kvstore//comm.h:634:56对GPU中只有32对启用了直接访问。这可能会影响性能。您可以将MXNET\u ENABLE\u\u GPU\P2P=0设置为关闭
[21:41:02]/opt/brazic pkg cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.2.x.288.0/RHEL5_64/generic flavor/src/src/kvstore//comm.h:643:.vvv。。。
[21:41:02]/opt/brazic pkg cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.2.x.288.0/RHEL5_64/generic flavor/src/src/kvstore//comm.h:643:v.vv。。
[21:41:02]/opt/brazic pkg cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.2.x.288.0/RHEL5_64/generic flavor/src/src/kvstore//comm.h:643:vv.v..v。
[21:41:02]/opt/brazic pkg cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.2.x.288.0/RHEL5_64/generic flavor/src/src/kvstore//comm.h:643:vvv….v
[21:41:02]/opt/brazic pkg cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.2.x.288.0/RHEL5_64/generic flavor/src/src/kvstore//comm.h:643:v….vvv
[21:41:02]/opt/brazic pkg cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.2.x.288.0/RHEL5_64/generic flavor/src/src/kvstore//comm.h:643:.v..v.vv
[21:41:02]/opt/brazic pkg cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.2.x.288.0/RHEL5_64/generic flavor/src/src/kvstore//comm.h:643:…v.vv.v
[21:41:02]/opt/brazic pkg cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.2.x.288.0/RHEL5_64/generic flavor/src/src/kvstore//comm.h:643:…vvv。
[21:41:03]/opt/brazic pkg cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.2.x.288.0/RHEL5_64/generic flavor/src/src/operator/nn//cudnn//cudnn_algoreg-inl.h:107：运行性能测试以找到最佳卷积算法，这可能需要一段时间…（将环境变量MXNET_cudnn_AUTOTUNE_AUTOTUNE_默认设置为0以禁用）
[10/02/2018 21:41:18信息139764860892992]历代[0]批次[20]#011速度：903.34样本/秒#011准确度=0.020554
[10/02/2018 21:41:23信息139764860892992]历元[0]列车精度=0.055990
[10/02/2018 21:41:23信息139764860892992]时代[0]时间成本=21.168
[10/02/2018 21:41:30信息139764860892992]历元[0]验证精度=0.257747
[10/02/2018 21:41:4