Amazon web services 如何保存S3中的模型和Sagemaker中的检查点
我正在使用带有Tensorflow版本1.12.0和conda_tensorflor_p36内核的Sagemaker。我能够在部署的模型中成功地进行培训、评估验证数据和预测。我正在用训练脚本进行训练 我希望能够在S3中保存模型，然后加载它。它假定模型将自动保存，但不会发生 我正在使用以下命令。 在脚本中：Amazon web services 如何保存S3中的模型和Sagemaker中的检查点,amazon-web-services,amazon-s3,save,amazon-sagemaker,checkpoint,Amazon Web Services,Amazon S3,Save,Amazon Sagemaker,Checkpoint,我正在使用带有Tensorflow版本1.12.0和conda_tensorflor_p36内核的Sagemaker。我能够在部署的模型中成功地进行培训、评估验证数据和预测。我正在用训练脚本进行训练 我希望能够在S3中保存模型，然后加载它。它假定模型将自动保存，但不会发生 我正在使用以下命令。 在脚本中： parser.add_argument('--epochs', type=int, default=10) parser.add_argument('--learning-ra
parser.add_argument('--epochs', type=int, default=10)
parser.add_argument('--learning-rate', type=float, default=0.001)
parser.add_argument('--batch-size', type=int, default=32)
parser.add_argument('--gpu-count', type=int, default=os.environ['SM_NUM_GPUS'])
parser.add_argument('--model-dir', type=str, default=os.environ['SM_MODEL_DIR'])
parser.add_argument('--training', type=str, default=os.environ['SM_CHANNEL_TRAINING'])
parser.add_argument('--validation', type=str, default=os.environ['SM_CHANNEL_VALIDATION'])
args, _ = parser.parse_known_args()
epochs = args.epochs
lr = args.learning_rate
batch_size = args.batch_size
gpu_count = args.gpu_count
model_dir = args.model_dir
training_dir = args.training
validation_dir = args.validation
...
# save Keras model for Tensorflow Serving
sess = K.get_session()
tensorflow.saved_model.simple_save(
sess,
os.path.join(model_dir, 'model/1'),
inputs={'inputs': model.input},
outputs={t.name: t for t in model.outputs})
model_path = 's3://{}/{}'.format(bucket, model_channel)
tf_estimator = TensorFlow(entry_point='train_cnn.py',
role=role,
train_instance_count=1,
train_instance_type='local', #We use the local instance
framework_version='1.12',
py_version='py3',
script_mode=True,
hyperparameters={'epochs': 1}, #One epoch just to check everything is ok
model_dir=model_path
)
tf_estimator.fit({'training': training_input_path, 'validation': validation_input_path})
Creating tmp9hs8n9sq_algo-1-xp4eu_1 ...
Attaching to tmp9hs8n9sq_algo-1-xp4eu_12mdone
algo-1-xp4eu_1 | 2020-07-17 21:40:18,234 sagemaker-containers INFO Imported framework sagemaker_tensorflow_container.training
algo-1-xp4eu_1 | 2020-07-17 21:40:18,240 sagemaker-containers INFO No GPUs detected (normal if no gpus installed)
algo-1-xp4eu_1 | 2020-07-17 21:40:18,523 sagemaker-containers INFO No GPUs detected (normal if no gpus installed)
algo-1-xp4eu_1 | 2020-07-17 21:40:18,541 sagemaker-containers INFO No GPUs detected (normal if no gpus installed)
algo-1-xp4eu_1 | 2020-07-17 21:40:18,555 sagemaker-containers INFO Invoking user script
algo-1-xp4eu_1 |
algo-1-xp4eu_1 | Training Env:
algo-1-xp4eu_1 |
algo-1-xp4eu_1 | {
algo-1-xp4eu_1 | "additional_framework_parameters": {},
algo-1-xp4eu_1 | "channel_input_dirs": {
algo-1-xp4eu_1 | "training": "/opt/ml/input/data/training",
algo-1-xp4eu_1 | "validation": "/opt/ml/input/data/validation"
algo-1-xp4eu_1 | },
algo-1-xp4eu_1 | "current_host": "algo-1-xp4eu",
algo-1-xp4eu_1 | "framework_module": "sagemaker_tensorflow_container.training:main",
algo-1-xp4eu_1 | "hosts": [
algo-1-xp4eu_1 | "algo-1-xp4eu"
algo-1-xp4eu_1 | ],
algo-1-xp4eu_1 | "hyperparameters": {
algo-1-xp4eu_1 | "epochs": 1,
algo-1-xp4eu_1 | "model_dir": "s3://mybucket/myprefix/models"
algo-1-xp4eu_1 | },
algo-1-xp4eu_1 | "input_config_dir": "/opt/ml/input/config",
algo-1-xp4eu_1 | "input_data_config": {
algo-1-xp4eu_1 | "training": {
algo-1-xp4eu_1 | "TrainingInputMode": "File"
algo-1-xp4eu_1 | },
algo-1-xp4eu_1 | "validation": {
algo-1-xp4eu_1 | "TrainingInputMode": "File"
algo-1-xp4eu_1 | }
algo-1-xp4eu_1 | },
algo-1-xp4eu_1 | "input_dir": "/opt/ml/input",
algo-1-xp4eu_1 | "is_master": true,
algo-1-xp4eu_1 | "job_name": "sagemaker-tensorflow-scriptmode-2020-07-17-21-40-14-146",
algo-1-xp4eu_1 | "log_level": 20,
algo-1-xp4eu_1 | "master_hostname": "algo-1-xp4eu",
algo-1-xp4eu_1 | "model_dir": "/opt/ml/model",
algo-1-xp4eu_1 | "module_dir": "s3://sagemaker-us-east-1-487764570858/sagemaker-tensorflow-scriptmode-2020-07-17-21-40-14-146/source/sourcedir.tar.gz",
algo-1-xp4eu_1 | "module_name": "train_cnn",
algo-1-xp4eu_1 | "network_interface_name": "eth0",
algo-1-xp4eu_1 | "num_cpus": 4,
algo-1-xp4eu_1 | "num_gpus": 0,
algo-1-xp4eu_1 | "output_data_dir": "/opt/ml/output/data",
algo-1-xp4eu_1 | "output_dir": "/opt/ml/output",
algo-1-xp4eu_1 | "output_intermediate_dir": "/opt/ml/output/intermediate",
algo-1-xp4eu_1 | "resource_config": {
algo-1-xp4eu_1 | "current_host": "algo-1-xp4eu",
algo-1-xp4eu_1 | "hosts": [
algo-1-xp4eu_1 | "algo-1-xp4eu"
algo-1-xp4eu_1 | ]
algo-1-xp4eu_1 | },
algo-1-xp4eu_1 | "user_entry_point": "train_cnn.py"
algo-1-xp4eu_1 | }
algo-1-xp4eu_1 |
algo-1-xp4eu_1 | Environment variables:
algo-1-xp4eu_1 |
algo-1-xp4eu_1 | SM_HOSTS=["algo-1-xp4eu"]
algo-1-xp4eu_1 | SM_NETWORK_INTERFACE_NAME=eth0
algo-1-xp4eu_1 | SM_HPS={"epochs":1,"model_dir":"s3://mybucket/myprefix/models"}
algo-1-xp4eu_1 | SM_USER_ENTRY_POINT=train_cnn.py
algo-1-xp4eu_1 | SM_FRAMEWORK_PARAMS={}
algo-1-xp4eu_1 | SM_RESOURCE_CONFIG={"current_host":"algo-1-xp4eu","hosts":["algo-1-xp4eu"]}
algo-1-xp4eu_1 | SM_INPUT_DATA_CONFIG={"training":{"TrainingInputMode":"File"},"validation":{"TrainingInputMode":"File"}}
algo-1-xp4eu_1 | SM_OUTPUT_DATA_DIR=/opt/ml/output/data
algo-1-xp4eu_1 | SM_CHANNELS=["training","validation"]
algo-1-xp4eu_1 | SM_CURRENT_HOST=algo-1-xp4eu
algo-1-xp4eu_1 | SM_MODULE_NAME=train_cnn
algo-1-xp4eu_1 | SM_LOG_LEVEL=20
algo-1-xp4eu_1 | SM_FRAMEWORK_MODULE=sagemaker_tensorflow_container.training:main
algo-1-xp4eu_1 | SM_INPUT_DIR=/opt/ml/input
algo-1-xp4eu_1 | SM_INPUT_CONFIG_DIR=/opt/ml/input/config
algo-1-xp4eu_1 | SM_OUTPUT_DIR=/opt/ml/output
algo-1-xp4eu_1 | SM_NUM_CPUS=4
algo-1-xp4eu_1 | SM_NUM_GPUS=0
algo-1-xp4eu_1 | SM_MODEL_DIR=/opt/ml/model
algo-1-xp4eu_1 | SM_MODULE_DIR=s3://sagemaker-us-east-1-487764570858/sagemaker-tensorflow-scriptmode-2020-07-17-21-40-14-146/source/sourcedir.tar.gz
algo-1-xp4eu_1 | SM_TRAINING_ENV={"additional_framework_parameters":{},"channel_input_dirs":{"training":"/opt/ml/input/data/training","validation":"/opt/ml/input/data/validation"},"current_host":"algo-1-xp4eu","framework_module":"sagemaker_tensorflow_container.training:main","hosts":["algo-1-xp4eu"],"hyperparameters":{"epochs":1,"model_dir":"s3://mybucket/myprefix/models"},"input_config_dir":"/opt/ml/input/config","input_data_config":{"training":{"TrainingInputMode":"File"},"validation":{"TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","is_master":true,"job_name":"sagemaker-tensorflow-scriptmode-2020-07-17-21-40-14-146","log_level":20,"master_hostname":"algo-1-xp4eu","model_dir":"/opt/ml/model","module_dir":"s3://sagemaker-us-east-1-487764570858/sagemaker-tensorflow-scriptmode-2020-07-17-21-40-14-146/source/sourcedir.tar.gz","module_name":"train_cnn","network_interface_name":"eth0","num_cpus":4,"num_gpus":0,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1-xp4eu","hosts":["algo-1-xp4eu"]},"user_entry_point":"train_cnn.py"}
algo-1-xp4eu_1 | SM_USER_ARGS=["--epochs","1","--model_dir","s3://mybucket/myprefix/models"]
algo-1-xp4eu_1 | SM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate
algo-1-xp4eu_1 | SM_CHANNEL_TRAINING=/opt/ml/input/data/training
algo-1-xp4eu_1 | SM_CHANNEL_VALIDATION=/opt/ml/input/data/validation
algo-1-xp4eu_1 | SM_HP_EPOCHS=1
algo-1-xp4eu_1 | SM_HP_MODEL_DIR=s3://mybucket/myprefix/models
algo-1-xp4eu_1 | PYTHONPATH=/opt/ml/code:/usr/local/bin:/usr/lib/python36.zip:/usr/lib/python3.6:/usr/lib/python3.6/lib-dynload:/usr/local/lib/python3.6/dist-packages:/usr/lib/python3/dist-packages
algo-1-xp4eu_1 |
algo-1-xp4eu_1 | Invoking script with the following command:
algo-1-xp4eu_1 |
algo-1-xp4eu_1 | /usr/bin/python train_cnn.py --epochs 1 --model_dir s3://mybucket/myprefix/models
algo-1-xp4eu_1 |
algo-1-xp4eu_1 |
algo-1-xp4eu_1 | Collecting matplotlib
algo-1-xp4eu_1 | Downloading https://files.pythonhosted.org/packages/93/ae/81b1c98ae97350711adb021ee12ea678b37f608ec2faa35c3a7db11795fa/matplotlib-3.3.0-1-cp36-cp36m-manylinux1_x86_64.whl (11.5MB)
100% |████████████████████████████████| 11.5MB 4.3MB/s eta 0:00:01
algo-1-xp4eu_1 | Collecting kiwisolver>=1.0.1 (from matplotlib)
algo-1-xp4eu_1 | Downloading https://files.pythonhosted.org/packages/ae/23/147de658aabbf968324551ea22c0c13a00284c4ef49a77002e91f79657b7/kiwisolver-1.2.0-cp36-cp36m-manylinux1_x86_64.whl (88kB)
100% |████████████████████████████████| 92kB 29.2MB/s ta 0:00:01
algo-1-xp4eu_1 | Requirement already satisfied: numpy>=1.15 in /usr/local/lib/python3.6/dist-packages (from matplotlib) (1.16.3)
algo-1-xp4eu_1 | Collecting cycler>=0.10 (from matplotlib)
algo-1-xp4eu_1 | Downloading https://files.pythonhosted.org/packages/f7/d2/e07d3ebb2bd7af696440ce7e754c59dd546ffe1bbe732c8ab68b9c834e61/cycler-0.10.0-py2.py3-none-any.whl
algo-1-xp4eu_1 | Collecting pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 (from matplotlib)
algo-1-xp4eu_1 | Downloading https://files.pythonhosted.org/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/pyparsing-2.4.7-py2.py3-none-any.whl (67kB)
100% |████████████████████████████████| 71kB 25.2MB/s ta 0:00:01
algo-1-xp4eu_1 | Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib) (2.8.0)
algo-1-xp4eu_1 | Collecting pillow>=6.2.0 (from matplotlib)
algo-1-xp4eu_1 | Downloading https://files.pythonhosted.org/packages/30/bf/92385b4262178ca22b34f82e0e09c2922eb351fe39f3cc7b8ba9ea555b41/Pillow-7.2.0-cp36-cp36m-manylinux1_x86_64.whl (2.2MB)
100% |████████████████████████████████| 2.2MB 15.3MB/s ta 0:00:01
algo-1-xp4eu_1 | Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from cycler>=0.10->matplotlib) (1.12.0)
algo-1-xp4eu_1 | Installing collected packages: kiwisolver, cycler, pyparsing, pillow, matplotlib
algo-1-xp4eu_1 | Found existing installation: Pillow 6.0.0
algo-1-xp4eu_1 | Uninstalling Pillow-6.0.0:
algo-1-xp4eu_1 | Successfully uninstalled Pillow-6.0.0
algo-1-xp4eu_1 | Successfully installed cycler-0.10.0 kiwisolver-1.2.0 matplotlib-3.3.0 pillow-7.2.0 pyparsing-2.4.7
algo-1-xp4eu_1 | You are using pip version 18.1, however version 20.2b1 is available.
algo-1-xp4eu_1 | You should consider upgrading via the 'pip install --upgrade pip' command.
algo-1-xp4eu_1 | _________________________________________________________________
algo-1-xp4eu_1 | Layer (type) Output Shape Param #
algo-1-xp4eu_1 | =================================================================
algo-1-xp4eu_1 | conv2d (Conv2D) (None, 64, 64, 32) 320
algo-1-xp4eu_1 | _________________________________________________________________
algo-1-xp4eu_1 | max_pooling2d (MaxPooling2D) (None, 32, 32, 32) 0
algo-1-xp4eu_1 | _________________________________________________________________
algo-1-xp4eu_1 | dropout (Dropout) (None, 32, 32, 32) 0
algo-1-xp4eu_1 | _________________________________________________________________
algo-1-xp4eu_1 | conv2d_1 (Conv2D) (None, 32, 32, 64) 18496
algo-1-xp4eu_1 | _________________________________________________________________
algo-1-xp4eu_1 | max_pooling2d_1 (MaxPooling2 (None, 16, 16, 64) 0
algo-1-xp4eu_1 | _________________________________________________________________
algo-1-xp4eu_1 | dropout_1 (Dropout) (None, 16, 16, 64) 0
algo-1-xp4eu_1 | _________________________________________________________________
algo-1-xp4eu_1 | conv2d_2 (Conv2D) (None, 16, 16, 128) 73856
algo-1-xp4eu_1 | _________________________________________________________________
algo-1-xp4eu_1 | max_pooling2d_2 (MaxPooling2 (None, 8, 8, 128) 0
algo-1-xp4eu_1 | _________________________________________________________________
algo-1-xp4eu_1 | dropout_2 (Dropout) (None, 8, 8, 128) 0
algo-1-xp4eu_1 | _________________________________________________________________
algo-1-xp4eu_1 | conv2d_3 (Conv2D) (None, 8, 8, 256) 295168
algo-1-xp4eu_1 | _________________________________________________________________
algo-1-xp4eu_1 | max_pooling2d_3 (MaxPooling2 (None, 4, 4, 256) 0
algo-1-xp4eu_1 | _________________________________________________________________
algo-1-xp4eu_1 | flatten (Flatten) (None, 4096) 0
algo-1-xp4eu_1 | _________________________________________________________________
algo-1-xp4eu_1 | dense (Dense) (None, 128) 524416
algo-1-xp4eu_1 | _________________________________________________________________
algo-1-xp4eu_1 | dropout_3 (Dropout) (None, 128) 0
algo-1-xp4eu_1 | _________________________________________________________________
algo-1-xp4eu_1 | dense_1 (Dense) (None, 1) 129
algo-1-xp4eu_1 | =================================================================
algo-1-xp4eu_1 | Total params: 912,385
algo-1-xp4eu_1 | Trainable params: 912,385
algo-1-xp4eu_1 | Non-trainable params: 0
algo-1-xp4eu_1 | _________________________________________________________________
algo-1-xp4eu_1 | None
algo-1-xp4eu_1 | Train on 26 samples, validate on 2 samples
algo-1-xp4eu_1 | Epoch 1/1
algo-1-xp4eu_1 | - 2s - loss: 1.1576 - acc: 0.5000 - val_loss: 2.5328 - val_acc: 0.5000
algo-1-xp4eu_1 | Validation loss : 2.5328116416931152
algo-1-xp4eu_1 | Validation accuracy: 0.5
algo-1-xp4eu_1 | Keys dict_keys(['val_loss', 'val_acc', 'loss', 'acc'])
algo-1-xp4eu_1 | WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/saved_model/simple_save.py:85: calling SavedModelBuilder.add_meta_graph_and_variables (from tensorflow.python.saved_model.builder_impl) with legacy_init_op is deprecated and will be removed in a future version.
algo-1-xp4eu_1 | Instructions for updating:
algo-1-xp4eu_1 | Pass your op to the equivalent parameter main_op instead.
algo-1-xp4eu_1 | 2020-07-17 21:40:28,943 sagemaker-containers INFO Reporting training SUCCESS
tmp9hs8n9sq_algo-1-xp4eu_1 exited with code 0
Aborting on container exit...
===== Job Complete =====