Keras 将gcloud任务提交到google cloud ML引擎时出错

我是谷歌ML云引擎的新手。我想将Keras模型发布到云端进行训练，但我总是会遇到以下错误：

I  master-replica-0 Running module trainer.bot.  master-replica-0
I  master-replica-0 Downloading the package: gs://zadravecm-bot/jobs/test_job4/packages/84f3c60920e885020405e1eb7afa5f509313d2a5406a1f1551a81b81993ac66c/trainer-1.0.tar.gz  master-replica-0
I  master-replica-0 Running command: gsutil -q cp gs://zadravecm-bot/jobs/test_job4/packages/84f3c60920e885020405e1eb7afa5f509313d2a5406a1f1551a81b81993ac66c/trainer-1.0.tar.gz trainer-1.0.tar.gz  master-replica-0
I  master-replica-0 Installing the package: gs://zadravecm-bot/jobs/test_job4/packages/84f3c60920e885020405e1eb7afa5f509313d2a5406a1f1551a81b81993ac66c/trainer-1.0.tar.gz  master-replica-0
I  master-replica-0 Running command: pip3 install --user --upgrade --force-reinstall --no-deps trainer-1.0.tar.gz  master-replica-0
I  master-replica-0 Processing ./trainer-1.0.tar.gz  master-replica-0
E  master-replica-0 Exception:  master-replica-0
E  master-replica-0 Traceback (most recent call last):  master-replica-0
E  master-replica-0   File "/usr/local/lib/python3.5/dist-packages/pip/_internal/basecommand.py", line 228, in main  master-replica-0
E  master-replica-0     status = self.run(options, args)  master-replica-0
E  master-replica-0   File "/usr/local/lib/python3.5/dist-packages/pip/_internal/commands/install.py", line 291, in run  master-replica-0
E  master-replica-0     resolver.resolve(requirement_set)  master-replica-0
E  master-replica-0   File "/usr/local/lib/python3.5/dist-packages/pip/_internal/resolve.py", line 103, in resolve  master-replica-0
E  master-replica-0     self._resolve_one(requirement_set, req)  master-replica-0
E  master-replica-0   File "/usr/local/lib/python3.5/dist-packages/pip/_internal/resolve.py", line 257, in _resolve_one  master-replica-0
E  master-replica-0     abstract_dist = self._get_abstract_dist_for(req_to_install)  master-replica-0
E  master-replica-0   File "/usr/local/lib/python3.5/dist-packages/pip/_internal/resolve.py", line 210, in _get_abstract_dist_for  master-replica-0
E  master-replica-0     self.require_hashes  master-replica-0
E  master-replica-0   File "/usr/local/lib/python3.5/dist-packages/pip/_internal/operations/prepare.py", line 310, in prepare_linked_requirement  master-replica-0
E  master-replica-0     progress_bar=self.progress_bar  master-replica-0
E  master-replica-0   File "/usr/local/lib/python3.5/dist-packages/pip/_internal/download.py", line 824, in unpack_url  master-replica-0
E  master-replica-0     unpack_file_url(link, location, download_dir, hashes=hashes)  master-replica-0
E  master-replica-0   File "/usr/local/lib/python3.5/dist-packages/pip/_internal/download.py", line 729, in unpack_file_url  master-replica-0
E  master-replica-0     unpack_file(from_path, location, content_type, link)  master-replica-0
E  master-replica-0   File "/usr/local/lib/python3.5/dist-packages/pip/_internal/utils/misc.py", line 578, in unpack_file  master-replica-0
E  master-replica-0     tarfile.is_tarfile(filename) or  master-replica-0
E  master-replica-0   File "/usr/lib/python3.5/tarfile.py", line 2448, in is_tarfile  master-replica-0
E  master-replica-0     t = open(name)  master-replica-0
E  master-replica-0   File "/usr/lib/python3.5/tarfile.py", line 1557, in open  master-replica-0
E  master-replica-0     return func(name, "r", fileobj, **kwargs)  master-replica-0
E  master-replica-0   File "/usr/lib/python3.5/tarfile.py", line 1629, in gzopen  master-replica-0
E  master-replica-0     t = cls.taropen(name, mode, fileobj, **kwargs)  master-replica-0
E  master-replica-0   File "/usr/lib/python3.5/tarfile.py", line 1605, in taropen  master-replica-0
E  master-replica-0     return cls(name, mode, fileobj, **kwargs)  master-replica-0
E  master-replica-0   File "/usr/lib/python3.5/tarfile.py", line 1470, in __init__  master-replica-0
E  master-replica-0     self.firstmember = self.next()  master-replica-0
E  master-replica-0   File "/usr/lib/python3.5/tarfile.py", line 2279, in next  master-replica-0
E  master-replica-0     tarinfo = self.tarinfo.fromtarfile(self)  master-replica-0
E  master-replica-0   File "/usr/lib/python3.5/tarfile.py", line 1082, in fromtarfile  master-replica-0
E  master-replica-0     buf = tarfile.fileobj.read(BLOCKSIZE)  master-replica-0
E  master-replica-0   File "/usr/lib/python3.5/gzip.py", line 274, in read  master-replica-0
E  master-replica-0     return self._buffer.read(size)  master-replica-0
E  master-replica-0   File "/usr/lib/python3.5/_compression.py", line 68, in readinto  master-replica-0
E  master-replica-0     data = self.read(len(byte_view))  master-replica-0
E  master-replica-0   File "/usr/lib/python3.5/gzip.py", line 469, in read  master-replica-0
E  master-replica-0     uncompress = self._decompressor.decompress(buf, size)  master-replica-0
E  master-replica-0 zlib.error: Error -3 while decompressing data: invalid distance too far back  master-replica-0
E  master-replica-0 You are using pip version 10.0.1, however version 18.0 is available.  master-replica-0
E  master-replica-0 You should consider upgrading via the 'pip install --upgrade pip' command.  master-replica-0
W  master-replica-0 Installation of package failed on try 1/2: Command '['pip3', 'install', '--user', '--upgrade', '--force-reinstall', '--no-deps', 'trainer-1.0.tar.gz']' returned non-zero exit status 2
Retrying ...  master-replica-0
I  master-replica-0 Running command: pip3 install --user --upgrade --force-reinstall --no-deps trainer-1.0.tar.gz  master-replica-0
I  master-replica-0 Processing ./trainer-1.0.tar.gz  master-replica-0
E  master-replica-0 Exception:  master-replica-0
E  master-replica-0 Traceback (most recent call last):  master-replica-0
E  master-replica-0   File "/usr/local/lib/python3.5/dist-packages/pip/_internal/basecommand.py", line 228, in main  master-replica-0
E  master-replica-0     status = self.run(options, args)  master-replica-0
E  master-replica-0   File "/usr/local/lib/python3.5/dist-packages/pip/_internal/commands/install.py", line 291, in run  master-replica-0
E  master-replica-0     resolver.resolve(requirement_set)  master-replica-0
E  master-replica-0   File "/usr/local/lib/python3.5/dist-packages/pip/_internal/resolve.py", line 103, in resolve  master-replica-0
E  master-replica-0     self._resolve_one(requirement_set, req)  master-replica-0
E  master-replica-0   File "/usr/local/lib/python3.5/dist-packages/pip/_internal/resolve.py", line 257, in _resolve_one  master-replica-0
E  master-replica-0     abstract_dist = self._get_abstract_dist_for(req_to_install)  master-replica-0
E  master-replica-0   File "/usr/local/lib/python3.5/dist-packages/pip/_internal/resolve.py", line 210, in _get_abstract_dist_for  master-replica-0
E  master-replica-0     self.require_hashes  master-replica-0
E  master-replica-0   File "/usr/local/lib/python3.5/dist-packages/pip/_internal/operations/prepare.py", line 310, in prepare_linked_requirement  master-replica-0
E  master-replica-0     progress_bar=self.progress_bar  master-replica-0
E  master-replica-0   File "/usr/local/lib/python3.5/dist-packages/pip/_internal/download.py", line 824, in unpack_url  master-replica-0
E  master-replica-0     unpack_file_url(link, location, download_dir, hashes=hashes)  master-replica-0
E  master-replica-0   File "/usr/local/lib/python3.5/dist-packages/pip/_internal/download.py", line 729, in unpack_file_url  master-replica-0
E  master-replica-0     unpack_file(from_path, location, content_type, link)  master-replica-0
E  master-replica-0   File "/usr/local/lib/python3.5/dist-packages/pip/_internal/utils/misc.py", line 578, in unpack_file  master-replica-0
E  master-replica-0     tarfile.is_tarfile(filename) or  master-replica-0
E  master-replica-0   File "/usr/lib/python3.5/tarfile.py", line 2448, in is_tarfile  master-replica-0
E  master-replica-0     t = open(name)  master-replica-0
E  master-replica-0   File "/usr/lib/python3.5/tarfile.py", line 1557, in open  master-replica-0
E  master-replica-0     return func(name, "r", fileobj, **kwargs)  master-replica-0
E  master-replica-0   File "/usr/lib/python3.5/tarfile.py", line 1629, in gzopen  master-replica-0
E  master-replica-0     t = cls.taropen(name, mode, fileobj, **kwargs)  master-replica-0
E  master-replica-0   File "/usr/lib/python3.5/tarfile.py", line 1605, in taropen  master-replica-0
E  master-replica-0     return cls(name, mode, fileobj, **kwargs)  master-replica-0
E  master-replica-0   File "/usr/lib/python3.5/tarfile.py", line 1470, in __init__  master-replica-0
E  master-replica-0     self.firstmember = self.next()  master-replica-0
E  master-replica-0   File "/usr/lib/python3.5/tarfile.py", line 2279, in next  master-replica-0
E  master-replica-0     tarinfo = self.tarinfo.fromtarfile(self)  master-replica-0
E  master-replica-0   File "/usr/lib/python3.5/tarfile.py", line 1082, in fromtarfile  master-replica-0
E  master-replica-0     buf = tarfile.fileobj.read(BLOCKSIZE)  master-replica-0
E  master-replica-0   File "/usr/lib/python3.5/gzip.py", line 274, in read  master-replica-0
E  master-replica-0     return self._buffer.read(size)  master-replica-0
E  master-replica-0   File "/usr/lib/python3.5/_compression.py", line 68, in readinto  master-replica-0
E  master-replica-0     data = self.read(len(byte_view))  master-replica-0
E  master-replica-0   File "/usr/lib/python3.5/gzip.py", line 469, in read  master-replica-0
E  master-replica-0     uncompress = self._decompressor.decompress(buf, size)  master-replica-0
E  master-replica-0 zlib.error: Error -3 while decompressing data: invalid distance too far back  master-replica-0
E  master-replica-0 You are using pip version 10.0.1, however version 18.0 is available.  master-replica-0
E  master-replica-0 You should consider upgrading via the 'pip install --upgrade pip' command.  master-replica-0
E  master-replica-0 Command '['pip3', 'install', '--user', '--upgrade', '--force-reinstall', '--no-deps', 'trainer-1.0.tar.gz']' returned non-zero exit status 
2  master-replica-0
I  master-replica-0 Module completed; cleaning up.  master-replica-0
I  master-replica-0 Clean up finished.  master-replica-0
E  The replica master 0 exited with a non-zero status of 2. 
I  Job failed. 

我的终端脚本是：

export JOB_NAME="test_job4"
export BUCKET_NAME="zadravecm-bot"
export CLOUD_CONFIG=trainer/cloudml-gpu.yaml
export JOB_DIR=gs://zadravecm-bot/jobs/$JOB_NAME
export MODULE=trainer.bot
export PACKAGE_PATH=./trainer
export REGION=us-central1
export RUNTIME=1.8

gcloud ml-engine jobs submit training $JOB_NAME \
    --job-dir $JOB_DIR \
    --runtime-version $RUNTIME \
    --module-name $MODULE \
    --package-path $PACKAGE_PATH \
    --region $REGION \
    --config $CLOUD_CONFIG

以及GPU配置

trainingInput:
  scaleTier: BASIC_GPU
  runtimeVersion: "1.8"
  pythonVersion: "3.5"

我的应用程序层次结构是：

Bot
|
|---> trainer
    |
    | ---> __init__.py
    | ---> bot.py
    | ---> cloudml-gpu.yaml
|
|---> setup.py

---

通过查看stacktrace，您是否尝试运行

pip安装--升级pip

并再次执行？否，我不知道如何使用gcloud实现这一点……我的第一个想法是在云Shell中运行

pip安装--升级pip

，因为错误似乎与

trainer-1.0.tar.gz

安装和

pip

版本有关。我不确定本地更改是否会影响提交的作业，但您可以尝试一下，看看它是否解决了问题。另一方面，您可以查看Stackdriver日志（如果激活）或本地日志，以搜索有关错误的相关信息。@MarkoZadravec。这真是奇怪。能否将项目和作业id发送到cloudml-feedback@google.com所以我们可以调查？