Python GCFS写入超过配额并引发HTTP错误429_Python_Google Cloud Storage_Gcsfuse_Papermill

Python GCFS写入超过配额并引发HTTP错误429

python google-cloud-storage

Python GCFS写入超过配额并引发HTTP错误429,python,google-cloud-storage,gcsfuse,papermill,Python,Google Cloud Storage,Gcsfuse,Papermill,通过papermill[gcs] papermill gs://my-bucket/test.ipynb gs://my-bucket/output/test.ipynb 我收到错误消息：HTTP 429超出了速率限制如果输出笔记本是本地写入的，则工作正常： papermill gs://my-bucket/test.ipynb /tmp/test.ipynb 本地文件大小为：57K ls -alh /tmp/test.ipynb -rw-r--r-- 1 user wheel

通过

papermill[gcs]

papermill gs://my-bucket/test.ipynb gs://my-bucket/output/test.ipynb

我收到错误消息：HTTP 429超出了速率限制

如果输出笔记本是本地写入的，则工作正常：

papermill gs://my-bucket/test.ipynb /tmp/test.ipynb

本地文件大小为：57K

ls -alh /tmp/test.ipynb 
-rw-r--r--  1 user  wheel    57K Feb 14 10:37 /tmp/test.ipynb

造纸厂代码：

如何繁殖

pip install papermill[gcs]
papermill gs://cloud-samples-data/papermill/samples/test.ipynb gs://<your bucket>/test.ipynb

在MacOS环境中，我会遇到类似的错误：（添加了调试）

papermill gs://cloud samples data/papermill/samples/test.ipynb gs://dpe sandbox/test.ipynb
输入笔记本：gs://cloud samples data/papermill/samples/test.ipynb
输出笔记本：gs://dpe sandbox/test.ipynb
（‘dpe-cloud-mle’、‘完全控制’、无、无、无）
0%|                                                                                                   | 0/28 [00:00当您尝试在短时间内过于频繁或过快地创建或更新对象时，会发生此错误。如Google Cloud Storage中所述，单个特定对象每秒最多只能更新或覆盖一次。在这种情况下，我建议您修改更新或创建对象的方式。
sa的速率限制谷歌云存储的升级是在papermill
0.18.1
中添加的，所以这个问题不应该再由papermill直接引起了
pip install papermill[gcs]
papermill gs://cloud-samples-data/papermill/samples/test.ipynb gs://<your bucket>/test.ipynb

10
Ending Cell 6------------------------------------------
Exception gcsfs.utils.HtmlError: HtmlError(u'The total number of changes to the object dpe-sandbox/test.ipynb exceeds the rate limit. Please reduce the rate of create, update, and delete requests.',) in <bound method GCSFile.__del__ of <GCSFile d
pe-sandbox/test.ipynb>> ignored
Traceback (most recent call last):
  File "/usr/local/bin/papermill", line 11, in <module>
    sys.exit(papermill())
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/papermill/cli.py", line 165, in papermill
    cwd=cwd,
  File "/usr/local/lib/python2.7/dist-packages/papermill/execute.py", line 90, in execute_notebook
    start_timeout=start_timeout,
  File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 56, in execute_notebook_with_engine
    return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 296, in execute_notebook
    nb = cls.execute_managed_notebook(nb_man, kernel_name, log_output=log_output, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 352, in execute_managed_notebook
    preprocessor.preprocess(nb_man, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/papermill/preprocess.py", line 27, in preprocess
    nb, resources = self.papermill_process(nb_man, resources)
  File "/usr/local/lib/python2.7/dist-packages/papermill/preprocess.py", line 81, in papermill_process
    nb_man.cell_complete(nb.cells[index])
  File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 76, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 219, in cell_complete
    self.save()
  File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 76, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 138, in save
    write_ipynb(self.nb, self.output_path)
  File "/usr/local/lib/python2.7/dist-packages/papermill/iorw.py", line 280, in write_ipynb
    papermill_io.write(nbformat.writes(nb), path)
  File "/usr/local/lib/python2.7/dist-packages/papermill/iorw.py", line 82, in write
    return self.get_handler(path).write(buf, path)
  File "/usr/local/lib/python2.7/dist-packages/papermill/iorw.py", line 251, in write
    return f.write(buf)
  File "</usr/local/lib/python2.7/dist-packages/decorator.pyc:decorator-gen-150>", line 2, in close
  File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 1548, in close
    self.flush(force=True)
  File "</usr/local/lib/python2.7/dist-packages/decorator.pyc:decorator-gen-145>", line 2, in flush
  File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 1367, in flush
    self._simple_upload()
  File "</usr/local/lib/python2.7/dist-packages/decorator.pyc:decorator-gen-148>", line 2, in _simple_upload
  File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 1465, in _simple_upload
    validate_response(r, path)
  File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 162, in validate_response
    raise HtmlError(error)
gcsfs.utils.HtmlError: The total number of changes to the object dpe-sandbox/test.ipynb exceeds the rate limit. Please reduce the rate of create, update, and delete requests.
+ err 'Processing notebook failed'
++ date +%Y-%m-%dT%H:%M:%S%z
+ echo '[2019-02-14T18:32:58+0000]: Processing notebook failed'
[2019-02-14T18:32:58+0000]: Processing notebook failed
+ exit 1


export GOOGLE_APPLICATION_CREDENTIALS=/keys/my-project.json
gcloud config set account XXXXXXX-compute@developer.gserviceaccount.com
gcloud auth activate-service-account --key-file=/keys/my-project.json

papermill gs://cloud-samples-data/papermill/samples/test.ipynb gs://dpe-sandbox/test.ipynb
Input Notebook:  gs://cloud-samples-data/papermill/samples/test.ipynb
Output Notebook: gs://dpe-sandbox/test.ipynb
('dpe-cloud-mle', 'full_control', None, None, 'none', None)
  0%|                                                                                                   | 0/28 [00:00<?, ?it/s]uploading
11317
uploading
11342
uploading
11372
  4%|███▎                                                                                       | 1/28 [00:01<00:50,  1.88s/it]uploading
11397
uploading
11427
  7%|██████▌                                                                                    | 2/28 [00:02<00:41,  1.59s/it]uploading
11452
uploading
11482
 11%|█████████▊                                                                                 | 3/28 [00:03<00:33,  1.32s/it]uploading
11507
uploading
11537
 14%|█████████████                                                                              | 4/28 [00:04<00:26,  1.10s/it]uploading
11562
429

429
Traceback (most recent call last):
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 296, in execute_notebook
    nb = cls.execute_managed_notebook(nb_man, kernel_name, log_output=log_output, **kwargs)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 352, in execute_managed_notebook
    preprocessor.preprocess(nb_man, kwargs)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/preprocess.py", line 27, in preprocess
    nb, resources = self.papermill_process(nb_man, resources)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/preprocess.py", line 81, in papermill_process
    nb_man.cell_complete(nb.cells[index])
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 76, in wrapper
    return func(self, *args, **kwargs)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 219, in cell_complete
    self.save()
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 76, in wrapper
    return func(self, *args, **kwargs)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 138, in save
    write_ipynb(self.nb, self.output_path)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/iorw.py", line 280, in write_ipynb
    papermill_io.write(nbformat.writes(nb), path)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/iorw.py", line 82, in write
    return self.get_handler(path).write(buf, path)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/iorw.py", line 251, in write
    return f.write(buf)
  File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-152>", line 2, in close
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1552, in close
    self.flush(force=True)
  File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-147>", line 2, in flush
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1369, in flush
    self._simple_upload()
  File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-150>", line 2, in _simple_upload
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1467, in _simple_upload
    validate_response(r, path)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 163, in validate_response
    raise HtmlError(error)
gcsfs.utils.HtmlError: The total number of changes to the object dpe-sandbox/test.ipynb exceeds the rate limit. Please reduce the rate of create, update, and delete requests.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/user/Documents/Development/dpe/venv/papermill/bin/papermill", line 10, in <module>
    sys.exit(papermill())
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/cli.py", line 165, in papermill
    cwd=cwd,
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/execute.py", line 90, in execute_notebook
    start_timeout=start_timeout,
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 56, in execute_notebook_with_engine
    return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 302, in execute_notebook
    nb_man.notebook_complete()
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 76, in wrapper
    return func(self, *args, **kwargs)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 249, in notebook_complete
    self.save()
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 76, in wrapper
    return func(self, *args, **kwargs)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 138, in save
    write_ipynb(self.nb, self.output_path)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/iorw.py", line 280, in write_ipynb
    papermill_io.write(nbformat.writes(nb), path)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/iorw.py", line 82, in write
    return self.get_handler(path).write(buf, path)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/iorw.py", line 251, in write
    return f.write(buf)
  File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-152>", line 2, in close
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1552, in close
    self.flush(force=True)
  File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-147>", line 2, in flush
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1369, in flush
    self._simple_upload()
  File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-150>", line 2, in _simple_upload
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1467, in _simple_upload
    validate_response(r, path)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 163, in validate_response
    raise HtmlError(error)
gcsfs.utils.HtmlError: The total number of changes to the object dpe-sandbox/test.ipynb exceeds the rate limit. Please reduce the rate of create, update, and delete requests.
429
429
Exception ignored in: <bound method GCSFile.__del__ of <GCSFile dpe-sandbox/test.ipynb>>
Traceback (most recent call last):
  File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-153>", line 2, in __del__
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1575, in __del__
    self.close()
  File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-152>", line 2, in close
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1552, in close
    self.flush(force=True)
  File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-147>", line 2, in flush
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1369, in flush
    self._simple_upload()
  File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-150>", line 2, in _simple_upload
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1467, in _simple_upload
    validate_response(r, path)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 163, in validate_response
    raise HtmlError(error)
gcsfs.utils.HtmlError: The total number of changes to the object dpe-sandbox/test.ipynb exceeds the rate limit. Please reduce the rate of create, update, and delete requests.
429
429
Exception ignored in: <bound method GCSFile.__del__ of <GCSFile dpe-sandbox/test.ipynb>>
Traceback (most recent call last):
  File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-153>", line 2, in __del__
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1575, in __del__
    self.close()
  File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-152>", line 2, in close
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1552, in close
    self.flush(force=True)
  File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-147>", line 2, in flush
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1369, in flush
    self._simple_upload()
  File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-150>", line 2, in _simple_upload
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1467, in _simple_upload
    validate_response(r, path)
  File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 163, in validate_response
    raise HtmlError(error)
gcsfs.utils.HtmlError: The total number of changes to the object dpe-sandbox/test.ipynb exceeds the rate limit. Please reduce the rate of create, update, and delete requests.