Tensorflow 将SessionRunHook与TPU一起使用

Tensorflow 将SessionRunHook与TPU一起使用,tensorflow,tpu,Tensorflow,Tpu,我已经开发了一个SessionRunHook,它连接到TPUEstimator。SessionRunHook在CPU上工作得非常好,但是如果我使用TPU,我会得到一个错误: INFO:tensorflow:Error recorded from outfeed: Attempted to use a closed Session. INFO:tensorflow:Error recorded from evaluation_loop: Operation 'mean_1' has been ma

我已经开发了一个SessionRunHook,它连接到TPUEstimator。SessionRunHook在CPU上工作得非常好,但是如果我使用TPU,我会得到一个错误:

INFO:tensorflow:Error recorded from outfeed: Attempted to use a closed Session.
INFO:tensorflow:Error recorded from evaluation_loop: Operation 'mean_1' has been marked as not fetchable.
INFO:tensorflow:evaluation_loop marked as finished
WARNING:tensorflow:Reraising captured error
Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/donatas_repecka/workspace/bert/run_pretraining.py", line 549, in <module>
    tf.app.run()
INFO:tensorflow:Error recorded from infeed: Step was cancelled by an explicit call to `Session::Close()`.
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "/home/donatas_repecka/workspace/bert/run_pretraining.py", line 505, in main
    result = estimator.evaluate(input_fn=input_fn, steps=FLAGS.max_eval_steps)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py", line 2424, in evaluate
    rendezvous.raise_errors()
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/tpu/python/tpu/error_handling.py", line 128, in raise_errors
    six.reraise(typ, value, traceback)
  File "/usr/local/lib/python3.5/dist-packages/six.py", line 693, in reraise
    raise value
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/tpu/python/tpu/error_handling.py", line 101, in catch_errors
    yield
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py", line 451, in _run_outfeed
    session.run(self._dequeue_ops)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1075, in _run
    raise RuntimeError('Attempted to use a closed Session.')
RuntimeError: Attempted to use a closed Session.
INFO:tensorflow:从出料口记录的错误:试图使用关闭的会话。
信息:tensorflow:评估循环记录的错误:操作“mean_1”已标记为不可获取。
信息:tensorflow:评估循环标记为已完成
警告:tensorflow:重新释放捕获的错误
回溯(最近一次呼叫最后一次):
文件“/usr/lib/python3.5/runpy.py”,第193行,在“运行”模块中作为“主”
“\uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu
文件“/usr/lib/python3.5/runpy.py”,第85行,在运行代码中
exec(代码、运行\全局)
文件“/home/donatas_repecka/workspace/bert/run_pretraining.py”,第549行,在
tf.app.run()
信息:tensorflow:从馈入记录的错误:步骤被对“Session::Close()”的显式调用取消。
文件“/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py”,第125行,运行中
_系统出口(主(argv))
文件“/home/donatas_repecka/workspace/bert/run_pretraining.py”,第505行,在main中
结果=估计器。评估(输入=输入,步骤=标志。最大评估步骤)
文件“/usr/local/lib/python3.5/dist packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py”,第2424行,在evaluate中
集合。引发错误()
文件“/usr/local/lib/python3.5/dist packages/tensorflow/contrib/tpu/python/tpu/error\u handling.py”,raise\u errors中的第128行
六、重放(类型、值、回溯)
文件“/usr/local/lib/python3.5/dist-packages/six.py”,第693行,在reraise中
增值
文件“/usr/local/lib/python3.5/dist packages/tensorflow/contrib/tpu/python/tpu/error\u handling.py”,第101行,在catch\u errors中
产量
文件“/usr/local/lib/python3.5/dist packages/tensorflow/contrib/tpu/python/tpu/tpu\u estimator.py”,第451行,in\u run\u outfeed
session.run(self.\u dequeue\u ops)
文件“/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py”,第929行,正在运行
运行_元数据_ptr)
文件“/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py”,第1075行,正在运行
raise RUNTIMERROR('试图使用关闭的会话')
RuntimeError:试图使用已关闭的会话。

还有其他人遇到过这个问题并找到了解决方法吗?

你能找到解决方法吗?不幸的是,没有,我必须重构代码