Validation 验证数据集迭代器重置三次后的OOM问题

我最近将一组tensorflow代码从队列更改为数据集，以利用迭代器重置函数来验证数据集。它工作良好，除了在第三次验证的中间总是OOM问题。 （验证有700多个步骤，在第三次验证中，代码总是在400多个步骤左右崩溃）

重置迭代器并开始验证步骤

其他资料： Tensorflow 1.3.0 张力层1.6.5 cuda 8 cudnn 6

*val_step = 434, step_cost = 62.8875665665,  total_cost = 24353.1902707
2017-10-27 11:47:08.602743: W tensorflow/core/common_runtime/bfc_allocator.cc:273] Allocator (GPU_0_bfc) ran out of   memory trying to allocate 7.50MiB.  Current allocation summary follows.
2017-10-27 11:47:08.605489: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (256):   Total Chunks: 0,      Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in    use in bin.
2017-10-27 11:47:08.605520: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (512):   Total Chunks: 0,      Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in    use in bin.
2017-10-27 11:47:08.605536: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1024):  Total Chunks: 0,      Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in    use in bin.
....
....
....
2017-10-27 11:47:08.749632: I tensorflow/core/common_runtime/bfc_allocator.cc:700] Sum Total of in-use chunks: 14.    83GiB
2017-10-27 11:47:08.749646: I tensorflow/core/common_runtime/bfc_allocator.cc:702] Stats:
Limit:                 15927646618
InUse:                 15927646464
MaxInUse:              15927646464
NumAllocs:               457706168
MaxAllocSize:             42035200

2017-10-27 11:47:08.750506: W tensorflow/core/common_runtime/bfc_allocator.cc:                                        277] ****************************************************************************************************
2017-10-27 11:47:08.750540: W tensorflow/core/framework/op_kernel.cc:1192] Resource exhausted: OOM when allocating    tensor with shape[128,60,256]
Traceback (most recent call last):
  File "lstm_cls_att_emb.py", line 1091, in <module>
    tf.app.run()
  File "/data/anaconda2/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "lstm_cls_att_emb.py", line 1087, in main
    training()
  File "lstm_cls_att_emb.py", line 746, in training
    batch_check = FLAGS.val_batch_check)
  File "lstm_cls_att_emb.py", line 411, in process_chunk
    feed_dict=feed_dict)
  File "/data/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 895, in run
    run_metadata_ptr)
  File "/data/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1124, in _run
    feed_dict_tensor, options, run_metadata)
  File "/data/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1321, in _do_run
    options, run_metadata)
  File "/data/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1340, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[128,60,256]
   [[Node: model_att_1/attention/strided_slice_199 = StridedSlice[Index=DT_INT32, T=DT_FLOAT, begin_mask=5,           ellipsis_mask=0, end_mask=5, new_axis_mask=0, shrink_axis_mask=0, _device="/job:localhost/replica:0/task:0/gpu:       0"](model_att_1/attention/concat, model_att_1/attention/strided_slice_199/stack, model_att_1/attention/               strided_slice_199/stack_1, model_att_1/attention/strided_slice_199/stack_2)]]
   [[Node: div_1/_69195 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0",         send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_11676_div_1",       tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

Caused by op u'model_att_1/attention/strided_slice_199', defined at:
  File "lstm_cls_att_emb.py", line 1091, in <module>
    tf.app.run()
  File "/data/anaconda2/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "lstm_cls_att_emb.py", line 1087, in main
    training()
  File "lstm_cls_att_emb.py", line 594, in training
    scope=model_scope)
  File "lstm_cls_att_emb.py", line 320, in build_model
    name="attention")
  File "/data/wzhan/workspace/new_fraud_models/attention_emb.py", line 223, in __init__
    memory = self.memory[:, time_step:time_step+attention_window_size, :]  # batch_size * window_size * dim
  File "/data/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 509, in _SliceHelper
    name=name)
  File "/data/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 677, in strided_slice
    shrink_axis_mask=shrink_axis_mask)
  File "/data/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 3744, in            strided_slice
    shrink_axis_mask=shrink_axis_mask, name=name)
  File "/data/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in      apply_op
    op_def=op_def)
  File "/data/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/data/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1204, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[128,60,256]
   [[Node: model_att_1/attention/strided_slice_199 = StridedSlice[Index=DT_INT32, T=DT_FLOAT, begin_mask=5,           ellipsis_mask=0, end_mask=5, new_axis_mask=0, shrink_axis_mask=0, _device="/job:localhost/replica:0/task:0/gpu:       0"](model_att_1/attention/concat, model_att_1/attention/strided_slice_199/stack, model_att_1/attention/               strided_slice_199/stack_1, model_att_1/attention/strided_slice_199/stack_2)]]
   [[Node: div_1/_69195 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0",         send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_11676_div_1",       tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]*

sess.run(validation_iterator.initializer)