Python Tensorflow:使用py_func的自定义数据读取器
我正在尝试将hdf5文件中的数据排队。由于Tensorflow不支持hdf5,因此我创建了一个python函数,该函数从hdf5文件中读取示例,并在到达文件末尾时引发Python Tensorflow:使用py_func的自定义数据读取器,python,tensorflow,Python,Tensorflow,我正在尝试将hdf5文件中的数据排队。由于Tensorflow不支持hdf5,因此我创建了一个python函数,该函数从hdf5文件中读取示例,并在到达文件末尾时引发tf.errors.OutOfRangeError。然后我用tf.py_func包装这个python函数,并将其用作队列的排队操作 这是我的代码: import h5py import tensorflow as tf from tensorflow.python.framework import errors import num
tf.errors.OutOfRangeError
。然后我用tf.py_func
包装这个python函数,并将其用作队列的排队操作
这是我的代码:
import h5py
import tensorflow as tf
from tensorflow.python.framework import errors
import numpy as np
def read_from_hdf5(hdf5_file, batch_size):
h5py_handle = h5py.File(hdf5_file)
# Check shapes from the hdf5 file so that we can set the tensor shapes
feature_shape = h5py_handle['features'].shape[1:]
label_shape = h5py_handle['labels'].shape[1:]
#generator that produces examples for training. It will be wrapped by tf.pyfunc to simulate a reader
def example_generator(h5py_handle):
for i in xrange(0, h5py_handle['features'].shape[0]-batch_size+1, batch_size):
features = h5py_handle['features'][i:i+batch_size]
labels = h5py_handle['labels'][i:i+batch_size]
yield [features, labels]
raise errors.OutOfRangeError(node_def=None, op=None, message='completed all examples in %s'%hdf5_file)
[features_tensor, labels_tensor] = tf.py_func(
example_generator(h5py_handle).next,
[],
[tf.float32, tf.float32],
stateful=True)
# Set the shape so that we can infer sizes etc in later layers.
features_tensor.set_shape([batch_size, feature_shape[0], feature_shape[1], feature_shape[2]])
labels_tensor.set_shape([batch_size, label_shape[0]])
return features_tensor, labels_tensor
def load_data_from_filename_list(hdf5_files, batch_size, shuffle_seed=0):
example_list = [read_from_hdf5(hdf5_file, batch_size) for hdf5_file in hdf5_files]
min_after_dequeue = 10000
capacity = min_after_dequeue + (len(example_list)+1) * batch_size #min_after_dequeue + (num_threads + a small safety margin) * batch_size
features, labels = tf.train.shuffle_batch_join(example_list, batch_size, capacity=capacity, min_after_dequeue=min_after_dequeue, seed=shuffle_seed, enqueue_many=True)
return features, labels, metadata
我希望tf.errors.OutOfRangeError
将由QueueRunner处理,但是,我得到以下错误,程序崩溃。有可能从py_func中进行这种读取吗?如果有,我做错了什么?如果不是,我应该使用什么方法来代替
Traceback (most recent call last):
File "/users/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/script_ops.py", line 85, in __call__
ret = func(*args)
File "build/bdist.linux-x86_64/egg/tronn/datalayer.py", line 27, in example_generator
raise errors.OutOfRangeError(node_def=None, op=None, message='completed all examples in %s'%hdf5_file)
tensorflow.python.framework.errors_impl.OutOfRangeError: completed all examples
W tensorflow/core/framework/op_kernel.cc:993] Internal: Failed to run py callback pyfunc_13: see error log.
似乎不支持在
py_func
中处理异常
在py_func.cc
// Invokes the trampoline.
PyObject* result = PyEval_CallObject(trampoline, args);
Py_DECREF(args);
if (result == nullptr) {
if (PyErr_Occurred()) {
// TODO(zhifengc): Consider pretty-print error using LOG(STDERR).
PyErr_Print();
}
return errors::Internal("Failed to run py callback ", call->token,
": see error log.");
}
PyErr\u occurrent
是在生成异常时设置的,因此这将导致执行抛出无法运行py回调
py_func
是一个奇怪的生物,因为它运行在Python客户机环境中。通常,当op(如读卡器op)失败时,从TF运行时传播的op将返回not ok状态到Python客户端,然后Python客户端将其转换为Python异常(在raise\u exception\u not\u ok\u status
(在client.py:session.run中)。由于py_func
body在Python客户端中运行,因此需要修改TensorFlow以处理发生的PyErr_
,以便将错误状态插入TensorFlow运行时。如果您显示代码以及错误消息,以便人们可以重现您的问题,可能会有所帮助。谢谢您的提示。我已经添加了我的代码。非常感谢您的解释。这似乎表明我不能使用py_func作为读取器。我可以采取什么方法来代替呢?我不需要使用读取器,而是使用一个Python线程对示例进行排队,然后关闭队列。然后,在使用示例时,常规出列将引发OutOfRange错误。这显示了如何将自定义生成器的所有输出排队