Python 使用Tensorflow数据集时,如何在decode_csv中声明分类列?
我正在尝试使用Tensorflow数据集API从csv文件中读取列 我首先列出我列的名称以及我有多少列:Python 使用Tensorflow数据集时,如何在decode_csv中声明分类列?,python,tensorflow,Python,Tensorflow,我正在尝试使用Tensorflow数据集API从csv文件中读取列 我首先列出我列的名称以及我有多少列: numerical_feature_names = ["N1", "N2"] categorical_feature_names = ["C3", "C4", "C5"] amount_of_columns_csv = 5 然后声明我的列类型: feature_columns = [tf.feature_column.numeric_column(k) for k in numerical
numerical_feature_names = ["N1", "N2"]
categorical_feature_names = ["C3", "C4", "C5"]
amount_of_columns_csv = 5
然后声明我的列类型:
feature_columns = [tf.feature_column.numeric_column(k) for k in numerical_feature_names]
for k in categorical_feature_names:
current_categorical_column = tf.feature_column.categorical_column_with_hash_bucket(
key=k,
hash_bucket_size=40)
feature_columns.append(tf.feature_column.indicator_column(current_categorical_column))
最后是我的输入函数:
def my_input_fn(file_path, perform_shuffle=False, repeat_count=1):
def decode_csv(line):
parsed_line = tf.decode_csv(line, [[0.]]*amount_of_columns_csv, field_delim=';', na_value='-1')
d = dict(zip(feature_names, parsed_line)), label
return d
dataset = (tf.data.TextLineDataset(file_path) # Read text file
.skip(1) # Skip header row
.map(decode_csv)) # Transform each elem by applying decode_csv fn
if perform_shuffle:
# Randomizes input using a window of 512 elements (read into memory)
dataset = dataset.shuffle(buffer_size=BATCH_SIZE)
dataset = dataset.repeat(repeat_count) # Repeats dataset this # times
dataset = dataset.batch(BATCH_SIZE) # Batch size to use
iterator = dataset.make_one_shot_iterator()
batch_features, batch_labels = iterator.get_next()
return batch_features, batch_labels
如何在decode\u csv
调用中声明我的record\u defaults
参数?
目前,我只捕获带有[[0.]]
如果我有上千个混合了数字和分类列的列,我如何避免在decode_csv函数中手动声明结构?我首先将其加载到panda数据帧中,而不是尝试直接在Tensorflow中加载csv,迭代列dtype并设置我的类型数组,以便在Tensorflow输入函数中重用它,代码如下:
CSV_COLUMN_NAMES = pd.read_csv(FILE_TRAIN, nrows=1).columns.tolist()
train = pd.read_csv(FILE_TRAIN, names=CSV_COLUMN_NAMES, header=0)
train_x, train_y = train, train.pop('labels')
test = pd.read_csv(FILE_TEST, names=CSV_COLUMN_NAMES, header=0)
test_x, test_y = test, test.pop('labels')
# iterate over the columns type to create my column array
for column in train.columns:
print (train[column].dtype)
if(train[column].dtype == np.float64 or train[column].dtype == np.int64):
numerical_feature_names.append(column)
else:
categorical_feature_names.append(column)
feature_columns = [tf.feature_column.numeric_column(k) for k in numerical_feature_names]
# here an example of how you could process categorical columns
for k in categorical_feature_names:
current_bucket = train[k].nunique()
if current_bucket>10:
feature_columns.append(
tf.feature_column.indicator_column(
tf.feature_column.categorical_column_with_vocabulary_list(key=k, vocabulary_list=train[k].unique())
)
)
else:
feature_columns.append(
tf.feature_column.indicator_column(
tf.feature_column.categorical_column_with_hash_bucket(key=k, hash_bucket_size=current_bucket)
)
)
最后是输入函数
# input_fn for training, convertion of dataframe to dataset
def train_input_fn(features, labels, batch_size, repeat_count):
dataset = tf.data.Dataset.from_tensor_slices((dict(features), labels))
dataset = dataset.shuffle(256).repeat(repeat_count).batch(batch_size)
return dataset
tf.data.experimental.make_csv_数据集将为您完成这项工作。当涉及到预测时,将把您从CSV带到tf.feature_列。您需要在之前执行转换吗?示例:dataset:{'age':int,'country':str},我按照您的建议对train和eval进行转换,但在预测方面,我们可能需要将数据转换为我训练过的相同格式,对吗?