Python 带估计器的Tensorflow嵌入最佳实践

我有一个CNN for CIFAR-10数据集，其中包含以下层：

[IN] -> [CONV] -> [POOL] -> [CONV] -> [POOL] -> [FC] -> [DROPOUT] -> [LOGITS] -> [OUT]
                                                                 \-> [EMBEDDINGS]

估算器代码：

config = tf.contrib.learn.RunConfig(save_checkpoints_secs=30)

# Create the Estimator
classifier = tf.estimator.Estimator(model_fn=inference, config=config, model_dir=LOG_DIR)
train_images, train_labels, train_labels_onehot = Utils.load_training_data()

hooks = [
    # logging hook
    tf.train.LoggingTensorHook(tensors=tensors_to_log, every_n_iter=50),
]

train_input_fn = tf.estimator.inputs.numpy_input_fn(x={'x': train_images}, y=train_labels)

classifier.train(input_fn=train_input_fn, steps=FLAGS.steps, hooks=hooks)

推理功能代码：

def inference(self, features, labels, mode):
    try:
        images = tf.cast(features['x'], tf.float32)
        # Input Layer
        with tf.name_scope('Data'):
            input_layer = tf.reshape(images, [-1, img_width, img_height, num_channels])

        # Convolutional Layer 1
        with tf.variable_scope('ConvLayer1'):
            conv1 = tf.layers.conv2d(inputs=input_layer, filters=32, kernel_size=[5, 5], 
                                     padding="same", activation=tf.nn.relu)
            pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[2, 2], strides=2)

        logging.info('Convolutional Layer 1 build successful..')

        # Convolutional Layer 1
        with tf.variable_scope('ConvLayer2'):
            conv2 = tf.layers.conv2d(inputs=pool1, filters=64, kernel_size=[5, 5], 
                                     padding="same", activation=tf.nn.relu)
            pool2 = tf.layers.max_pooling2d(inputs=conv2, pool_size=[2, 2], strides=2)

        logging.info('Convolutional Layer 2 build successful..')

        # Fully Connected Layer
        with tf.variable_scope('FullyConnectedLayer'):
            pool2_flat = tf.reshape(pool2, [-1, 8 * 8 * 64])
            dense = tf.layers.dense(inputs=pool2_flat, units=1024, activation=tf.nn.relu)
            dropout = tf.layers.dropout(inputs=dense, rate=0.4, 
                                        training=(mode == tf.estimator.ModeKeys.TRAIN))

        logging.info('Fully Connected Layer build successful..')

        tf.summary.histogram('dropout', dropout)

        # Logits Layer
        logits = tf.layers.dense(inputs=dropout, units=10)

        tf.summary.histogram('logits', logits)

        logging.info('Logits Layer build successful..')

        predictions = {
            # Generate predictions (for PREDICT and EVAL mode)
            "classes": tf.argmax(input=logits, axis=1),
            # Add `softmax_tensor` to the graph. It is used for PREDICT and by the
            # `logging_hook`.
            "probabilities": tf.nn.softmax(logits, name="softmax_tensor")
        }

        if mode == tf.estimator.ModeKeys.PREDICT:
            return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions, 
                                              evaluation_hooks=[])

        # Calculate Loss (for both TRAIN and EVAL modes)
        onehot_labels = tf.one_hot(indices=tf.cast(labels, tf.int32), depth=10)
        loss = tf.losses.softmax_cross_entropy(onehot_labels=onehot_labels,
                                               logits=logits)

        tf.summary.histogram('loss', loss)

        logging.info('Losses build successful..')

        # Configure the Training Op (for TRAIN mode)
        if mode == tf.estimator.ModeKeys.TRAIN:
            learning_rate = tf.train.exponential_decay(start_learning_rate, 
                                tf.train.get_global_step(), 1000, 0.9, staircase=True)
            optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
            train_op = optimizer.minimize(loss=loss, global_step=tf.train.get_global_step())
            return tf.estimator.EstimatorSpec(mode=mode, loss=loss, train_op=train_op,
                                              scaffold=tf.train.Scaffold(
                                                  summary_op=tf.summary.merge_all(),
                                              ))

        # Add evaluation metrics (for EVAL mode)
        accuracy = tf.metrics.accuracy(labels=labels, predictions=predictions["classes"])

        tf.summary.histogram('accuracy', accuracy)

        logging.info('Accuracy metric build successful..')

        return tf.estimator.EstimatorSpec(mode=mode, loss=loss,
                                      train_op=train_op,
                                      scaffold=tf.train.Scaffold(
                                          summary_op=tf.summary.merge_all()
                                      ))

我试图在tensorflow中使用嵌入可视化，这里我想将退出输出可视化为嵌入

我发现使用嵌入的代码：

sess = tf.InteractiveSession()

# Input set for Embedded TensorBoard visualization
# Performed with cpu to conserve memory and processing power
with tf.device("/cpu:0"):
    embedding = tf.Variable(self._data, trainable=False, name='embedding')

sess.run(embedding.initializer)

writer = tf.summary.FileWriter(LOG_DIR + '/projector', sess.graph)

config = projector.ProjectorConfig()
embed = config.embeddings.add()
embed.tensor_name = embedding.name

embed.metadata_path = os.path.join(LOG_DIR + '/projector/metadata.tsv')

embed.sprite.image_path = os.path.join(DATA_DIR + '/cifar_10k_sprite.png')
embed.sprite.single_image_dim.extend([img_width, img_height])

projector.visualize_embeddings(writer, config)

saver = tf.train.Saver([embedding])
saver.save(sess, os.path.join(LOG_DIR, 'projector/a_model.ckpt'))

在我的例子中，它不起作用，因为我使用的是Estimator类，并且我没有访问会话的权限

我尝试过的方法：

将numpy.array变量传递给估计器的model_fn，在那里我可以设置该变量的值，然后将该变量传递给SessionRunHook，在那里我可以访问会话并将数据保存到文件中。不起作用，因为传递给估计器的所有参数都变成张量。所以这种方法不起作用，因为我已经有了辍学层张量

创建一个全局变量，我可以把所有的值的辍学层。不太有效，因为对于这种方法，我需要访问张量值

就我所理解的估计器架构而言，主要问题是从估计器中获取掉层输出信号，并以某种方式将其传递给SessionRunHook，以将其保存为嵌入。但我认为这不是最好的办法

在估计器中使用嵌入的正确方法是什么？

我就是这样做的（但这可能不是最有效的方法）：

会话取消挂钩：

import tensorflow as tf

from classes.Utils import Utils


class EmbeddingSaverHook(tf.train.SessionRunHook):

    def __init__(self, values, labels, captions):
        self._saver = None

        self._classes = Utils.get_classnames()

        self._dense3 = None
        self._labels = None

        self._emb_values = values
        self._emb_labels = labels
        self._emb_captions = captions

    def begin(self):
        self._dense3 = tf.get_default_graph().get_tensor_by_name("dense3/BiasAdd:0")

        self._labels = tf.get_default_graph().get_tensor_by_name("labels:0")

    def before_run(self, run_context):
        return tf.train.SessionRunArgs([self._dense3, self._labels])

    def after_run(self, run_context, run_values):
        self._emb_values.extend(run_values[0][0])
        self._emb_labels.extend(run_values[0][1])
        self._emb_captions.extend([self._classes[x] for x in run_values[0][1]])

    def end(self, session):
        pass

---

您可以看到的完整代码

SessionRunHooks是让任意操作运行的正确方法，就像这里的保护程序一样。@AlexandrePasso是的，我尝试过这种方法，但问题是，当我试图创建tf.Variable from drop-out tensor时，模型停止了输出：2017-12-01 10:36:37.578243:I tensorflow/core/common\u runtime/gpu/gpu\device.cc:1120]创建tensorflow设备（/device:gpu:0）->（设备：0，名称：GeForce GTX 1080 Ti，pci总线id:0000:02:00.0，计算能力：6.1）当您试图获取某些内容时，似乎队列尚未启动