Python tensorflow：如何使用tf.estimator.train_和_评估进行分布式训练

参考上的tf.estimator.train_和_评估示例 我想让示例运行分布式模式，但培训过程没有在分布式模式下启动似乎不起作用

我的问题是： 如何使源代码分布式运行？

INFO:tensorflow:Using default config.

INFO:tensorflow:Using config: {'_save_checkpoints_secs': 600, '_session_config': None, '_keep_checkpoint_max': 5, '_task_type': u'chief', '_train_distribute': None, '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fc0024f18d0>, '_evaluation_master': '', '_save_checkpoints_steps': None, '_keep_checkpoint_every_n_hours': 10000, '_service': None, '_num_ps_replicas': 1, '_tf_random_seed': None, '_master': u'grpc://192.168.6.99.123:2222', '_num_worker_replicas': 2, '_task_id': 0, '_log_step_count_steps': 100, '_model_dir': '/home/clxman/tf/', '_global_id_in_cluster': 0, '_save_summary_steps': 100}

INFO:tensorflow:Start Tensorflow server.

2018-06-12 23:23:25.694865: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2

2018-06-12 23:23:25.697065: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job chief -> {0 -> localhost:2222}

2018-06-12 23:23:25.697159: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> 192.168.6.99.123:2400}

2018-06-12 23:23:25.697180: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> 192.168.6.99.123:2300}

2018-06-12 23:23:25.698882: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:332] Started server with target: grpc://localhost:2222

**data**

INFO:tensorflow:Calling model_fn.

DEBUG:tensorflow:Transforming feature_column _NumericColumn(key='x', shape=(), default_value=None, dtype=tf.float32, normalizer_fn=None).

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Graph was finalized.

平台：系统：Ubuntu 18 tensorflow:v1.8

以下日志是我的源代码和操作步骤：

1源代码：

import tensorflow as tf
import os
import sys
import json
import logging
import numpy as np
x = np.random.rand(1000)
y = np.random.choice([0,1],1000)
def data_input():
    ret={}
    ret['x'] = x   
    y_batch = y 
    print "data"
    return ret,y_batch  
tf.logging.set_verbosity(tf.logging.DEBUG)
my_feature_columns=[]
v_feature_column = tf.feature_column.numeric_column(key="x",shape=[])
my_feature_columns.append(v_feature_column)

estimator = tf.estimator.DNNClassifier(
    feature_columns=my_feature_columns,
    hidden_units=[1024, 512, 256],
    model_dir='/home/clxman/tf/')


train_spec = tf.estimator.TrainSpec(input_fn=lambda:data_input(), max_steps=1000)
eval_spec = tf.estimator.EvalSpec(input_fn=lambda:data_input())

tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)

2在列车运行良好的单一模式下运行。查看以下日志

INFO:tensorflow:Using default config.

INFO:tensorflow:Using config: {'_save_checkpoints_secs': 600, '_session_config': None, '_keep_checkpoint_max': 5, '_task_type': u'chief', '_train_distribute': None, '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fc0024f18d0>, '_evaluation_master': '', '_save_checkpoints_steps': None, '_keep_checkpoint_every_n_hours': 10000, '_service': None, '_num_ps_replicas': 1, '_tf_random_seed': None, '_master': u'grpc://192.168.6.99.123:2222', '_num_worker_replicas': 2, '_task_id': 0, '_log_step_count_steps': 100, '_model_dir': '/home/clxman/tf/', '_global_id_in_cluster': 0, '_save_summary_steps': 100}

INFO:tensorflow:Start Tensorflow server.

2018-06-12 23:23:25.694865: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2

2018-06-12 23:23:25.697065: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job chief -> {0 -> localhost:2222}

2018-06-12 23:23:25.697159: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> 192.168.6.99.123:2400}

2018-06-12 23:23:25.697180: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> 192.168.6.99.123:2300}

2018-06-12 23:23:25.698882: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:332] Started server with target: grpc://localhost:2222

**data**

INFO:tensorflow:Calling model_fn.

DEBUG:tensorflow:Transforming feature_column _NumericColumn(key='x', shape=(), default_value=None, dtype=tf.float32, normalizer_fn=None).

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Graph was finalized.

clxman@clxman-VirtualBox:~/test$python test\u c.py

/home/clxman/.local/lib/python2.7/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters

INFO:tensorflow:Using default config.

INFO:tensorflow:Using config: {'_save_checkpoints_secs': 600, '_session_config': None, '_keep_checkpoint_max': 5, '_task_type': 'worker', '_train_distribute': None, '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f4e37077890>, '_evaluation_master': '', '_save_checkpoints_steps': None, '_keep_checkpoint_every_n_hours': 10000, '_service': None, '_num_ps_replicas': 0, '_tf_random_seed': None, '_master': '', '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_model_dir': '/home/clxman/tf/', '_global_id_in_cluster': 0, '_save_summary_steps': 100}

INFO:tensorflow:Running training and evaluation locally (non-distributed).

INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after 600 secs (eval_spec.throttle_secs) or training is finished.
data
INFO:tensorflow:Calling model_fn.

DEBUG:tensorflow:Transforming feature_column _NumericColumn(key='x', shape=(), default_value=None, dtype=tf.float32, normalizer_fn=None).

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Graph was finalized.

2018-06-12 23:50:25.702344: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2


INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

**INFO:tensorflow:Saving checkpoints for 1 into /home/clxman/tf/model.ckpt**.

**INFO:tensorflow:loss = 693.219, step = 1**


INFO:tensorflow:global_step/sec: 8.12489


**INFO:tensorflow:loss = 691.08575, step = 101 (12.309 sec)**

INFO:tensorflow:global_step/sec: 8.11321

INFO:tensorflow:loss = 690.9834, step = 201 (12.325 sec)

4下面的日志来自首席培训课程，我们可以看到培训过程没有开始。 字符串“data”表示训练过程调用了数据输入函数，该函数提供训练数据

INFO:tensorflow:Using default config.

INFO:tensorflow:Using config: {'_save_checkpoints_secs': 600, '_session_config': None, '_keep_checkpoint_max': 5, '_task_type': u'chief', '_train_distribute': None, '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fc0024f18d0>, '_evaluation_master': '', '_save_checkpoints_steps': None, '_keep_checkpoint_every_n_hours': 10000, '_service': None, '_num_ps_replicas': 1, '_tf_random_seed': None, '_master': u'grpc://192.168.6.99.123:2222', '_num_worker_replicas': 2, '_task_id': 0, '_log_step_count_steps': 100, '_model_dir': '/home/clxman/tf/', '_global_id_in_cluster': 0, '_save_summary_steps': 100}

INFO:tensorflow:Start Tensorflow server.

2018-06-12 23:23:25.694865: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2

2018-06-12 23:23:25.697065: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job chief -> {0 -> localhost:2222}

2018-06-12 23:23:25.697159: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> 192.168.6.99.123:2400}

2018-06-12 23:23:25.697180: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> 192.168.6.99.123:2300}

2018-06-12 23:23:25.698882: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:332] Started server with target: grpc://localhost:2222

**data**

INFO:tensorflow:Calling model_fn.

DEBUG:tensorflow:Transforming feature_column _NumericColumn(key='x', shape=(), default_value=None, dtype=tf.float32, normalizer_fn=None).

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:使用默认配置。
信息：tensorflow:使用配置：{“保存”检查点：600，“会话配置：无”，“保留”检查点：5，“任务类型：主要”，“训练分布：无”，“是主要”，“正确”，“集群规格：u'grpc://192.168.6.99.123:2222“，”，“\u num\u worker\u replications”：2，“\u task\u id”：0，“\u log\u step\u count\u steps”：100，“\u model\u dir”：“/home/clxman/tf/”，“\u cluster中的全局\u id\u”：0，“\u save\u summary\u steps”：100}”
信息：tensorflow：启动tensorflow服务器。
2018-06-12 23:23:25.694865:I tensorflow/core/platform/cpu\u feature\u guard.cc:140]您的cpu支持该tensorflow二进制文件未编译为使用的指令：AVX2
2018-06-12 23:23:25.697065:I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215]为作业主管初始化GrpcChannelCache->{0->localhost:2222}
2018-06-12 23:23:25.697159:I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215]为作业ps初始化GrpcChannelCache->{0->192.168.6.99.123:2400}
2018-06-12 23:23:25.697180:I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215]为作业工人初始化GrpcChannelCache->{0->192.168.6.99.123:2300}
2018-06-12 23:23:25.698882:I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:332]已启动目标为的服务器：grpc://localhost:2222
**资料**
信息：tensorflow:正在调用模型\u fn。
调试：tensorflow:Transforming feature_column _NumericColumn（key='x'，shape=（），默认值=None，dtype=tf.float32，normalizer_fn=None）。
信息：tensorflow:已完成调用模型\u fn。
信息：tensorflow:创建检查点SaveRhook。
信息：tensorflow：图表已定稿。

您需要运行tf.estimator.RunConfig（）。此外，还需要设置环境变量tf_CONFIG，以指向定义集群设置的json文本

---

查看了解更多详细信息。

您必须在估计器的运行配置中指定分配策略，如本文档所述：

但是，我担心您正在寻找TF1.8中不可用的参数服务器策略（因为您有一个“ps”任务）

请注意，在TF的最新版本中，此API已更改，文档更为完整：