Python 如何在Spark上分发Tensorflow/TFLearn模型？

我正在尝试根据以下模型在Tensorflow中运行分布式图像分类：。不幸的是，我的数据集太大，无法在一台机器上可靠地计算。所以我尝试将模型和数据集移动到工作节点上

# Initialize Spark
sc_name = "MNIST"
sc_master = "local[*]"
from pyspark import SparkContext, SparkConf
conf = SparkConf().setAppName(sc_name).setMaster(sc_master)
sc = SparkContext(conf=conf)

#Import TF/TFLearn
from __future__ import division, print_function, absolute_import
import tensorflow as tf

import tflearn
from tflearn.layers.core import input_data, dropout, fully_connected
from tflearn.layers.conv import conv_2d, max_pool_2d
from tflearn.layers.normalization import local_response_normalization
from tflearn.layers.estimator import regression

# Model to be run on workers
def build_graph(X):
    with tf.Graph().as_default() as g:
        network = input_data(shape=[None, 28, 28, 1], name='input')
        network = conv_2d(network, 32, 3, activation='relu', regularizer="L2")
        network = max_pool_2d(network, 2)
        network = local_response_normalization(network)
        network = conv_2d(network, 64, 3, activation='relu', regularizer="L2")
        network = max_pool_2d(network, 2)
        network = local_response_normalization(network)
        network = fully_connected(network, 128, activation='tanh')
        network = dropout(network, 0.8)
        network = fully_connected(network, 256, activation='tanh')
        network = dropout(network, 0.8)
        network = fully_connected(network, 10, activation='softmax')
        network = regression(network, optimizer='adam', learning_rate=0.01,
                             loss='categorical_crossentropy', name='target')

        # Training
        model = tflearn.DNN(network, tensorboard_verbose=0)

        with tf.Session() as sess:
            model.fit({'input': X}, {'target': Y}, n_epoch=20,
                   validation_set=({'input': testX}, {'target': testY}),
                   snapshot_step=100, show_metric=True, run_id='convnet_mnist')

# Data loading and preprocessing
import tflearn.datasets.mnist as mnist
X, Y, testX, testY = mnist.load_data(one_hot=True)
X = X.reshape([-1, 28, 28, 1])
testX = testX.reshape([-1, 28, 28, 1])

#Run model on data
rdd = sc.parallelize(X)
rdd.map(build_graph).collect()

这给了我一个错误： ValueError:无法为具有形状“（？，28，28，1）”的张量“输入/X:0”馈送形状（28，28，1）的值

我怀疑这是实际问题，因为X_火车的形状（55000,28,28,1）

---

我认为问题在于我没有正确地并行化模型或数据。如果我基本上误解了如何执行此操作，请告诉我。

传递给map的函数将分别为每个记录调用。因此，

build\u graph

看到的只是一条shape

（28，28，1）

记录。因此，基本上您尝试创建N个模型，并且由于

build\u graph

不返回模型，所以丢弃它们。将数组转换为子张量列表修复了问题。例如，从[10000,28,28,1]到[101000,28,28,1]。Spark然后适当地将[1000,28,28,1]张量平行化给每个工人。@Davidot我很高兴也很感激你能解决这个问题。您可以回答下面自己的问题，使问题显示为已回答状态。人们经常忽略评论部分