使用tensorflow drop_余数时，张量形状不正确

我正在尝试从生成器创建

tf.data.Dataset

。我想确保所有批次的大小完全相同，因此我在数据集上调用

.batch（batch\u size，drop\u rements=True）

。以下是相关代码：

train_data = tf.data.Dataset.from_generator(
    lambda: map(tuple, train_generator),
    (tf.float32, tf.float32),
    (
        tf.TensorShape([batch_size, crop_height, crop_width, 3]),
        tf.TensorShape([batch_size, crop_height, crop_width, 3]),
    ),
)

val_data = tf.data.Dataset.from_generator(
    lambda: map(tuple, val_generator),
    (tf.float32, tf.float32),
    (
        tf.TensorShape([batch_size, crop_height, crop_width, 3]),
        tf.TensorShape([batch_size, crop_height, crop_width, 3]),
    ),
)

my_train_data = train_data.batch(batch_size, drop_remainder=True)
my_val_data = val_data.batch(batch_size, drop_remainder=True)

但我在运行它时会出现以下错误：

tensorflow.python.framework.errors_impl.InvalidArgumentError: input must be 4-dimensional[4,4,64,64,48] [Op:FusedBatchNormV3]

我得到这个错误是因为我对数据进行了两次批处理（

batch\u size

在我的错误消息中是4）。我试图用

生成器
命令中的
无
替换
批量大小
，但我得到了相同的错误。如果我完全像这样删除第一个参数：

(tf.TensorShape([options["crop_height"], options["crop_width"], 3]),
            tf.TensorShape([options["crop_height"], options["crop_width"], 3]),
        )

我得到这个错误：

ValueError: `generator` yielded an element of shape (4, 128, 128, 3) where an element of shape (128, 128, 3) was expected.

我如何使用
drop_remains
而不对数据进行两次批处理

编辑：

添加与生成器关联的代码：

class BaseGenerator(Sequence):
    def __init__(
        self,
        image_filenames,
        label_filenames,
        batch_size=1,
        is_train=True,
        preprocess=None,
        augment=None,
        height=128,
        width=128,
        shuffle=False,
    ):
        self.indices = np.arange(0, len(image_filenames))
        self.image_filenames = np.array(image_filenames)
        self.label_filenames = np.array(label_filenames)
        self.batch_size = batch_size
        self.is_train = is_train
        self.preprocess = preprocess
        self.augment = augment
        self.crop_height = height
        self.crop_width = width
        self.shuffle = shuffle
        self.on_epoch_end()  # shuffle data
    
    def __len__(self):
        return int(np.ceil(len(self.indices) / float(self.batch_size)))
    
    def __getitem__(self, index):
        min_index = index * self.batch_size
        max_index = min((index + 1) * self.batch_size, len(self.indices))
        batch_indices = self.indices[min_index:max_index]
        return self.generate(self.image_filenames[batch_indices], self.label_filenames[batch_indices])
    
    def __call__(self):
        return next(iter(self))
    
    def on_epoch_end(self):
        if self.is_train and self.shuffle:
            np.random.shuffle(self.indices)
    
    def generate(self, image_filenames, label_filenames):
        X = np.zeros((self.batch_size, self.crop_height, self.crop_width, 3), dtype=np.float32)
        y = np.zeros((self.batch_size, self.crop_height, self.crop_width), dtype=np.float32,)
    
        for i, (image_fn, label_fn) in enumerate(zip(image_filenames, label_filenames)):
            image = utils.load_image(image_fn)
            label = utils.load_image(label_fn)
    
            if self.augment:
                augmented = self.augment(image=image, mask=label)
                image = augmented["image"]
                label = augmented["mask"]
    
            if self.preprocess:
                image = self.preprocess(image)
    
            label = np.float32(helpers.one_hot_it(label=label))
    
            X[i, :, :, :] = image
            y[i, :, :, :] = label
    
        return X, y


train_generator = BaseGenerator(
    image_filenames=train_input_names,
    label_filenames=train_output_names,
    batch_size=batch_size,
    is_train=True,
    preprocess=preprocessing,
    augment=None,
    height=128,
    width=128,
)

val_generator = BaseGenerator(
    image_filenames=val_input_names,
    label_filenames=val_output_names,
    batch_size=batch_size,
    is_train=False,
    preprocess=preprocessing,
    augment=None,
    height=128,
    width=128,
)

正如您在问题中提到的，问题是您要对数据进行两次批处理。要解决此问题，您可以：

首先，定义生成单个图像的生成器（例如，没有批次维度）
然后，使用
tf.data.Dataset
的方法将示例分组

为了重新定义
BaseGenerator
，以便生成单个图像，您可以执行以下步骤

首先，在
\uuuu init\uuuu
方法中，删除
批处理大小，因为不再需要它：

def __init__(
        self,
        image_filenames,
        label_filenames,
        is_train=True,
        preprocess=None,
        augment=None,
        height=128,
        width=128,
        shuffle=False,
):
    self.indices = np.arange(0, len(image_filenames))
    self.image_filenames = np.array(image_filenames)
    self.label_filenames = np.array(label_filenames)
    self.is_train = is_train
    self.preprocess = preprocess
    self.augment = augment
    self.crop_height = height
    self.crop_width = width
    self.shuffle = shuffle
    self.on_epoch_end()  # shuffle data

其次，调整方法
generate
，以便生成一个示例：

def generate(self, image_filename, label_filename):
    image = utils.load_image(image_filename)
    label = utils.load_label(label_filename)

    if self.augment:
        augmented = self.augment(image=image, mask=label)
        image = augmented["image"]
        label = augmented["mask"]

    if self.preprocess:
        image = self.preprocess(image)

    label = np.float32(helpers.one_hot_it(label=label))

    X = image  # Shape=(self.crop_height, self.crop_width, 3)
    Y = label  # Shape=(self.crop_height, self.crop_width)

    return X, y

第三，在方法
\uuu getitem\uuu
中，只传递一个文件名：

def __getitem__(self, index):
    return self.generate(self.image_filenames[index], self.label_filenames[index])

最后，在定义
tf.data.Dataset
时排除批处理维度：

train_data = tf.data.Dataset.from_generator(
    lambda: map(tuple, train_generator),
    (tf.float32, tf.float32),
    (
        tf.TensorShape([crop_height, crop_width, 3]),
        tf.TensorShape([crop_height, crop_width]),
    ),
)

my_train_data = train_data.batch(batch_size, drop_remainder=True)

it = iter(my_train_data)
x, y = next(it)

print(x.shape)  # (4, 128, 128, 3)
print(y.shape)  # (4, 128, 128)

请添加计算图以及输入数据形状。输入图像为（128、128、3）。我必须看看如何添加计算图形请显示
train\u generator
和
val\u generator
的代码，以便进行调试，如果您能提供我添加的生成器代码，那就更好了。我知道这还不是一个完整的例子。@rvinas是的，是你。我还在努力，希望明天能完成。