Machine learning 如何避免给定网络中的过度装配
我正在尝试实现一个用于句子分类的CNN网络；我正试图遵循中提出的架构。我用Keras（张量流）来做这个。以下是我的模型摘要：Machine learning 如何避免给定网络中的过度装配,machine-learning,neural-network,deep-learning,Machine Learning,Neural Network,Deep Learning,我正在尝试实现一个用于句子分类的CNN网络；我正试图遵循中提出的架构。我用Keras（张量流）来做这个。以下是我的模型摘要： ____________________________________________________________________________________________________ Layer (type) Output Shape Param # Connected to
____________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
====================================================================================================
input_4 (InputLayer) (None, 56) 0
____________________________________________________________________________________________________
embedding (Embedding) (None, 56, 300) 6510000
____________________________________________________________________________________________________
dropout_7 (Dropout) (None, 56, 300) 0
____________________________________________________________________________________________________
conv1d_10 (Conv1D) (None, 54, 100) 90100
____________________________________________________________________________________________________
conv1d_11 (Conv1D) (None, 53, 100) 120100
____________________________________________________________________________________________________
conv1d_12 (Conv1D) (None, 52, 100) 150100
____________________________________________________________________________________________________
max_pooling1d_10 (MaxPooling1D) (None, 27, 100) 0
____________________________________________________________________________________________________
max_pooling1d_11 (MaxPooling1D) (None, 26, 100) 0
____________________________________________________________________________________________________
max_pooling1d_12 (MaxPooling1D) (None, 26, 100) 0
____________________________________________________________________________________________________
flatten_10 (Flatten) (None, 2700) 0
____________________________________________________________________________________________________
flatten_11 (Flatten) (None, 2600) 0
____________________________________________________________________________________________________
flatten_12 (Flatten) (None, 2600) 0
____________________________________________________________________________________________________
concatenate_4 (Concatenate) (None, 7900) 0
____________________________________________________________________________________________________
dropout_8 (Dropout) (None, 7900) 0
____________________________________________________________________________________________________
dense_7 (Dense) (None, 50) 395050
____________________________________________________________________________________________________
dense_8 (Dense) (None, 5) 255
====================================================================================================
Total params: 7,265,605.0
Trainable params: 7,265,605.0
Non-trainable params: 0.0
if model_type in ['CNN-non-static', 'CNN-static']:
embedding_wts = train_word2vec( np.vstack((x_train, x_test, x_valid)),
ind_to_wrd, num_features = embedding_dim)
if model_type == 'CNN-static':
x_train = embedding_wts[0][x_train]
x_test = embedding_wts[0][x_test]
x_valid = embedding_wts[0][x_valid]
elif model_type == 'CNN-rand':
embedding_wts = None
else:
raise ValueError("Unknown model type")
batch_size = 50
filter_sizes = [3,4,5]
num_filters = 75
dropout_prob = (0.5, 0.8)
hidden_dims = 50
l2_reg = 0.3
# Deciding dimension of input based on the model
input_shape = (max_sent_len, embedding_dim) if model_type == "CNN-static" else (max_sent_len,)
model_input = Input(shape = input_shape)
# Static model do not have embedding layer
if model_type == "CNN-static":
z = Dropout(dropout_prob[0])(model_input)
else:
z = Embedding(vocab_size, embedding_dim, input_length = max_sent_len, name="embedding")(model_input)
z = Dropout(dropout_prob[0])(z)
# Convolution layers
z1 = Conv1D( filters=num_filters, kernel_size=3,
padding="valid", activation="relu",
strides=1)(z)
z1 = MaxPooling1D(pool_size=2)(z1)
z1 = Flatten()(z1)
z2 = Conv1D( filters=num_filters, kernel_size=4,
padding="valid", activation="relu",
strides=1)(z)
z2 = MaxPooling1D(pool_size=2)(z2)
z2 = Flatten()(z2)
z3 = Conv1D( filters=num_filters, kernel_size=5,
padding="valid", activation="relu",
strides=1)(z)
z3 = MaxPooling1D(pool_size=2)(z3)
z3 = Flatten()(z3)
# Concatenate the output of all convolution layers
z = Concatenate()([z1, z2, z3])
z = Dropout(dropout_prob[1])(z)
# Dense(64, input_dim=64, kernel_regularizer=regularizers.l2(0.01), activity_regularizer=regularizers.l1(0.01))
z = Dense(hidden_dims, activation="relu", kernel_regularizer=regularizers.l2(0.01))(z)
model_output = Dense(N_category, activation="sigmoid")(z)
model = Model(model_input, model_output)
model.compile(loss="categorical_crossentropy", optimizer=optimizers.Adadelta(lr=1, decay=0.005), metrics=["accuracy"])
model.summary()
在不深入研究模型的情况下，我想说的是，您应该尽量不训练嵌入和重用可下载矩阵中的一个。即使你削减了，你仍然有几乎10倍于你有数据点的参数，因此模型必然是过度拟合 应该是相反的。对于800k参数，您应该有8M个数据点
如果您查看图表，验证LOO在第一个（几个）历元时下降，然后上升，这是没有足够数据的另一个迹象。如果不深入查看模型，我会说您应该尝试不训练嵌入并重用其中一个可下载矩阵。即使你削减了，你仍然有几乎10倍于你有数据点的参数，因此模型必然是过度拟合 应该是相反的。对于800k参数，您应该有8M个数据点
如果您查看图表，验证LOO在第一个（几个）历元下降，然后上升，这是没有足够数据的另一个迹象。从模型摘要看，您正在处理5个输出类的多类问题。对于多类设置，
softmaxsigmoid- 我猜您正在遵循中解释的类似方法。但是，如果您的训练数据很小，那么最佳做法是使用预训练单词嵌入并允许微调。您可以找到预训练的嵌入初始化示例。但允许通过在嵌入层中设置
进行微调trainable=True - 在上面提到的论文中，他从每个过滤器中提取单个特征，
。这将从整个句子中只提取一个重要特征。此设置将允许您降低模型复杂性。参考资料中解释得很好：pool\u size=max\u sent\u len-filter\u size+1 - 在相同的方法中，他们只使用了一个致密层，你们可以移除你们的一个隐藏的致密层。从拉动层中退出的输出特征数量较少（将等于过滤器数量，每个过滤器仅输出一个特征）
softmaxsigmoid- 我猜您正在遵循中解释的类似方法。但是，如果您的训练数据很小，那么最佳做法是使用预训练单词嵌入并允许微调。您可以找到预训练的嵌入初始化示例。但允许通过在嵌入层中设置
进行微调trainable=True - 在上面提到的论文中，他从每个过滤器中提取单个特征，
。这将从整个句子中只提取一个重要特征。此设置将允许您降低模型复杂性。参考资料中解释得很好：pool\u size=max\u sent\u len-filter\u size+1 - 在相同的方法中，他们只使用了一个致密层，你们可以移除你们的一个隐藏的致密层。从拉动层中退出的输出特征数量较少（将等于过滤器数量，每个过滤器仅输出一个特征）