Machine learning BERT训练前准确度不增加

我试图在包含150k个句子的数据集（wiki103）上预训练BERT。12个纪元后，nsp（下一句预测）任务的准确度约为0.76（如果我继续使用更多纪元，则会超出范围），而mlm（蒙面语言建模）任务的准确度从0.01 acc开始，最大为0.2。这里出了什么问题？我是否可以在某一点上停止nsp，并在更长的时间内继续进行传销？我的火车装载机长度为2486（每个时代2486个训练步数），这意味着40*2486=99440个训练步数

这是模型配置和训练配置

class Train_Config():
""" Hyperparameters for training """
seed: int = 391275 # random seed
batch_size: int = 64
lr: int = 1e-5 # learning rate
n_epochs: int = 40 # the number of epoch
# `warm up` period = warmup(0.1)*total_steps
# linearly increasing learning rate from zero to the specified value(5e-5)
warmup: float = 0.1
is_dibert: bool = False


class Model_Config():
vocab_size: int = 30522  # Size of Vocabulary
hidden_size: int = 768  # Dimension of Hidden Layer in Transformer Encoder
num_hidden_layers: int = 8  # Numher of Hidden Layers
num_attention_heads: int = 8  # Numher of Heads in Multi-Headed Attention Layers
intermediate_size: int = 768 * 4  # Dimension of Intermediate Layers in Positionwise Feedforward Net
# activ_fn: str = "gelu" # Non-linear Activation Function Type in Hidden Layers
max_len: int = 312  # Maximum Length for Positional Embeddings
n_segments: int = 2  # Number of Sentence Segments
attention_probs_dropout_prob: int = 0.1

---

在这种情况下，nsm是什么？@Ashwingeted'Sa我的错。这是下一句预测。现在有问题了。谢谢。这种情况下nsm是什么？@Ashwingeted'Sa我的错。这是下一句预测。现在有问题了。谢谢