Python 在pytorch的嵌入层中是否可以只冻结某些嵌入权重？

在NLP任务中使用手套嵌入时，手套中可能不存在数据集中的某些单词。因此，我们为这些未知单词实例化随机权重

是否有可能冻结从手套获得的重量，只训练新实例化的重量

我只知道我们可以设置： model.embedding.weight.requires_grad=False

但是这使得生词无法训练

还是有更好的方法来提取单词的语义

一,。将嵌入划分为两个单独的对象 一种方法是使用两个独立的嵌入件，一个用于预训练，另一个用于训练

手套应冻结，而没有预训练表示的手套应取自可训练层

如果您将数据格式化为预训练令牌表示，其范围小于不带手套表示的令牌，则可以执行此操作。假设你的预训练指数在[0300]范围内，而那些没有表示的指数是[301500]。我会遵循以下思路：

import numpy as np
import torch


class YourNetwork(torch.nn.Module):
    def __init__(self, glove_embeddings: np.array, how_many_tokens_not_present: int):
        self.pretrained_embedding = torch.nn.Embedding.from_pretrained(glove_embeddings)
        self.trainable_embedding = torch.nn.Embedding(
            how_many_tokens_not_present, glove_embeddings.shape[1]
        )
        # Rest of your network setup

    def forward(self, batch):
        # Which tokens in batch do not have representation, should have indices BIGGER
        # than the pretrained ones, adjust your data creating function accordingly
        mask = batch > self.pretrained_embedding.shape[0]

        # You may want to optimize it, you could probably get away without copy, though
        # I'm not currently sure how
        pretrained_batch = batch.copy()
        pretrained_batch[mask] = 0

        embedded_batch = self.pretrained_embedding[pretrained_batch]

        # Every token without representation has to be brought into appropriate range
        batch -= self.pretrained_embedding.shape[0]
        # Zero out the ones which already have pretrained embedding
        batch[~mask] = 0
        non_pretrained_embedded_batch = self.trainable_embedding(batch)

        # And finally change appropriate tokens from placeholder embedding created by
        # pretrained into trainable embeddings.
        embedded_batch[mask] = non_pretrained_embedded_batch[mask]

        # Rest of your code
        ...

假设你的预训练指数在[0300]范围内，而那些没有表示的指数是[301500]

2.指定标记的零梯度。 这是一个有点棘手，但我认为它相当简洁，易于实现。因此，如果您获得没有手套表示的令牌的索引，您可以显式地在backprop之后将其梯度归零，这样这些行就不会得到更新

import torch

embedding = torch.nn.Embedding(10, 3)
X = torch.LongTensor([[1, 2, 4, 5], [4, 3, 2, 9]])

values = embedding(X)
loss = values.mean()

# Use whatever loss you want
loss.backward()

# Let's say those indices in your embedding are pretrained (have GloVe representation)
indices = torch.LongTensor([2, 4, 5])

print("Before zeroing out gradient")
print(embedding.weight.grad)

print("After zeroing out gradient")
embedding.weight.grad[indices] = 0
print(embedding.weight.grad)

以及第二种方法的输出：

Before zeroing out gradient
tensor([[0.0000, 0.0000, 0.0000],
        [0.0417, 0.0417, 0.0417],
        [0.0833, 0.0833, 0.0833],
        [0.0417, 0.0417, 0.0417],
        [0.0833, 0.0833, 0.0833],
        [0.0417, 0.0417, 0.0417],
        [0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000],
        [0.0417, 0.0417, 0.0417]])
After zeroing out gradient
tensor([[0.0000, 0.0000, 0.0000],
        [0.0417, 0.0417, 0.0417],
        [0.0000, 0.0000, 0.0000],
        [0.0417, 0.0417, 0.0417],
        [0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000],
        [0.0417, 0.0417, 0.0417]])

---

在第二种方法中，由于您只是将一些获得的梯度设置为零，那么计算不会仍然像从一开始就训练所有嵌入一样繁重吗？@AndreaRossi是的，因此第一种方法要好得多。刚刚概述了另一种可能性