Python 在Pytorch模型中更新权重和偏差时如何防止内存使用增长

我正在尝试构建一个VGG16模型，以便使用Pytork进行ONNX导出。我想用我自己的一组权重和偏差强制模型。但在这个过程中，我的电脑很快就耗尽了内存

下面是我想要做的（这只是一个测试，在真实版本中，我读取了一组文件中的权重和偏差），这个示例仅强制所有值为0.5

# Create empty VGG16 model (random weights)
from torchvision import models
from torchsummary import summary

vgg16 = models.vgg16()
# la structure est : vgg16.__dict__
summary(vgg16, (3, 224, 224))

#  convolutive layers
for layer in vgg16.features:
    print()
    print(layer)
    if (hasattr(layer,'weight')):
        dim = layer.weight.shape
        print(dim)
        print(str(dim[0]*(dim[1]*dim[2]*dim[3]+1))+' params')

        # Remplacement des poids et biais
        for i in range (dim[0]):
            layer.bias[i] = 0.5
            for j in range (dim[1]):
                for k in range (dim[2]):
                    for l in range (dim[3]):
                        layer.weight[i][j][k][l] = 0.5

# Dense layers
for layer in vgg16.classifier:
    print()
    print(layer)
    if (hasattr(layer,'weight')):
        dim = layer.weight.shape
        print(str(dim)+' --> '+str(dim[0]*(dim[1]+1))+' params')
        for i in range(dim[0]):
            layer.bias[i] = 0.5
            for j in range(dim[1]):
                layer.weight[i][j] = 0.5

当我查看计算机的内存使用情况时，它会线性增长，并在第一个密集层处理期间使16GB RAM饱和。然后python崩溃了

有没有其他更好的方法来实现这一点，请记住，我希望在以后使用onnx导出模型？

---

谢谢你的帮助。

记忆增长是因为需要为每次重量和偏差变化调整梯度。尝试在更新之前将

。requires_grad

属性设置为

False

，并在更新之后将其还原。例如：

for layer in vgg16.features:
    print()
    print(layer)
    if (hasattr(layer,'weight')):
        
        # supress .requires_grad
        layer.bias.requires_grad = False
        layer.weight.requires_grad = False
        
        dim = layer.weight.shape
        print(dim)
        print(str(dim[0]*(dim[1]*dim[2]*dim[3]+1))+' params')

        # Remplacement des poids et biais
        for i in range (dim[0]):
            layer.bias[i] = 0.5
            for j in range (dim[1]):
                for k in range (dim[2]):
                    for l in range (dim[3]):
                        layer.weight[i][j][k][l] = 0.5
        
        # restore .requires_grad
        layer.bias.requires_grad = True
        layer.weight.requires_grad = True

我还找到了一个解决方案，它基于创建一个具有新权重的字典并使用它来更新模型：

weight_dict.update(new_weight_dict)

你觉得怎么样？有关于这个方法的文档吗？

---

也许速度更快？

我终于这样做了：它不会使RAM饱和，而且比我以前的版本快得多

# Create empty VGG16 model (random weights)
import torch
from torchvision import models
from torchsummary import summary

vgg16 = models.vgg16()
# The structure is in : vgg16.__dict__
summary(vgg16, (3, 224, 224))
state_dict = vgg16.state_dict()
print(state_dict.keys())

# Prepare to read files
import os
workdir = 'C:\\Users\\...\\'
path = workdir + '...\\'

# Convolutive layers
index = 1
nCouche = 0
with torch.no_grad():
    for layer in vgg16.features:
        print()
        print(layer)
        if (hasattr(layer,'weight')):

            dim = layer.weight.shape
            print(str(dim) + ' : ' +str(dim[0]*(dim[1]*dim[2]*dim[3]+1))+' parameters')

            # Read the weights
            nomFichier = nomCouche + '_weights.syntxt'
            print('Reading file: ' + nomFichier)
            file = open (path + nomFichier,"r")
            weights = []
            for line in file:
                wprov = [int(x) for x in line.split()]
                weights.append([[[wprov[(k*dim[3]+j)*dim[2]+i] for i in range(dim[3])] for j in range(dim[2])] for k in range(dim[1])])
            file.close()

            # Read the biases
            nomFichier = nomCouche + '_biases.syntxt'
            print('Reading file: ' + nomFichier)
            file = open (path + nomFichier,"r")
            biases = []
            for line in file:
                biases.append(int(line))
            file.close()

            # Replace weights and biases
            nameW = 'features.'+str(nCouche)+'.weight'
            nameB = 'features.'+str(nCouche)+'.bias'
            print ('Features: ' + nameW + ' & ' + nameB)
            w = torch.empty(dim)
            b = torch.empty(dim[0])

            w = torch.IntTensor(weights)
            b = torch.IntTensor(biases)
            state_dict[nameW].copy_(w)
            state_dict[nameB].copy_(b)

            index += 1
        nCouche += 1

    index = 1
    nCouche = 0
    # Dense layers
    for layer in vgg16.classifier:
        print()
        print(layer)
        if (hasattr(layer,'weight')):
            dim = layer.weight.shape
            print(str(dim)+' --> '+str(dim[0]*(dim[1]+1))+' params')

            # Prepare file names
            nomFichier = nomCouche + '_weights.syntxt'
            print('Reading file: ' + nomFichier)
            file = open (path+nomFichier,"r")

            # Prepare to replace weights and biases
            nameW = 'classifier.'+str(nCouche)+'.weight'
            nameB = 'classifier.'+str(nCouche)+'.bias'
            print ('Features: ' + nameW + ' & ' + nameB)
            w = torch.empty(dim)
            b = torch.empty(dim[0])

            # Read the weights
            i = 0
            interval = 500
            print('Countdown:') # because it can be quite long
            for line in file:
                weights = [int(x) for x in line.split()]
                w[i,:] = torch.IntTensor(weights)
                i += 1
                if ((dim[0] - i) % interval == 0):
                    print(f"{i/dim[0]*100:.1f}"+'%')
            print('Read ' + str(i) + ' lines of ' + str(len(weights)) + ' weights')
            file.close()

            # Read the biases
            nomFichier = 'dense'+str(index)+'_biases.syntxt'
            print('Reading file: ' + nomFichier)
            file = open (path+nomFichier,"r")
            i = 0
            for line in file:
                biases = [int(y) for y in line.split()]
                b[i] = biases[0]
                i += 1
            file.close()

            state_dict[nameW].copy_(w)
            state_dict[nameB].copy_(b)

            index += 1
        nCouche += 1

# Update state dictionary
vgg16.load_state_dict(state_dict)

它读取一系列文件中的权重和偏差，并将它们替换为网络的各种张量。

---

也许它可以帮助可能有类似问题的人…

如果您可以提供新的权重作为完整张量，则无需在嵌套循环中逐个更新它们：

for layer in vgg16.features:
    if (hasattr(layer,'weight')):
        # Remplacement des poids et biais
        new_bias = torch.rand (layer.bias.shape)  # <-- YOUR DATA GOES HERE
        new_weight = torch.rand (layer.weight.shape) # <-- YOUR DATA GOES HERE
        layer_bias = torch.nn.Parameter(new_bias, requires_grad=True) 
        layer.weight = torch.nn.Parameter(new_weight, requires_grad=True)

vgg16.1中的图层功能： 如果（hasattr（层，'weight'））： #波依德和比亚斯酒店

---

new_bias=torch.rand（layer.bias.shape）#谢谢。我来试试这个，太完美了！内存使用保持稳定，即使对于密集层也是如此。再次感谢你，怎么办？（抱歉，这里是新手）要将答案标记为已接受，请单击答案旁边的复选标记，将其从灰色变为已填写。这也是一种很有希望的方式。通常，最好在一次操作中更新权重，而不是使用4个嵌套循环。除非你出于某种原因必须使用嵌套循环，否则这会影响我们。谢谢，你能给我指一个文档或一个示例吗？我指的是一个关于weight_dict.update的文档，这是关于autograd的（这也很有帮助）查看我的最新答案和新示例