Python PyTorch:如何实现图形注意层的注意
我已经实现了的注意(等式1),但它显然没有内存效率,只能在我的GPU上运行单个型号(需要7-10GB) 目前,我有Python PyTorch:如何实现图形注意层的注意,python,graph,deep-learning,pytorch,attention-model,Python,Graph,Deep Learning,Pytorch,Attention Model,我已经实现了的注意(等式1),但它显然没有内存效率,只能在我的GPU上运行单个型号(需要7-10GB) 目前,我有 class MyModule(nn.Module): def __init__(self, in_features, out_features): super(MyModule, self).__init__() self.in_features = in_features self.out_features = out_features sel
class MyModule(nn.Module):
def __init__(self, in_features, out_features):
super(MyModule, self).__init__()
self.in_features = in_features
self.out_features = out_features
self.W = nn.Parameter(nn.init.xavier_uniform(torch.Tensor(in_features, out_features).type(torch.cuda.FloatTensor if torch.cuda.is_available() else torch.FloatTensor), gain=np.sqrt(2.0)), requires_grad=True)
self.a = nn.Parameter(nn.init.xavier_uniform(torch.Tensor(2*out_features, 1).type(torch.cuda.FloatTensor if torch.cuda.is_available() else torch.FloatTensor), gain=np.sqrt(2.0)), requires_grad=True)
def forward(self, input):
h = torch.mm(input, self.W)
N = h.size()[0]
a_input = torch.cat([h.repeat(1, N).view(N * N, -1), h.repeat(N, 1)], dim=1).view(N, -1, 2 * self.out_features)
e = F.elu(torch.matmul(a_input, self.a).squeeze(2))
return e
我计算所有e_ij项的洞察力是
In [8]: import torch
在[9]中:将numpy作为np导入
[10]中:h=火炬式长传感器(np.阵列([[1,1],[2,2],[3,3]]))
In[11]:N=3
[12]:h.重复(1,N).视图(N*N,-1)
出[12]:
1 1
1 1
1 1
2 2
2 2
2 2
3 3
3 3
3 3
[尺寸为9x2的火炬式传感器]
In[13]:h.重复(N,1)
出[13]:
1 1
2 2
3 3
1 1
2 2
3 3
1 1
2 2
3 3
[尺寸为9x2的火炬式传感器]
最后连接hs和feed矩阵a
有没有一种更方便记忆的方法呢?也许你可以用稀疏张量来存储调整矩阵
def sparse_mx_to_torch_sparse_tensor(sparse_mx):
"""Convert a scipy sparse matrix to a torch sparse tensor."""
sparse_mx = sparse_mx.tocoo().astype(np.float32)
indices = torch.from_numpy(np.vstack((sparse_mx.row,
sparse_mx.col))).long()
values = torch.from_numpy(sparse_mx.data)
shape = torch.Size(sparse_mx.shape)
return torch.sparse.FloatTensor(indices, values, shape)