Machine learning Google Colab上的运行时崩溃

为什么运行时总是在GoogleColab上崩溃

我有一个简单的MLP代码，在我的机器上运行。我尝试在Colab上运行相同的代码，但加载数据文件后它立即崩溃

数据文件的总容量约为3GB。Colab虚拟机的CPU和GPU内存很容易超过这一点

那为什么我的程序还没开始训练就崩溃了呢

我的代码：

def load_raw(name):
  return (np.load(name + '.npy', encoding='bytes'), np.load(name + '_labels.npy', encoding='bytes'))

class WSJ():
 def __init__(self):
    self.dev_set = None
    self.train_set = None
    self.test_set = None


@property
def dev(self):
    if self.dev_set is None:
        self.dev_set = load_raw('dev')
    return self.dev_set

@property

def train(self):
    if self.train_set is None:
        self.train_set = load_raw('train')
    return self.train_set


@property
def test(self):
    if self.test_set is None:
        self.test_set = (np.load('test.npy', encoding='bytes'), None)
    return self.test_set

def preprocess_data(self, trainX, trainY, k):
     # some form of preprocessing that pads and flattens the data into the format required        

    return trainX_padded, trainY, y_to_x_map







def main():

 global index
 padding = 3
 epochs = 1
 batch_size = 512
 lr = 0.1
 momentum = 0.9

 input_dim = 40 * ((2*padding) + 1)
 output_dim = 138

 neural_net = MLP(input_dim, output_dim) 
 !free -g

 print("Starting...")
 loader = WSJ()
 trainX, trainY = loader.train
 print("Training Data obtained...")
 !free -g
 trainX, trainY, y_to_x_map = loader.preprocess_data(trainX, trainY, k = padding)
 print("Training Data preprocessed...")

 !free -g


 devX, devY = loader.dev
 devX, devY, y_to_x_map_dev = loader.preprocess_data(devX, devY, k = padding)
 print("Development data preprocessed...")

 !free -g


 print("Scaling...")
 input_scaler = preprocessing.StandardScaler().fit(trainX)

 !free -g

 trainX = input_scaler.transform(trainX)
 devX = input_scaler.transform(devX)

---

打印缩放后它会立即崩溃…

您可以发布一个最小的示例来重现错误吗？问题中添加了请共享一个可以执行的示例笔记本。上面的示例代码没有执行任何操作。我怀疑您达到了内存限制——似乎有一些相关信息。