Python 使用numpy为RNN准备数据的最快方法是什么？

我目前有一个

（1631160,78）

np数组作为神经网络的输入。我想尝试一下LSTM，它需要一个3D结构作为输入数据。我目前正在使用以下代码生成所需的3D结构，但速度非常慢（ETA>1day）。有没有更好的办法让numpy做到这一点

我当前生成数据的代码：

def transform_for_rnn(input_x, input_y, window_size):
    output_x = None
    start_t = time.time()
    for i in range(len(input_x)):
        if i > 100 and i % 100 == 0:
            sys.stdout.write('\rTransform Data: %d/%d\tETA:%s'%(i, len(input_x), str(datetime.timedelta(seconds=(time.time()-start_t)/i * (len(input_x) - i)))))
            sys.stdout.flush()
        if output_x is None:
            output_x = np.array([input_x[i:i+window_size, :]])
        else:
            tmp = np.array([input_x[i:i+window_size, :]])
            output_x = np.concatenate((output_x, tmp))

    print
    output_y = input_y[window_size:]
    assert len(output_x) == len(output_y)
    return output_x, output_y

下面是一种用于矢量化

output\ux

-

nrows = input_x.shape[0] - window_size + 1
p,q = input_x.shape
m,n = input_x.strides
strided = np.lib.stride_tricks.as_strided
out = strided(input_x,shape=(nrows,window_size,q),strides=(m,m,n))

样本运行-

In [83]: input_x
Out[83]: 
array([[ 0.73089384,  0.98555845,  0.59818726],
       [ 0.08763718,  0.30853945,  0.77390923],
       [ 0.88835985,  0.90506367,  0.06204614],
       [ 0.21791334,  0.77523643,  0.47313278],
       [ 0.93324799,  0.61507976,  0.40587073],
       [ 0.49462016,  0.00400835,  0.66401908]])

In [84]: window_size = 4

In [85]: out
Out[85]: 
array([[[ 0.73089384,  0.98555845,  0.59818726],
        [ 0.08763718,  0.30853945,  0.77390923],
        [ 0.88835985,  0.90506367,  0.06204614],
        [ 0.21791334,  0.77523643,  0.47313278]],

       [[ 0.08763718,  0.30853945,  0.77390923],
        [ 0.88835985,  0.90506367,  0.06204614],
        [ 0.21791334,  0.77523643,  0.47313278],
        [ 0.93324799,  0.61507976,  0.40587073]],

       [[ 0.88835985,  0.90506367,  0.06204614],
        [ 0.21791334,  0.77523643,  0.47313278],
        [ 0.93324799,  0.61507976,  0.40587073],
        [ 0.49462016,  0.00400835,  0.66401908]]])

这将在输入数组中创建一个视图，因此在内存方面我们是高效的。在大多数情况下，随着涉及it的进一步操作，这也会转化为性能方面的好处。让我们验证一下这确实是一个视图-

In [86]: np.may_share_memory(out,input_x)
Out[86]: True   # Doesn't guarantee, but is sufficient in most cases

另一种可靠的验证方法是在

输出中设置一些值，然后检查输入-

In [87]: out[0] = 0

In [88]: input_x
Out[88]: 
array([[ 0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.93324799,  0.61507976,  0.40587073],
       [ 0.49462016,  0.00400835,  0.66401908]])

下面是一种用于矢量化
output\ux
-

nrows = input_x.shape[0] - window_size + 1
p,q = input_x.shape
m,n = input_x.strides
strided = np.lib.stride_tricks.as_strided
out = strided(input_x,shape=(nrows,window_size,q),strides=(m,m,n))

样本运行-

In [83]: input_x
Out[83]: 
array([[ 0.73089384,  0.98555845,  0.59818726],
       [ 0.08763718,  0.30853945,  0.77390923],
       [ 0.88835985,  0.90506367,  0.06204614],
       [ 0.21791334,  0.77523643,  0.47313278],
       [ 0.93324799,  0.61507976,  0.40587073],
       [ 0.49462016,  0.00400835,  0.66401908]])

In [84]: window_size = 4

In [85]: out
Out[85]: 
array([[[ 0.73089384,  0.98555845,  0.59818726],
        [ 0.08763718,  0.30853945,  0.77390923],
        [ 0.88835985,  0.90506367,  0.06204614],
        [ 0.21791334,  0.77523643,  0.47313278]],

       [[ 0.08763718,  0.30853945,  0.77390923],
        [ 0.88835985,  0.90506367,  0.06204614],
        [ 0.21791334,  0.77523643,  0.47313278],
        [ 0.93324799,  0.61507976,  0.40587073]],

       [[ 0.88835985,  0.90506367,  0.06204614],
        [ 0.21791334,  0.77523643,  0.47313278],
        [ 0.93324799,  0.61507976,  0.40587073],
        [ 0.49462016,  0.00400835,  0.66401908]]])

这将在输入数组中创建一个视图，因此在内存方面我们是高效的。在大多数情况下，随着涉及it的进一步操作，这也会转化为性能方面的好处。让我们验证一下这确实是一个视图-

In [86]: np.may_share_memory(out,input_x)
Out[86]: True   # Doesn't guarantee, but is sufficient in most cases

另一种可靠的验证方法是在
输出中设置一些值，然后检查输入-

In [87]: out[0] = 0

In [88]: input_x
Out[88]: 
array([[ 0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.93324799,  0.61507976,  0.40587073],
       [ 0.49462016,  0.00400835,  0.66401908]])

这里是为LSTM/RNN创建数据的最快方法：这里是为LSTM/RNN创建数据的最快方法：