Python 使用float32和float64 slow分配数据帧

带有浮动32和浮动64不同数据类型的Pandas数据帧的赋值对于某些组合来说比我的方式慢得多

下面的代码设置数据帧，对部分数据进行Numpy/Scipy计算，通过复制旧数据帧设置新数据帧，并将计算结果分配给新数据帧：

import pandas as pd
import numpy as np
from scipy.signal import lfilter

N = 1000
M = 1000

def f(dtype1, dtype2):
    coi = [str(m) for m in range(M)]
    df = pd.DataFrame([[m for m in range(M)] + ['Hello', 'World'] for n in range(N)],
                      columns=coi + ['A', 'B'], dtype=dtype1)
    Y = lfilter([1], [0.5, 0.5], df.ix[:, coi])
    Y = Y.astype(dtype2)
    new = pd.DataFrame(df, copy=True)
    print(new.iloc[0, 0].dtype)
    print(Y.dtype)
    new.ix[:, coi] = Y    # This statement is considerably slow
    print(new.iloc[0, 0].dtype)


from time import time

dtypes = [np.float32, np.float64]
for dtype1 in dtypes:
    for dtype2 in dtypes:
        print('-' * 10)
        start_time = time()
        f(dtype1, dtype2)
        print(time() - start_time)

计时结果是：

----------
float32
float32
float64
10.1998147964
----------
float32
float64
float64
10.2371120453
----------
float64
float32
float64
0.864870071411
----------
float64
float64
float64
0.866265058517

----------
float32
float32
float32
0.809890985489
----------
float32
float64
float64
21.4767119884
----------
float64
float32
float32
20.5611870289
----------
float64
float64
float64
0.765362977982

这里的临界线是

new.ix[：，coi]=Y

：对于某些组合，它的速度是原来的十倍

我可以理解，当有一个float32数据帧并且它被分配了一个float64时，重新分配需要一些开销。但为什么开销如此巨大呢

---

此外，float32和float32赋值的组合也很慢，结果是float64，这也让我感到困扰。

单列赋值不会改变类型，对于非类型转换赋值，使用for循环遍历列似乎相当快，-float32和float64。对于涉及类型转换的赋值，性能通常是多列赋值最差性能的两倍

import pandas as pd
import numpy as np
from scipy.signal import lfilter

N = 1000
M = 1000

def f(dtype1, dtype2):
    coi = [str(m) for m in range(M)]
    df = pd.DataFrame([[m for m in range(M)] + ['Hello', 'World'] for n in range(N)],
                      columns=coi + ['A', 'B'], dtype=dtype1)
    Y = lfilter([1], [0.5, 0.5], df.ix[:, coi])
    Y = Y.astype(dtype2)
    new = df.copy()
    print(new.iloc[0, 0].dtype)
    print(Y.dtype)
    for n, column in enumerate(coi):  # For-loop over columns new!
        new.ix[:, column] = Y[:, n]
    print(new.iloc[0, 0].dtype)

from time import time

dtypes = [np.float32, np.float64]
for dtype1 in dtypes:
    for dtype2 in dtypes:
        print('-' * 10)
        start_time = time()
        f(dtype1, dtype2)
        print(time() - start_time)

结果是：

----------
float32
float32
float64
10.1998147964
----------
float32
float64
float64
10.2371120453
----------
float64
float32
float64
0.864870071411
----------
float64
float64
float64
0.866265058517

----------
float32
float32
float32
0.809890985489
----------
float32
float64
float64
21.4767119884
----------
float64
float32
float32
20.5611870289
----------
float64
float64
float64
0.765362977982

---

这可能与胁迫有关。请参见后续问题：我在GitHub中添加了一个问题：