Python 熊猫：如何避免嵌套for循环

我有一些代码将实际数据与目标数据进行比较，其中实际数据位于一个数据帧中，目标位于另一个数据帧中。我需要查找目标，将其与实际数据一起放入df，然后比较两者。在下面的简化示例中，我有一组产品和一组位置，它们都有唯一的目标

我使用一个嵌套的for循环来实现这一点：循环遍历产品，然后遍历位置。问题是，我的真实生活数据在所有维度上都比较大，循环遍历所有内容需要花费大量的时间

我看过各种各样的SO文章，没有一篇是我能找到的！似乎与熊猫有关和/或与我的问题有关。有人对如何矢量化这段代码有好的想法吗

import pandas as pd
import numpy as np
import time

employee_list = ['Joe', 'Bernie', 'Elizabeth', 'Kamala', 'Cory', 'Pete', 
                'Amy', 'Andrew', 'Beto', 'Jay', 'Kristen', 'Julian', 
                'Mike', 'John', 'Tulsi', 'Tim', 'Eric', 'Seth', 'Howard',
                'Bill']
location_list = ['Denver', 'Boulder', 'Phoenix', 'Reno', 'Portland',
                'Eugene', 'San Francisco']
product_list = ['Product1', 'Product2', 'Product3', 'Product4', 'Product5']

tgt_data = {'Location' : location_list, 
            'Product1' : [600, 200, 750, 225, 450, 175, 900],
            'Product2' : [300, 100, 350, 125, 200, 90, 450],
            'Product3' : [700, 250, 950, 275, 600, 225, 1200],
            'Product4' : [200, 100, 250, 75, 150, 75, 300],
            'Product5' : [900, 300, 1000, 400, 600, 275, 1300]}
tgt_df = pd.DataFrame(data = tgt_data)

employee_data = {'Employee' : employee_list,
                'Location' : ['Boulder', 'Denver', 'Portland', 'Denver',
                            'San Francisco', 'Phoenix', 'San Francisco',
                            'Eugene', 'San Francisco', 'Reno', 'Denver',
                            'Phoenix', 'Denver', 'Portland', 'Reno', 
                            'Boulder', 'San Francisco', 'Phoenix', 
                            'San Francisco', 'Phoenix'],
                'Product1' : np.random.randint(1, 1000, 20),
                'Product2' : np.random.randint(1, 700, 20),
                'Product3' : np.random.randint(1, 1500, 20),
                'Product4' : np.random.randint(1, 500, 20),
                'Product5' : np.random.randint(1, 1500, 20)}
emp_df = pd.DataFrame(data = employee_data)


start = time.time()
for p in product_list:
    for l in location_list:
        emp_df.loc[emp_df['Location'] == l, p + '_tgt'] = (
            tgt_df.loc[tgt_df['Location']==l, p].values)
    emp_df[p + '_pct'] = emp_df[p] / emp_df[p + '_tgt']

print(emp_df)
end = time.time()
print(end - start)

---

您正在使用宽格式数据帧。我觉得长格式更容易操作

# turn emp_df into long
# indexed by "Employee", "Location", and "Product"
emp_df = (emp_df.set_index(['Employee', 'Location'])
                .stack().to_frame())
emp_df.head()

                                      0
Employee    Location        
Joe         Boulder     Product1    238
                        Product2    135
                        Product3    873
                        Product4    153
                        Product5    373

# turn tmp_df into a long series
# indexed by "Location" and "Product"

tgt_df = tgt_df.set_index('Location').stack()
tgt_df.head()


# set target for employees by locations:
emp_df['target'] = (emp_df.groupby('Employee')[0]
                          .apply(lambda x: tgt_df))

# percentage
emp_df['pct'] = emp_df[0]/emp_df['target']

# you can get the wide format back by
# emp_df = emp_df.unstack(level=2)
# which will give you a  dataframe with 
# multi-level index and multi-level column

---

如果保证目标数据帧具有唯一的位置，则可以使用连接使此过程非常快速

import pandas as pd
import numpy as np
import time

employee_list = ['Joe', 'Bernie', 'Elizabeth', 'Kamala', 'Cory', 'Pete', 
                'Amy', 'Andrew', 'Beto', 'Jay', 'Kristen', 'Julian', 
                'Mike', 'John', 'Tulsi', 'Tim', 'Eric', 'Seth', 'Howard',
                'Bill']
location_list = ['Denver', 'Boulder', 'Phoenix', 'Reno', 'Portland',
                'Eugene', 'San Francisco']
product_list = ['Product1', 'Product2', 'Product3', 'Product4', 'Product5']

tgt_data = {'Location' : location_list, 
            'Product1' : [600, 200, 750, 225, 450, 175, 900],
            'Product2' : [300, 100, 350, 125, 200, 90, 450],
            'Product3' : [700, 250, 950, 275, 600, 225, 1200],
            'Product4' : [200, 100, 250, 75, 150, 75, 300],
            'Product5' : [900, 300, 1000, 400, 600, 275, 1300]}
tgt_df = pd.DataFrame(data = tgt_data)

employee_data = {'Employee' : employee_list,
                'Location' : ['Boulder', 'Denver', 'Portland', 'Denver',
                            'San Francisco', 'Phoenix', 'San Francisco',
                            'Eugene', 'San Francisco', 'Reno', 'Denver',
                            'Phoenix', 'Denver', 'Portland', 'Reno', 
                            'Boulder', 'San Francisco', 'Phoenix', 
                            'San Francisco', 'Phoenix'],
                'Product1' : np.random.randint(1, 1000, 20),
                'Product2' : np.random.randint(1, 700, 20),
                'Product3' : np.random.randint(1, 1500, 20),
                'Product4' : np.random.randint(1, 500, 20),
                'Product5' : np.random.randint(1, 1500, 20)}
emp_df = pd.DataFrame(data = employee_data)

安装完成后，我们现在可以使用join

product_tgt_cols = [product+'_tgt' for product in product_list]
print(product_tgt_cols) #['Product1_tgt', 'Product2_tgt', 'Product3_tgt', 'Product4_tgt', 'Product5_tgt']
product_pct_cols = [product+'_pct' for product in product_list]
print(product_pct_cols) #['Product1_pct', 'Product2_pct', 'Product3_pct', 'Product4_pct', 'Product5_pct']

start = time.time()
#join on location to get _tgt columns
emp_df = emp_df.join(tgt_df.set_index('Location'), on='Location', rsuffix='_tgt')
#divide the entire product arrays using numpy, store in temp
temp = emp_df[product_list].values/emp_df[product_tgt_cols].values
#create a new temp df for the _pct results, and assign back to emp_df
emp_df = emp_df.assign(**pd.DataFrame(temp, columns = product_pct_cols))
print(emp_df)

end = time.time()
print("with join: ",end - start)

---

你的真实数据有多大？它在记忆中是否合适？解决方案取决于需求。描述一下你正试图用for循环做什么。喜欢MCVE，这也是一个问题！哭着开心地流泪这很有效。将我的真实df的时间从大约13秒缩短到0.2秒！