Python 运行多处理代码时,数据帧不会更新
我试图用下面的类用temp-df(self.df_temp['linkedin_profile'])中的一列更新数据帧(self.df),但它似乎没有更新任何内容。 守则:Python 运行多处理代码时,数据帧不会更新,python,pandas,dataframe,multiprocessing,screen-scraping,Python,Pandas,Dataframe,Multiprocessing,Screen Scraping,我试图用下面的类用temp-df(self.df_temp['linkedin_profile'])中的一列更新数据帧(self.df),但它似乎没有更新任何内容。 守则: class NameToSocialURLScraper: def __init__(self, csv_file_name, person_name_column, organization_name_column): self.proxy_list = PROXY_LIST pool = Pool()
class NameToSocialURLScraper:
def __init__(self, csv_file_name, person_name_column, organization_name_column):
self.proxy_list = PROXY_LIST
pool = Pool()
self.csv_file_name = csv_file_name
self.person_name_column = person_name_column
self.organization_name_column = organization_name_column
self.df = pd.read_csv(csv_file_name)
self.df_temp = pd.DataFrame()
def internal_linkedin_job(self):
self.df['linkedin_profile'] = np.nan
self.df_temp['linkedin_profile'] = np.nan
self.df_temp['linkedin_profile'] = self.df.apply(
lambda row: term_scraper(
str(row[self.person_name_column]) + " " + str(row[self.organization_name_column]), self.proxy_list,
'link', output_generic=False), axis=1)
self.df['linkedin_profile'] = self.df_temp['linkedin_profile']
print(self.df.values)
...
def multiprocess_job(self):
multiprocessing.log_to_stderr(logging.DEBUG)
linkedin_profile_proc = Process(target=self.internal_linkedin_job, args=())
jobs = [linkedin_profile_proc]
# Start the processes (i.e. calculate the random number lists)
for j in jobs:
j.start()
# Ensure all of the processes have finished
for j in jobs:
j.join()
在内部linkedin作业中打印时,它会显示带有新列“linkedin\u profile”的df,但在j.join()之后打印时,该列不存在。在执行多处理时,每个进程都在其自己的内存空间中运行。您需要重构代码,以便内部linkedin作业返回数据帧。明白了!借助以下答案,使用Queue解决了此问题: