如何存储一个Python脚本中变量的计算值,以便在另一个Python脚本中反复使用
我有一个python(3.7.0)脚本文件,它接收17000多个文本文件,然后逐行处理每个文件,将所需的数据填充到数据框中。因此,代码如下所示是Script1.py文件:如何存储一个Python脚本中变量的计算值,以便在另一个Python脚本中反复使用,python,python-3.x,pandas,Python,Python 3.x,Pandas,我有一个python(3.7.0)脚本文件,它接收17000多个文本文件,然后逐行处理每个文件,将所需的数据填充到数据框中。因此,代码如下所示是Script1.py文件: def filePrpcesserFunc(file): data = open(file, 'r') for line in data.readlines(): #Does some computing here to get var1, var2, var3 #var1, var2, v
def filePrpcesserFunc(file):
data = open(file, 'r')
for line in data.readlines():
#Does some computing here to get var1, var2, var3
#var1, var2, var3 are all integers
return var1, var2, var3
if __name__ == '__main__':
import glob2
import pandas
files = glob2.glob("*.txt")
#There are more than 17,000 text files to be computed
X = pandas.DataFrame()
for file in files:
rvar1, rvar2, rvar3 = fileProcesserFunc(file)
X.loc[str(rvar1), str(rvar2)] = rvar3
现在,上面的python脚本运行了17000多个文件来生成pandas dataframe X,它只需要计算一次就可以获得数据(而且计算非常昂贵)。我需要在另一个脚本文件(比如)中使用这个数据框来创建一个机器学习模型,因此为了测试我的模型,我需要一次又一次地访问X,也就是说,我需要一次又一次地运行Script2.py。
有人能提出什么建议吗?我将非常感谢您。您可以使用内置模块:
在script1.py
中:
import pickle
with open('dataframe.txt', 'wb') as f:
pickle.dump(X, f)
import pickle
X = pickle.load(open('dataframe.txt', 'rb'))
X.to_pickle("./df.pkl")
import pandas as pd
X = pd.read_pickle(¨./df.pkl¨)
在script2.py
中:
import pickle
with open('dataframe.txt', 'wb') as f:
pickle.dump(X, f)
import pickle
X = pickle.load(open('dataframe.txt', 'rb'))
X.to_pickle("./df.pkl")
import pandas as pd
X = pd.read_pickle(¨./df.pkl¨)
您还可以在数据帧上加载内置方法:
在script1.py
中:
import pickle
with open('dataframe.txt', 'wb') as f:
pickle.dump(X, f)
import pickle
X = pickle.load(open('dataframe.txt', 'rb'))
X.to_pickle("./df.pkl")
import pandas as pd
X = pd.read_pickle(¨./df.pkl¨)
在script2.py
中:
import pickle
with open('dataframe.txt', 'wb') as f:
pickle.dump(X, f)
import pickle
X = pickle.load(open('dataframe.txt', 'rb'))
X.to_pickle("./df.pkl")
import pandas as pd
X = pd.read_pickle(¨./df.pkl¨)
您可以使用内置模块:
在script1.py
中:
import pickle
with open('dataframe.txt', 'wb') as f:
pickle.dump(X, f)
import pickle
X = pickle.load(open('dataframe.txt', 'rb'))
X.to_pickle("./df.pkl")
import pandas as pd
X = pd.read_pickle(¨./df.pkl¨)
在script2.py
中:
import pickle
with open('dataframe.txt', 'wb') as f:
pickle.dump(X, f)
import pickle
X = pickle.load(open('dataframe.txt', 'rb'))
X.to_pickle("./df.pkl")
import pandas as pd
X = pd.read_pickle(¨./df.pkl¨)
您还可以在数据帧上加载内置方法:
在script1.py
中:
import pickle
with open('dataframe.txt', 'wb') as f:
pickle.dump(X, f)
import pickle
X = pickle.load(open('dataframe.txt', 'rb'))
X.to_pickle("./df.pkl")
import pandas as pd
X = pd.read_pickle(¨./df.pkl¨)
在script2.py
中:
import pickle
with open('dataframe.txt', 'wb') as f:
pickle.dump(X, f)
import pickle
X = pickle.load(open('dataframe.txt', 'rb'))
X.to_pickle("./df.pkl")
import pandas as pd
X = pd.read_pickle(¨./df.pkl¨)