pandas中的python异步读取sql
我想通过将查询拆分为4来加快从数据库获取数据的过程。我使用apply\u async编写了以下代码。然而,在使用get时,出现了酸洗错误。我该怎么办?多谢各位pandas中的python异步读取sql,python,multithreading,pandas,Python,Multithreading,Pandas,我想通过将查询拆分为4来加快从数据库获取数据的过程。我使用apply\u async编写了以下代码。然而,在使用get时,出现了酸洗错误。我该怎么办?多谢各位 from multiprocessing import Pool pool = Pool(processes=4) start_date = datetime.datetime(2017, 1, 1) end_date = datetime.datetime(2017, 6, 30) period = (end_date-start_da
from multiprocessing import Pool
pool = Pool(processes=4)
start_date = datetime.datetime(2017, 1, 1)
end_date = datetime.datetime(2017, 6, 30)
period = (end_date-start_date)/4
conn = pyodbc.connect(
r'DRIVER={SQL Server};'
r'SERVER=abc;'
r'PORT=111;'
r'DATABASE=db;'
r'UID=abc;'
r'PWD=xyz;'
r'TDS_Version=7.1'
)
for p in np.arange(start_date, end_date, period).astype(datetime.datetime):
sql = "SELECT * FROM db where date between \'" + str(p) + "\' and \'" + str(p + period) + "\'"
res.append(pool.apply_async(lambda x: pd.read_sql(x[0], con = x[1]), ([sql, conn],))) # runs in *only* one process
pool.close()
res[0].get()#<-------PicklingError: Can't pickle <function <lambda> at 0x00000045566BDAE8>: attribute lookup <lambda>
您需要将连接线移动到每个子流程中:替换lambda x。。。通过一个例程连接到服务器,然后发送请求。不能打开单个连接并在子流程之间共享它 或者,您可以用aioodbc替换pyodbc:
这将允许您使用asyncio实现您需要的功能有人能帮忙吗?您有没有弄明白这一点?