Python 将解析数据多重处理到sqlite时出错

Python 将解析数据多重处理到sqlite时出错,python,python-3.x,web-scraping,sqlite,multiprocessing,Python,Python 3.x,Web Scraping,Sqlite,Multiprocessing,我试图解析一组链接,并将解析后的数据附加到sqlite3。我得到了sqlite3数据库被锁定的错误,所以可能是因为我使用了太高的池值?我试图将其降低到5,但仍会出现如下所示的错误 我的代码基本上如下所示: from multiprocessing import Pool with Pool(5) as p: p.map(parse_link, links) with Pool(5) as p: p.map(Get_FT_OU, file_to_set('links.txt')

我试图解析一组链接,并将解析后的数据附加到sqlite3。我得到了sqlite3数据库被锁定的错误,所以可能是因为我使用了太高的池值?我试图将其降低到5,但仍会出现如下所示的错误

我的代码基本上如下所示:

from multiprocessing import Pool

with Pool(5) as p:
    p.map(parse_link, links)
with Pool(5) as p:
    p.map(Get_FT_OU, file_to_set('links.txt'))
    # Where Get_FT_OU(link) appends links to a sqlite3 database.
我的真实代码如下所示:

from multiprocessing import Pool

with Pool(5) as p:
    p.map(parse_link, links)
with Pool(5) as p:
    p.map(Get_FT_OU, file_to_set('links.txt'))
    # Where Get_FT_OU(link) appends links to a sqlite3 database.
当代码运行时,我经常会遇到这些错误。有人能帮我修一下吗

    multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/Users/christian/Documents/GitHub/odds/CP_Parser.py", line 166, in Get_FT_OU
    cursor.execute(sql_str)
sqlite3.OperationalError: database is locked
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/christian/Documents/GitHub/odds/CP_Parser.py", line 206, in <module>
    p.map(Get_FT_OU, file_to_set('links.txt'))
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
sqlite3.OperationalError: database is locked
>>> 
multiprocessing.pool.RemoteTraceback:
"""
回溯(最近一次呼叫最后一次):
worker中的文件“/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/pool.py”,第119行
结果=(True,func(*args,**kwds))
mapstar中的文件“/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/pool.py”,第44行
返回列表(映射(*args))
文件“/Users/christian/Documents/GitHub/赔率/CP_Parser.py”,第166行,在Get_FT_OU中
cursor.execute(sql\u str)
sqlite3.0错误:数据库已锁定
"""
上述异常是以下异常的直接原因:
回溯(最近一次呼叫最后一次):
文件“/Users/christian/Documents/GitHub/赔率/CP_Parser.py”,第206行,在
p、 映射(获取文件到集合('links.txt'))
文件“/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/pool.py”,地图第266行
返回self.\u map\u async(func、iterable、mapstar、chunksize).get()
get中的文件“/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/pool.py”,第644行
提升自我价值
sqlite3.0错误:数据库已锁定
>>> 

我可以在不使用多处理的情况下很好地运行代码,实际上,在使用池(2)时,我没有得到错误,但是如果我走得更高,我会得到这些错误。我正在使用最新的MacBook Air。

它通过在连接中添加一个超时=10的方式来工作

conn = sqlite3.connect(DB_FILENAME, timeout=10)