如何正确利用Python中的多处理模块?
我有110个PDF文件,我正试图从中提取图像。一旦图像被提取出来,我想删除所有重复的图像并删除小于4KB的图像。我的代码如下所示:如何正确利用Python中的多处理模块?,python,multithreading,parallel-processing,multiprocessing,Python,Multithreading,Parallel Processing,Multiprocessing,我有110个PDF文件,我正试图从中提取图像。一旦图像被提取出来,我想删除所有重复的图像并删除小于4KB的图像。我的代码如下所示: def extract_images_from_file(pdf_file): file_name = os.path.splitext(os.path.basename(pdf_file))[0] call(["pdfimages", "-png", pdf_file, file_name]) os.remove(pdf_file) de
def extract_images_from_file(pdf_file):
file_name = os.path.splitext(os.path.basename(pdf_file))[0]
call(["pdfimages", "-png", pdf_file, file_name])
os.remove(pdf_file)
def dedup_images():
os.mkdir("unique_images")
md5_library = []
images = glob("*.png")
print "Deleting images smaller than 4KB and generating the MD5 hash values for all other images..."
for image in images:
if os.path.getsize(image) <= 4000:
os.remove(image)
else:
m = md5.new()
image_data = list(Image.open(image).getdata())
image_string = "".join(["".join([str(tpl[0]), str(tpl[1]), str(tpl[2])]) for tpl in image_data])
m.update(image_string)
md5_library.append([image, m.digest()])
headers = ['image_file', 'md5']
dat = pd.DataFrame(md5_library, columns=headers).sort(['md5'])
dat.drop_duplicates(subset="md5", inplace=True)
print "Extracting the unique images."
unique_images = dat.image_file.tolist()
for image in unique_images:
old_file = image
new_file = "unique_images\\" + image
shutil.copy(old_file, new_file)
I/O Error: Couldn't open image If/iOl eE r'rSourb:p oICe/onOua l EdNrner'wot r Y:oo prCekon u Cliodmunan'gttey of1pi0e
l2ne1 1i'4mS auogbiepl o2fefinrlaee e N@'egSwmu abYipolor ekcn oaCm o Nupentwt y1Y -o18r16k11 8.C1po4nu gn3't4
y7 5160120821143 3p4t7I 9/49O-8 88E78r81r.3op rnp:gt ' C
3o-u3l6d0n.'ptn go'p
en image file 'Ia/ ON eEwr rYoorr:k CCIoo/uuOln dtEnyr' rt1o 0ro2:p1 e1Cn4o uiolmidalng2'eft r m '
ai gpceoo emfn iapl teN e1'w-S 8uY6bo2pr.okpe nnCgao' u
Nnetwy Y1o0r2k8 1C4o u3n4t7y9 918181881134 3p4t7 536-1306211.3p npgt'
4-879.png'
I/O Error: CoulId/nO' tE rorpoern: iCmoaugled nf'itl eo p'eub piomeangae fNielwe Y'oSrukb pCooeunnat yN e1w0 2Y8o1r
4k 3C4o7u9n9t8y8 811032 1p1t4 3o-i3l622f pt 1-863.png'
I/O Error: Couldn't open image file 'pt 1-864.png'
I/O Error: Couldn't open image file 'pt 1-865.png'
I/O Error: Couldn't open image file 'pt 1-866.png'
I/O Error: Couldn't open image file 'pt 1-867.png'
在提取图像时,一切似乎都很好,但随后一切都失控了。下面是我的问题:
1) 我是否正确设置了并行过程?2) 它是否继续尝试在
重复数据消除\u images()
上使用所有8个处理器?3) 是否有任何我遗漏和/或做得不正确的地方 提前谢谢 编辑以下是我所说的“乱七八糟”的意思。错误开始时有一大堆行,如下所示:
def extract_images_from_file(pdf_file):
file_name = os.path.splitext(os.path.basename(pdf_file))[0]
call(["pdfimages", "-png", pdf_file, file_name])
os.remove(pdf_file)
def dedup_images():
os.mkdir("unique_images")
md5_library = []
images = glob("*.png")
print "Deleting images smaller than 4KB and generating the MD5 hash values for all other images..."
for image in images:
if os.path.getsize(image) <= 4000:
os.remove(image)
else:
m = md5.new()
image_data = list(Image.open(image).getdata())
image_string = "".join(["".join([str(tpl[0]), str(tpl[1]), str(tpl[2])]) for tpl in image_data])
m.update(image_string)
md5_library.append([image, m.digest()])
headers = ['image_file', 'md5']
dat = pd.DataFrame(md5_library, columns=headers).sort(['md5'])
dat.drop_duplicates(subset="md5", inplace=True)
print "Extracting the unique images."
unique_images = dat.image_file.tolist()
for image in unique_images:
old_file = image
new_file = "unique_images\\" + image
shutil.copy(old_file, new_file)
I/O Error: Couldn't open image If/iOl eE r'rSourb:p oICe/onOua l EdNrner'wot r Y:oo prCekon u Cliodmunan'gttey of1pi0e
l2ne1 1i'4mS auogbiepl o2fefinrlaee e N@'egSwmu abYipolor ekcn oaCm o Nupentwt y1Y -o18r16k11 8.C1po4nu gn3't4
y7 5160120821143 3p4t7I 9/49O-8 88E78r81r.3op rnp:gt ' C
3o-u3l6d0n.'ptn go'p
en image file 'Ia/ ON eEwr rYoorr:k CCIoo/uuOln dtEnyr' rt1o 0ro2:p1 e1Cn4o uiolmidalng2'eft r m '
ai gpceoo emfn iapl teN e1'w-S 8uY6bo2pr.okpe nnCgao' u
Nnetwy Y1o0r2k8 1C4o u3n4t7y9 918181881134 3p4t7 536-1306211.3p npgt'
4-879.png'
I/O Error: CoulId/nO' tE rorpoern: iCmoaugled nf'itl eo p'eub piomeangae fNielwe Y'oSrukb pCooeunnat yN e1w0 2Y8o1r
4k 3C4o7u9n9t8y8 811032 1p1t4 3o-i3l622f pt 1-863.png'
I/O Error: Couldn't open image file 'pt 1-864.png'
I/O Error: Couldn't open image file 'pt 1-865.png'
I/O Error: Couldn't open image file 'pt 1-866.png'
I/O Error: Couldn't open image file 'pt 1-867.png'
然后使用以下多行代码更具可读性:
def extract_images_from_file(pdf_file):
file_name = os.path.splitext(os.path.basename(pdf_file))[0]
call(["pdfimages", "-png", pdf_file, file_name])
os.remove(pdf_file)
def dedup_images():
os.mkdir("unique_images")
md5_library = []
images = glob("*.png")
print "Deleting images smaller than 4KB and generating the MD5 hash values for all other images..."
for image in images:
if os.path.getsize(image) <= 4000:
os.remove(image)
else:
m = md5.new()
image_data = list(Image.open(image).getdata())
image_string = "".join(["".join([str(tpl[0]), str(tpl[1]), str(tpl[2])]) for tpl in image_data])
m.update(image_string)
md5_library.append([image, m.digest()])
headers = ['image_file', 'md5']
dat = pd.DataFrame(md5_library, columns=headers).sort(['md5'])
dat.drop_duplicates(subset="md5", inplace=True)
print "Extracting the unique images."
unique_images = dat.image_file.tolist()
for image in unique_images:
old_file = image
new_file = "unique_images\\" + image
shutil.copy(old_file, new_file)
I/O Error: Couldn't open image If/iOl eE r'rSourb:p oICe/onOua l EdNrner'wot r Y:oo prCekon u Cliodmunan'gttey of1pi0e
l2ne1 1i'4mS auogbiepl o2fefinrlaee e N@'egSwmu abYipolor ekcn oaCm o Nupentwt y1Y -o18r16k11 8.C1po4nu gn3't4
y7 5160120821143 3p4t7I 9/49O-8 88E78r81r.3op rnp:gt ' C
3o-u3l6d0n.'ptn go'p
en image file 'Ia/ ON eEwr rYoorr:k CCIoo/uuOln dtEnyr' rt1o 0ro2:p1 e1Cn4o uiolmidalng2'eft r m '
ai gpceoo emfn iapl teN e1'w-S 8uY6bo2pr.okpe nnCgao' u
Nnetwy Y1o0r2k8 1C4o u3n4t7y9 918181881134 3p4t7 536-1306211.3p npgt'
4-879.png'
I/O Error: CoulId/nO' tE rorpoern: iCmoaugled nf'itl eo p'eub piomeangae fNielwe Y'oSrukb pCooeunnat yN e1w0 2Y8o1r
4k 3C4o7u9n9t8y8 811032 1p1t4 3o-i3l622f pt 1-863.png'
I/O Error: Couldn't open image file 'pt 1-864.png'
I/O Error: Couldn't open image file 'pt 1-865.png'
I/O Error: Couldn't open image file 'pt 1-866.png'
I/O Error: Couldn't open image file 'pt 1-867.png'
这会重复一段时间,在乱码错误文本和可读文本之间来回移动
最后,这里是:
Deleting images smaller than 4KB and generating the MD5 hash values for all other images...
Extracting unique images into a new folder...
这意味着代码将恢复并继续处理该过程。出了什么问题
从\u文件()中提取\u图像的主体部分。
。另外,我要指出,您使用的是8个进程,而不是处理器。如果您碰巧有一个8核Intel CPU,并且启用了超线程,您将能够同时运行16个进程从\u文件中提取\u图像()
引发异常,它会破坏整个池,这可能不是您想要的。为了防止这种情况发生,你可以试着绕过那个街区
你正在处理的“乱七八糟”的性质是什么?我们能看到异常文本吗?您的代码基本上没有问题 乱码文本是所有试图将不同版本的
I/O错误
消息交错写入控制台的进程。错误消息是由pdfimages
命令生成的,可能是因为当您同时运行两个文件时,它们会发生冲突,可能是因为临时文件,或者两者都使用相同的文件名或类似的名称
尝试为每个单独的pdf文件使用不同的图像根。我觉得这样可以。你能更具体地解释一下“失控”吗?@strubbly我已经在上面添加了错误输出。“我已经开始涉足多线程。请随意解释为我说我不知道我在做什么。”你和其他所有开始并发工作的人。我接受了这个答案,因为它有效地解决了我遇到的问题。我在根名称中添加了一个随机的3位字母数字代码,这完全缓解了任何问题。谢谢酷-您在多处理方面做得很好-请记住,您调用的东西需要能够一起运行。当它们共享目录或文件等资源时,可能会发生冲突。