Python中的并行程序不产生输出
我有一个简单的任务。需要为大量文件运行特定函数。这个任务可以很容易地并行化 以下是工作代码:Python中的并行程序不产生输出,python,python-3.x,python-2.7,multiprocessing,python-multiprocessing,Python,Python 3.x,Python 2.7,Multiprocessing,Python Multiprocessing,我有一个简单的任务。需要为大量文件运行特定函数。这个任务可以很容易地并行化 以下是工作代码: # filelist is the directory containing two file, a.txt and b.txt. # a.txt is the first file, b.xt is the second file # I pass a file that lits the names of the two files to the main program from concurr
# filelist is the directory containing two file, a.txt and b.txt.
# a.txt is the first file, b.xt is the second file
# I pass a file that lits the names of the two files to the main program
from concurrent.futures import ProcessPoolExecutor, as_completed
from pathlib import Path
import sys
def translate(filename):
print(filename)
f = open(filename, "r")
g = open(filename + ".x", , "w")
for line in f:
g.write(line)
def main(path_to_file_with_list):
futures = []
with ProcessPoolExecutor(max_workers=8) as executor:
for filename in Path(path_to_file_with_list).open():
executor.submit(translate, "filelist/" + filename)
for future in as_completed(futures):
future.result()
if __name__ == "__main__":
main(sys.argv[1])
但是,不会创建新文件,即该文件夹不包含a.txt.x和b.txt.x文件
上面的代码有什么问题,我怎样才能使它工作
谢谢。这会让你走上正确的道路。如果它不工作并且不是一个明显的bug,那么我怀疑您可能没有正确的所有文件路径。。。我应该指出,从减少的开销中,线程比进程更有利于编写文件。文件I/O应该释放GIL,这样您就可以从加速中获益(如果您一次复制多行,则会受益匪浅)。也就是说,如果您只是复制文件,您应该真正使用
shutil.copy
或shutil.copy2
from concurrent.futures import ProcessPoolExecutor, wait
from pathlib import Path
import sys
def translate(filename):
print(filename)
with open(filename, "r") as f, open(filename + ".x", , "w") as g:
for line in f:
g.write(line)
def main(path_to_file_with_list):
futures = []
with ProcessPoolExecutor(max_workers=8) as executor:
for filename in Path(path_to_file_with_list).open():
futures.append(executor.submit(translate, "filelist/" + filename))
wait(futures) #simplify waiting on processes if you don't need the result.
for future in futures:
if future.excpetion() is not None:
raise future.exception() #ProcessPoolEcecutors swallow exceptions without telling you...
print("done")
if __name__ == "__main__":
main(sys.argv[1])
您的程序包含打印语句。您是否得到任何输出?对文件进行迭代会得到文件的行,包括行尾,因此您试图打开名称中包含新行的文件,而这些新行实际上不太可能存在。此外,您也不会关闭这些文件。如果不关闭“g”,则可能无法获得任何输出。如果使用“with”语句打开f和g,它们将始终在块的末尾关闭。
futures
在main
@PaulCornelius的第二个循环中仍然是一个空列表是的,print语句工作正常