Python 如何正确地从multiprocess pool.map获取输出并使用它？

简短介绍

对于学校课程，我们目前正在学习处理大型数据集和多处理的不同方法。 对于这个练习，我们必须解析一个大的fastq文件（2Gb），并每4行进行一次计算。然后，这些结果应打印或写入CSV文件（如果提供）

问题

我遇到的主要问题是从pool.map打印结果时出错（请参阅下面的代码）。第二个问题是效率。在我的process函数中，我打开/关闭输出文件，并为函数的每次调用编写代码。它目前可以工作，但我觉得应该有一种方法可以传递一个编写器或一个已经打开的文件，而不会出现任何问题

当前代码

import sys
import csv
from multiprocessing import Pool
from functools import partial

def process_qualities(qualities, outfile):
    """
    Calculates the phred values and saves these as lists for each read.
    When an outfile is provided, the base numbers and phred values will be written to it.
    """

    # Creating a list for the line to collect the phred scores
    phreds = []

    # If an output file has been provided, the file and writer are opened
    if outfile is not None:
        with open(outfile, "a", newline="") as out:
            out_writer = csv.writer(out)

            # Iterating over the base numbers and qualities
            for base_nr, quality in enumerate(qualities, start=1):
                phred = ord(quality)-33
                phreds.append(phred)

                # Writing to file
                out_writer.writerow([base_nr, phred])

    else:
        # Iterating over the base numbers and qualities
        for base_nr, quality in enumerate(qualities, start=1):
            phred = ord(quality)-33
            phreds.append(phred)

    return phreds


def main(argv=None):
    """Main function parsing the commandline and executing the functions"""

    # Opening the fastq file
    with open(filename, "r") as fastq:
        # Turning each line into a list
        fastq = fastq.read().split("\n")

    # Setting the outfile param to outfile for every call
    process_qualities_out = partial(process_qualities, outfile=outfile)

    # Creating lists with the phreds for each read
    with Pool(num_processes) as pool:
        
        # Calling the function on every third line, which contains the qualities
        phreds_lists = pool.map(process_qualities_out, [fastq[i] for i in range(3, len(fastq), 4)])

    print(type(phreds_lists))
    print(phreds_lists)

if __name__ == "__main__":
    sys.exit(main(sys.argv))

打印错误

<class 'list'>

Traceback (most recent call last):
  File test.py", line 115, in <module>
    sys.exit(main(sys.argv))
  File test.py", line 110, in main
    print(phreds_lists)
OSError: [Errno 22] Invalid argument

回溯（最近一次呼叫最后一次）：
文件test.py”，第115行，在
系统出口（主（系统argv））
文件test.py”，第110行，主
打印（短语列表）
OSError:[Errno 22]参数无效

结论

如果有人对如何解决这些问题有建议，或者有改进我的代码的一般编程技巧，请随时发表评论！非常感谢