Multiprocessing Python 3.4多进程抛出TypeError（“无法序列化'；”io.BufferedReader'；对象”

最近，我用Python3.4编写了一个多进程代码来下载一些图像，它一开始运行得非常快，然后我出现了以下错误，无法再启动程序

Traceback (most recent call last):
  File "multiprocessing_d.py", line 23, in <module>
    main()
  File "multiprocessing_d.py", line 16, in main
    p.map(download, lines)
  File "/usr/local/lib/python3.4/multiprocessing/pool.py", line 260, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/local/lib/python3.4/multiprocessing/pool.py", line 608, in get
    raise self._value
multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x7f1e047f32e8>'. Reason: 'TypeError("cannot serialize '_io.BufferedReader' object",)'

多处理

        from functools import partial
        from multiprocessing.pool import Pool
        import sys
        from time import time
        from download_helper import setup_download_dir, download_link

        def main():
            file_path = sys.argv[1]
            dic_path = sys.argv[2]
            download_dir = setup_download_dir(dic_path)
            download = partial(download_link, download_dir)
            with open(file_path, 'rb') as f:
                lines = f.readlines()
                ts = time()
                p = Pool(processes=16, maxtasksperchild=1)
                p.map(download, lines)
                p.close()
                p.join()
                print('Took {}s'.format(time() - ts))
                f.close()

        if __name__ == "__main__":
            main()

我试着在网上搜索，但没有找到多少有用的信息。我怀疑urlretrieve中可能出现了一些异常，但我不知道如何调试它。如有任何意见或建议，将不胜感激

---

James

问题已解决，有一些链接已断开，urlretrieve将返回HTTPError，无法序列化。添加异常处理程序可以解决问题您添加了什么异常处理程序？@JamesGuo您的评论可以被接受为正确答案，至少对我来说是这样。谢谢。可能有两份

        from functools import partial
        from multiprocessing.pool import Pool
        import sys
        from time import time
        from download_helper import setup_download_dir, download_link

        def main():
            file_path = sys.argv[1]
            dic_path = sys.argv[2]
            download_dir = setup_download_dir(dic_path)
            download = partial(download_link, download_dir)
            with open(file_path, 'rb') as f:
                lines = f.readlines()
                ts = time()
                p = Pool(processes=16, maxtasksperchild=1)
                p.map(download, lines)
                p.close()
                p.join()
                print('Took {}s'.format(time() - ts))
                f.close()

        if __name__ == "__main__":
            main()