为什么我的多处理Python脚本在Linux上运行,但在Windows上遇到与pickle相关的异常?
我有一种感觉,这与Windows中没有fork()有关,但我真的不确定该怎么做 我有一个名为workabj的类,它从多处理继承了进程。在Windows上,创建此类的新实例进展顺利,但调用实例的start()方法会引发以下异常:为什么我的多处理Python脚本在Linux上运行,但在Windows上遇到与pickle相关的异常?,linux,windows,pickle,python-multiprocessing,python-sockets,Linux,Windows,Pickle,Python Multiprocessing,Python Sockets,我有一种感觉,这与Windows中没有fork()有关,但我真的不确定该怎么做 我有一个名为workabj的类,它从多处理继承了进程。在Windows上,创建此类的新实例进展顺利,但调用实例的start()方法会引发以下异常: Traceback (most recent call last): File "<string>", line 1, in <module> File "C:\Python27\lib\multiprocessing\forking.py
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Python27\lib\multiprocessing\forking.py", line 381, in main
self = load(from_parent)
File "C:\Python27\lib\pickle.py", line 1384, in load
return Unpickler(file).load()
File "C:\Python27\lib\pickle.py", line 864, in load
dispatch[key](self)
File "C:\Python27\lib\pickle.py", line 1089, in load_newobj
obj = cls.__new__(cls, *args)
TypeError: __new__() takes at least 2 arguments (1 given)
您在主进程中创建的
workabj
对象需要复制到子进程中。在Linux上,可以使用fork
,使子进程成为父进程的完整副本。Windows上没有fork
系统调用,因此多进程
模块必须使用pickle将workbj
对象复制到子进程中。有关详细信息,请参阅多进程
模块文档:您已经回答了我的问题,谢谢。我现在确切地理解了问题所在,事实上,我在过去使用jsonpickle库解决了这个问题——只是没有在多处理的情况下!当Windows创建新进程时,它没有工作对象的副本,并且因为工作对象是自定义类,所以不能对其进行pickle。我想我可以通过把它变成一个函数来解决这个问题……自定义类可以被pickle。问题可能是您在workabj.\uuuu init\uuu
中设置的一个或多个实例成员无法被pickle。您可以尝试将套接字创建代码以及tprint
的初始化移动到run
方法。
class workObj(Process):
def __init__(self, name, ip, port, keyFile, *args, **kwargs):
Process.__init__(self)
global printer
self.tprint = printer.tprint # We'll hold a reference to printer internally.
self.workCounter = 0
self.ip = ip
self.port = port
self.conn = socket.socket()
self.workDone = False
self.connected = True
self.conn.settimeout(10)
self.secconn = ssl.wrap_socket(self.conn, ca_certs=keyFile, cert_reqs=ssl.CERT_REQUIRED)
self.name = name
try:
self.secconn.connect((self.ip, self.port))
except:
printer.tprint(self.name, "Fatal error in constructor: workObj {}: connect to {}:{} failed! WARNING! Object is unusable!".format(self.name, self.ip, self.port), error= True)
self.workDone = True
self.connected = False
# raise # Come back later and check this syntax.
def getWorkCount(self):
return self.workCounter
def isConnected(self):
return self.connected
def finishUp(self):
self.workDone = True
def run(self):
s = self.secconn
tprint = self.tprint
try:
while not self.workDone:
workStr = s.recv(128)
if workStr == "":
self.workDone = True
tprint(self.name, "Got blank job from server.")
break
sendStr = doWork(workStr)
s.send(sendStr)
#s.send(doWork(s.recv(128)))
self.workCounter += 1
except socket.timeout as e:
tprint(self.name, "Timeout getting or sending work. Assuming server broke.", error= True)
tprint(self.name, str(e), error=True)
self.workDone = True
except KeyboardInterrupt as e:
tprint(self.name, "Caught keyboard interrupt, sending server blank job.", error=True)
s.send("")
self.workDone = True
except:
# Literally anything else?
import sys
e = sys.exc_info()[:]
tprint(self.name, "workObj {} broke.".format(self.name), error=True)
tprint(self.name, "{}\n{}".format(e[1], e[2]), error=True)
self.workDone = True
#raise
s.close()
self.connected = False