具有多个事件的Python队列
我正试图编写一个小程序,通过HTTP接口发送和接收UDP通信以及接收命令。HTTP服务器位于一个具有多个事件的Python队列,python,multiprocessing,kqueue,Python,Multiprocessing,Kqueue,我正试图编写一个小程序,通过HTTP接口发送和接收UDP通信以及接收命令。HTTP服务器位于一个多处理进程中;UDP服务器位于另一个服务器中。这两个进程通过python多处理.Pipe进行通信。我在下面附上了完整的代码 我有两个相关问题: 如何使用python中的kqueue处理多个文件描述符/kevent(套接字文件描述符工作,管道文件描述符似乎不工作-不确定我使用的管道是否等效于一个文件) 如何区分这些Kevent,以便在读取管道和插座时应用不同的功能 我希望UDP服务器执行的操作的伪代
多处理进程中;UDP服务器位于另一个服务器中。这两个进程通过python多处理.Pipe进行通信。我在下面附上了完整的代码
我有两个相关问题:
- 如何使用python中的kqueue处理多个文件描述符/kevent(套接字文件描述符工作,管道文件描述符似乎不工作-不确定我使用的管道是否等效于一个文件)
- 如何区分这些Kevent,以便在读取管道和插座时应用不同的功能
我希望UDP服务器执行的操作的伪代码:
kq = new kqueue
udpEvent = kevent when socket read
pipeEvent = kevent when pipe read
while:
for event in kq.conrol([udpEvent, pipeEvent]):
if event == udpEvent:
# do something
elif event == pipeEvent:
print "HTTP command via pipe:", pipe.recv()
现在,UDP服务器识别套接字事件并正确读取套接字。但是,当我将pipekevent添加到kqueue时,程序会不停地吐出pipeevents。我将过滤器设置为已编写管道,但我假设1)这是错误的2)更具体地说,pythonmultiprocessing.pipe类似于常规unix管道,需要以不同的方式处理
.....
<select.kevent ident=4297866384 filter=-29216 flags=0x4000 fflags=0x1 data=0x16 udata=0x4000000000000>
<select.kevent ident=4297866384 filter=-29216 flags=0x4000 fflags=0x1 data=0x16 udata=0x4000000000000>
<select.kevent ident=4297866384 filter=-29216 flags=0x4000 fflags=0x1 data=0x16 udata=0x4000000000000>
<select.kevent ident=4297866384 filter=-29216 flags=0x4000 fflags=0x1 data=0x16 udata=0x4000000000000>
<select.kevent ident=4297866384 filter=-29216 flags=0x4000 fflags=0x1 data=0x16 udata=0x4000000000000>
<select.kevent ident=4297866384 filter=-29216 flags=0x4000 fflags=0x1 data=0x16 ^C<select.kevent ident=4297866384 filter=-29216 flags=0x4000 fflags=0x1 data=0x16 udata=0x4000000000000>
server.py(UDP)
handler.py(HTTP接口)
更新:
我用select重写了server listen方法。对于使用不超过3或4个FD的速度较慢的小型python原型,速度并不重要。Kqueue将是另一天的主题
def侦听(自身、IP地址、端口):
打印“开始!”
我知道这是一个老问题,但我可以给你一个我在多线程HTTP服务器上使用的kqueue套接字轮询的例子,我在阅读了C源代码和kqueue的手册页后发现了这个例子
#bsd socket polling
#I make all the relevant flags more C like to match the kqueue man pages
from select import kevent, kqueue
from select import KQ_EV_ADD as EV_ADD, KQ_EV_ONESHOT as EV_ONESHOT
from select import KQ_EV_EOF as EV_EOF
from .common import Client_Thread #a parent class who's implementation is irrelevant to the question, lol
class BSD_Client(Client_Thread):
def __init__(self, *args):
Client_Thread.__init__(self, *args)
#Make a kqueue object for the thread
kq = kqueue()
#Make a one-shot kev for this kqueue for when the kill socket is
#connected to. The connection is only made once, so why not tell
#that to our kqueue? The default filter is EVFILT_READ, so we don't
#need to specify that. The default flag is just EV_ADD.
kill_kev = kevent(self.kill_fd, flags=EV_ADD|EV_ONESHOT)
#using defaults for the client socket.
client_kev = kevent(self.client_sock)
#we only need to keep track of the kqueue's control func.
#This also makes things prettier in the run func.
self.control = kq.control
#now, we add thel list of events we just made to our kqueue.
#The first 0 means we want a list of at most 0 length in return.
#the second 0 means we want no timeout (i.e. do this in a
#non-blocking way.)
self.control([client_kev, kill_kev], 0, 0)
def run(self):
while True:
#Here we poll the kqueue object.
#The empty list means we are adding no new events to the kqueue.
#The one means we want a list of at most 1 element. Then None
#Means we want block until an event is triggered.
events = self.control([], 1, None)
#If we have an event, and the event is for the kill socket
#(meaning somebody made a connection to it), then we break the
#loop and die.
if events and events[0].ident == self.kill_fd:
self.die()
break
#If all that is left is an EOF in our socket, then we break
#the loop and die. Kqueues will keep returning a kevent
#that has been read once, even when they are empty.
if events and events[0].flags & EV_EOF:
self.die()
break
#Finally, if we have an event that isn't for the kill socket and
#does not have the EOF flag set, then there is work to do. If
#the handle client function (defined in the parent class) returns
#1, then we are done serving a page and we can die.
if events and self.handle_client():
self.die()
break
client.close()
self.die所做的就是将客户机ip:port字符串放入用于消息传递的队列中。另一个线程从队列中获取该字符串,打印一条消息并加入相关的线程对象。当然,我没有使用管道,只有插座。但我确实在kqueue的联机手册页上找到了这个
先进先出,管道
当有数据要读取时返回;数据包含的编号为
字节可用
当最后一个写入程序断开连接时,过滤器将在
旗帜。这可以通过进入EV_CLEAR来清除,此时
过滤器将继续等待数据变为可用,然后再重新启动
转向
因此,也许在udp服务器中,您可以在revents列表中循环,您应该按照手册页所说的做?实际上,你甚至不需要循环浏览一个最长的列表。也许你的监听功能应该是这样的
def listen(self, ip, port):
print "Starting!"
sock = socket.socket(AF_INET, SOCK_DGRAM)
sock.bind((ip, port))
kq = select.kqueue()
kev0 = select.kevent(sock)
kev1 = select.kevent(self.pipe)
kq.control([kev0, kev1], 0, 0)
while True: #this loop never breaks! so this whole function blocks forever like this
revents = kq.control([], 1, None)
if revents:
event = revents[0]
if event.flags & select.KQ_EV_EOF:
new_event = select.kevent(event.ident, flags=select.KQ_EV_CLEAR)
kq.control([new_event], 0, 0)
else:
print event
我真的建议像我这样导入标志和函数,这会使它更类似于基于C的手册页,你将不得不比较,我认为它看起来更漂亮。
我还想指出,我的类与您的类有点不同,因为每个新客户机都将获得该类的一个实例,并且每个实例都将在自己的线程中运行。这不是真正的答案,但是:您希望管道过滤器是“管道可读”,而不是“管道可写”。但是您还需要设置管道非阻塞并收集部分输入,我现在不知道如何做到这一点。另外,revents=kq.control([kevent0,kevent1],1,None)
味道不好:通常,您只需为每个fd添加一个控件(以及删除一个控件以删除fd,尽管它们在关闭时会自动清理),然后执行kcontrol(None,
)读取事件。此调用位于循环内,正在重新添加。但是,我也没有仔细研究过这些。是的,我试着找到了一些例子,但是没有看到python的任何东西,并且一天都不忍心进入C模式。所以我用select(参见更新)在3分钟内重写了它,它工作得很好,对于python原型来说已经足够好了。
import BaseHTTPServer
# Simple HTTP endpoints for controlling prototype Phantom implementation.
# The following commands are supported:
# 1. Open a connection via /open/[IP]:[PORT]
# 2. ????
class RequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
pipe = None
def __init__(self, pipe, *args):
RequestHandler.pipe = pipe
BaseHTTPServer.BaseHTTPRequestHandler.__init__(self, *args)
def do_HEAD(s):
s.send_response(200)
s.send_header("Content-type", "application/json")
s.end_headers()
def do_GET(s):
s.send_response(200)
s.send_header("Content-type", "application/json")
s.end_headers()
# Open connection command
if s.path.startswith('/open/'):
addrStr = s.path[6:len(s.path)]
(address, port) = tuple(filter(None, addrStr.split(':')))
port = int(port)
print "opening address: ", address, "port:", port
RequestHandler.pipe.send(['open', address, port])
def handleRequestsUsing(logic):
return lambda *args: RequestHandler(logic, *args)
# Initialize listening non-blocking UDP socket
sock = socket(AF_INET, SOCK_DGRAM)
sock.setblocking(0)
sock.bind((ipaddress, port))
inputs = [sock, self.pipe] # stuff we read
outputs = [] # stuff we expect to write
while inputs:
readable, writable, exceptional = select.select(inputs, outputs, inputs)
for event in readable:
if event is sock:
self.handleUDPData( sock.recvfrom(1024) )
if event is self.pipe:
print "pipe event", self.pipe.recv()
#bsd socket polling
#I make all the relevant flags more C like to match the kqueue man pages
from select import kevent, kqueue
from select import KQ_EV_ADD as EV_ADD, KQ_EV_ONESHOT as EV_ONESHOT
from select import KQ_EV_EOF as EV_EOF
from .common import Client_Thread #a parent class who's implementation is irrelevant to the question, lol
class BSD_Client(Client_Thread):
def __init__(self, *args):
Client_Thread.__init__(self, *args)
#Make a kqueue object for the thread
kq = kqueue()
#Make a one-shot kev for this kqueue for when the kill socket is
#connected to. The connection is only made once, so why not tell
#that to our kqueue? The default filter is EVFILT_READ, so we don't
#need to specify that. The default flag is just EV_ADD.
kill_kev = kevent(self.kill_fd, flags=EV_ADD|EV_ONESHOT)
#using defaults for the client socket.
client_kev = kevent(self.client_sock)
#we only need to keep track of the kqueue's control func.
#This also makes things prettier in the run func.
self.control = kq.control
#now, we add thel list of events we just made to our kqueue.
#The first 0 means we want a list of at most 0 length in return.
#the second 0 means we want no timeout (i.e. do this in a
#non-blocking way.)
self.control([client_kev, kill_kev], 0, 0)
def run(self):
while True:
#Here we poll the kqueue object.
#The empty list means we are adding no new events to the kqueue.
#The one means we want a list of at most 1 element. Then None
#Means we want block until an event is triggered.
events = self.control([], 1, None)
#If we have an event, and the event is for the kill socket
#(meaning somebody made a connection to it), then we break the
#loop and die.
if events and events[0].ident == self.kill_fd:
self.die()
break
#If all that is left is an EOF in our socket, then we break
#the loop and die. Kqueues will keep returning a kevent
#that has been read once, even when they are empty.
if events and events[0].flags & EV_EOF:
self.die()
break
#Finally, if we have an event that isn't for the kill socket and
#does not have the EOF flag set, then there is work to do. If
#the handle client function (defined in the parent class) returns
#1, then we are done serving a page and we can die.
if events and self.handle_client():
self.die()
break
client.close()
def listen(self, ip, port):
print "Starting!"
sock = socket.socket(AF_INET, SOCK_DGRAM)
sock.bind((ip, port))
kq = select.kqueue()
kev0 = select.kevent(sock)
kev1 = select.kevent(self.pipe)
kq.control([kev0, kev1], 0, 0)
while True: #this loop never breaks! so this whole function blocks forever like this
revents = kq.control([], 1, None)
if revents:
event = revents[0]
if event.flags & select.KQ_EV_EOF:
new_event = select.kevent(event.ident, flags=select.KQ_EV_CLEAR)
kq.control([new_event], 0, 0)
else:
print event