Python 非阻塞方式的Kombu
我正在寻找一种在tornado sockjs和Django应用服务器之间使用kombu作为MQ适配器的方法。我做了这样的事情:Python 非阻塞方式的Kombu,python,tornado,kombu,Python,Tornado,Kombu,我正在寻找一种在tornado sockjs和Django应用服务器之间使用kombu作为MQ适配器的方法。我做了这样的事情: class BrokerClient(ConsumerMixin): clients = [] def __init__(self): self.connection = BrokerConnection(settings.BROKER_URL) self.io_loop = ioloop.IOLoop.instance
class BrokerClient(ConsumerMixin):
clients = []
def __init__(self):
self.connection = BrokerConnection(settings.BROKER_URL)
self.io_loop = ioloop.IOLoop.instance()
self.queue = sockjs_queue
self._handle_loop()
@staticmethod
def instance():
if not hasattr(BrokerClient, '_instance'):
BrokerClient._instance = BrokerClient()
return BrokerClient._instance
def add_client(self, client):
self.clients.append(client)
def remove_client(self, client):
self.clients.remove(client)
def _handle_loop(self):
try:
if self.restart_limit.can_consume(1):
for _ in self.consume(limit=5):
pass
except self.connection.connection_errors:
print ('Connection to broker lost. '
'Trying to re-establish the connection...')
self.io_loop.add_timeout(datetime.timedelta(0.0001), self._handle_loop)
def get_consumers(self, Consumer, channel):
return [Consumer([self.queue, ], callbacks=[self.process_task])]
def process_task(self, body, message):
for client in self.clients:
if hasattr(body, 'users') and client.user.pk in body.users:
client.send(body)
message.ack()
但tornado在循环执行时受阻(如预期)
有没有办法防止这种情况
我知道Tornado的Pika库适配器,但我想使用kombu,因为它已经在项目中使用,并且具有灵活的传输
更新:
将“句柄”回路更改为“发电机”功能
def drain_events(self, callback):
with self.Consumer() as (connection, channel, consumers):
with self.extra_context(connection, channel):
try:
connection.drain_events(timeout=1)
except:
pass
callback(None)
@tornado.gen.engine
def _handle_loop(self):
response = yield tornado.gen.Task(self.drain_events)
self.io_loop.add_timeout(datetime.timedelta(0.0001), self._handle_loop)
最后,我找到了RabbitMQ后端的正确解决方案:
class BrokerClient(object):
clients = []
@staticmethod
def instance():
if not hasattr(BrokerClient, '_instance'):
BrokerClient._instance = BrokerClient()
return BrokerClient._instance
def __init__(self):
self.connection = BrokerConnection(settings.BROKER_URL)
self.consumer = Consumer(self.connection.channel(), [queue, ], callbacks=[self.process_task])
self.consumer.consume()
io_loop = tornado.ioloop.IOLoop.instance()
for sock, handler in self.connection.eventmap.items():
def drain_nowait(fd, events):
handler()
io_loop.add_handler(sock.fileno(), drain_nowait, l.READ | l.ERROR)
def process_task(self, body, message):
#something
message.ack()
def add_client(self, client):
self.clients.append(client)
def remove_client(self, client):
self.clients.remove(client)
对于其他后端,您可以使用问题中发布的解决方案
注意:不适用于librabbitmq我也有类似的需要,在Kombu/RabbitMQ和ZeroMQ之间以非阻塞方式切换。解决方案是使用Gevent对套接字库进行猴子补丁,这样Kombu也将成为非阻塞的。我的“主”线程运行Kombu drain_events回调,在另一个gevent线程中,我有一个从ZeroMQ套接字接收消息的循环。效果很好
这也不适用于librabbitmq,因为它在C中使用自己的套接字,不受Gevent的影响。注意:在撰写本文时,最新的py amqp和kombu不正式支持异步读取。但是,对于异步消耗,存在一些限制