Ubuntu 西芹温关机

Ubuntu 西芹温关机,ubuntu,redis,celery,supervisord,Ubuntu,Redis,Celery,Supervisord,每当我运行一个运行监视器和芹菜的Ubuntu实例上的“SUDO关机-H”时,芹菜就好像在处理任务的中间关闭,当看到尾部-F时,我不知道当你在前台运行芹菜时,键入CTRL C时通常会看到的“热关机”消息。p> 这对我来说是一个大问题,因为服务器随时可能被终止,所以等待芹菜完成其任务是至关重要的。这可能需要几秒钟、10分钟或10小时,但关机信号应该总是让芹菜完成它的工作。 我在celeryd配置文件中添加了stopasgroup和killasgroup,这看起来确实像是从supervisor停止芹菜

每当我运行一个运行监视器和芹菜的Ubuntu实例上的“SUDO关机-H”时,芹菜就好像在处理任务的中间关闭,当看到尾部-F时,我不知道当你在前台运行芹菜时,键入CTRL C时通常会看到的“热关机”消息。p> 这对我来说是一个大问题,因为服务器随时可能被终止,所以等待芹菜完成其任务是至关重要的。这可能需要几秒钟、10分钟或10小时,但关机信号应该总是让芹菜完成它的工作。 我在celeryd配置文件中添加了stopasgroup和killasgroup,这看起来确实像是从supervisor停止芹菜停止了所有子进程,但不知何故它似乎忽略了“sudo shutdown-h now”发送的关闭信号

有人知道为什么芹菜没有表现出正常的“热关机”吗

下面是一个worker conf的示例:

[program:celery-worker]
command=/path/to/script-running-celery.sh
directory=/path/to/celery/project/tasks
user=myuser
stdout_logfile=/var/log/celery/worker.log
stderr_logfile=/var/log/celery/worker.log
autostart=true
autorestart=true
startsecs=10
stopwaitsecs = 60
killasgroup=true
stopasgroup=true
priority=50
下面是正在运行的实际shell脚本:

#!/bin/bash

export PATH="$PATH:/mylocal-envs/localenv";
cd /path/to/celery/project/tasks;
exec /mylocal-envs/localenv/bin/celery worker -c8 -n my_worker_%h_$(uuid) --without-gossip -A tasks --queues=myqueue --loglevel=INFO -Ofair
您能理解为什么shutdown命令不让芹菜在热关机状态下运行并结束其当前任务吗

编辑:请注意,当在不使用Supervisor的情况下直接运行脚本并从另一个ssh窗口关闭时,我仍然会得到相同的行为:工作进程正在处理任务,然后接收到关闭信号并关闭ssh会话。下面是在运行脚本时从worker前台发生的情况:

[2014-05-17 19:29:02,955: INFO/MainProcess] Task tasks.a_task[5143a89d-e077-4145-ba94-0932d0f81d9e] succeeded in 15.523131618s: u''
[2014-05-17 19:29:03,621: INFO/MainProcess] Received task: tasks.a_task[b61a30ba-2fbb-475a-9cb2-ec88d20a8885]

Broadcast message from jdoe@10.0.0.12
    (/dev/pts/1) at 19:29 ...

The system is going down for halt NOW!
[2014-05-17 19:29:06,846: INFO/MainProcess] Received task: tasks.a_task[9146f9f4-f06c-4563-b8d1-43c8d9367d9b]
[2014-05-17 19:29:08,089: INFO/MainProcess] Task tasks.a_task[b4c7dacc-3f6d-42e5-ab67-f33748f0af14] succeeded in 16.49714411s: u''
Connection to xx.xx.xxx.xx closed by remote host.
Connection to xx.xx.xxx.xx closed.
您仍然可以看到一个任务成功,另一个任务被接收。看着flower,我看到了一堆已经开始的任务,现在被困在“已开始”或“已接收”中

编辑:在尝试直接在command=选项中运行芹菜而不是运行shell脚本之后,我注意到了相同的行为。如果我在前台通过ssh运行脚本(芹菜处于调试模式),在点击ctrl-C时会得到以下结果:

worker: Hitting Ctrl+C again will terminate all running tasks!
289
290worker: Warm shutdown (MainProcess)
291[2014-05-18 13:12:49,557: DEBUG/MainProcess] | Worker: Closing Hub...
292[2014-05-18 13:12:49,557: DEBUG/MainProcess] | Worker: Closing Pool...
293[2014-05-18 13:12:49,557: DEBUG/MainProcess] | Worker: Closing Consumer...
294[2014-05-18 13:12:49,558: DEBUG/MainProcess] | Worker: Stopping Consumer...
295[2014-05-18 13:12:49,558: DEBUG/MainProcess] | Consumer: Closing Connection...
296[2014-05-18 13:12:49,558: DEBUG/MainProcess] | Consumer: Closing Events...
297[2014-05-18 13:12:49,558: DEBUG/MainProcess] | Consumer: Closing Mingle...
298[2014-05-18 13:12:49,558: DEBUG/MainProcess] | Consumer: Closing Tasks...
299[2014-05-18 13:12:49,559: DEBUG/MainProcess] | Consumer: Closing Control...
300[2014-05-18 13:12:49,559: DEBUG/MainProcess] | Consumer: Closing Heart...
301[2014-05-18 13:12:49,559: DEBUG/MainProcess] | Consumer: Closing event loop...
302[2014-05-18 13:12:49,559: DEBUG/MainProcess] | Consumer: Stopping event loop...
303[2014-05-18 13:12:49,559: DEBUG/MainProcess] | Consumer: Stopping Heart...
304[2014-05-18 13:12:49,564: DEBUG/MainProcess] | Consumer: Stopping Control...
305[2014-05-18 13:12:49,570: DEBUG/MainProcess] | Consumer: Stopping Tasks...
306[2014-05-18 13:12:49,570: DEBUG/MainProcess] Cancelling task consumer...
307[2014-05-18 13:12:49,570: DEBUG/MainProcess] | Consumer: Stopping Mingle...
308[2014-05-18 13:12:49,570: DEBUG/MainProcess] | Consumer: Stopping Events...
309[2014-05-18 13:12:49,571: DEBUG/MainProcess] | Consumer: Stopping Connection...
310[2014-05-18 13:12:49,571: DEBUG/MainProcess] | Worker: Stopping Pool...
311[2014-05-18 13:12:50,608: DEBUG/MainProcess] | Worker: Stopping Hub...
312[2014-05-18 13:12:50,612: DEBUG/MainProcess] | Consumer: Shutdown Heart...
313[2014-05-18 13:12:50,612: DEBUG/MainProcess] | Consumer: Shutdown Control...
314[2014-05-18 13:12:50,613: DEBUG/MainProcess] | Consumer: Shutdown Tasks...
315[2014-05-18 13:12:50,614: DEBUG/MainProcess] Cancelling task consumer...
316[2014-05-18 13:12:50,614: DEBUG/MainProcess] Closing consumer channel...
317[2014-05-18 13:12:50,615: DEBUG/MainProcess] | Consumer: Shutdown Events...
318[2014-05-18 13:12:50,616: DEBUG/MainProcess] | Consumer: Shutdown Connection...
319[2014-05-18 13:12:50,617: DEBUG/MainProcess] removing tasks from inqueue until task handler finished
这看起来像是正常的关机过程,我希望看到它发生在主管身上。 另一方面,停止使用supervisor给了我以下信息:

[2014-05-18 13:14:28,369: DEBUG/MainProcess] pidbox received method ping() [reply_to:{u'routing_key': u'1bcaf3a6-0c77-33b9-9d46-56e8634e5b1d', u'exchange': u'reply.celery.pidbox'} ticket:161fcbdb-dac0-48c9-ac13-ab7a75bfdc85]

没有一个关闭跟踪像您在按下ctrl-c键时看到的那样。它只是停止。

这似乎是Ubuntu(3.0a.)附带的Supervisor版本中的一个错误,在切换到当前稳定的Supervisor版本(手动安装在/usr/local/bin中)之后,我可以看到确实接收到sigterm的日志。

看起来我应该以某种方式将术语信号从shell脚本传递到前台运行的芹菜进程中,并使用exec。不知何故,主管管理shell脚本,而不是芹菜工人。