Python 芹菜/RabbitMQ/Django未运行任务
我希望有人能帮助我,因为我已经看到了堆栈溢出,无法找到解决我的问题的方法。我正在运行一个Django项目,并安装了Supervisor、RabbitMQ和芹菜。RabbitMQ已启动并正在运行,主管正在确保my celerybeat正在运行,但是,虽然它记录beat已启动并每5分钟发送一次任务(见下文),但这些任务从未实际执行: 我的主管程序配置:Python 芹菜/RabbitMQ/Django未运行任务,python,django,rabbitmq,celery,Python,Django,Rabbitmq,Celery,我希望有人能帮助我,因为我已经看到了堆栈溢出,无法找到解决我的问题的方法。我正在运行一个Django项目,并安装了Supervisor、RabbitMQ和芹菜。RabbitMQ已启动并正在运行,主管正在确保my celerybeat正在运行,但是,虽然它记录beat已启动并每5分钟发送一次任务(见下文),但这些任务从未实际执行: 我的主管程序配置: [program:nrv_twitter] ; Set full path to celery program if using virtualen
[program:nrv_twitter]
; Set full path to celery program if using virtualenv
command=/Users/tsantor/.virtualenvs/nrv_env/bin/celery beat -A app --loglevel=INFO --pidfile=/tmp/nrv-celerybeat.pid --schedule=/tmp/nrv-celerybeat-schedule
; Project dir
directory=/Users/tsantor/Projects/NRV/nrv
; Logs
stdout_logfile=/Users/tsantor/Projects/NRV/nrv/logs/celerybeat_twitter.log
redirect_stderr=true
autorestart=true
autostart=true
startsecs=10
user=tsantor
; if rabbitmq is supervised, set its priority higher so it starts first
priority=999
以下是上述程序的日志输出:
[2014-12-16 20:29:42,293: INFO/MainProcess] beat: Starting...
[2014-12-16 20:34:08,161: INFO/MainProcess] Scheduler: Sending due task gettweets-every-5-mins (twitter.tasks.get_tweets)
[2014-12-16 20:39:08,186: INFO/MainProcess] Scheduler: Sending due task gettweets-every-5-mins (twitter.tasks.get_tweets)
[2014-12-16 20:44:08,204: INFO/MainProcess] Scheduler: Sending due task gettweets-every-5-mins (twitter.tasks.get_tweets)
[2014-12-16 20:49:08,205: INFO/MainProcess] Scheduler: Sending due task gettweets-every-5-mins (twitter.tasks.get_tweets)
[2014-12-16 20:54:08,223: INFO/MainProcess] Scheduler: Sending due task gettweets-every-5-mins (twitter.tasks.get_tweets)
这是我的芹菜.py
设置文件:
from datetime import timedelta
BROKER_URL = 'amqp://guest:guest@localhost//'
CELERY_DISABLE_RATE_LIMITS = True
CELERYBEAT_SCHEDULE = {
'gettweets-every-5-mins': {
'task': 'twitter.tasks.get_tweets',
'schedule': timedelta(seconds=300) # 300 = every 5 minutes
},
}
这是我的celeryapp.py
:
from __future__ import absolute_import
import os
from django.conf import settings
from celery import Celery
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'app.settings')
app = Celery('app')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
from __future__ import absolute_import
import logging
from celery import shared_task
from twitter.views import IngestTweets
log = logging.getLogger('custom.log')
@shared_task
def get_tweets():
"""
Get tweets and save them to the DB
"""
instance = IngestTweets()
IngestTweets.get_new_tweets(instance)
log.info('Successfully ingested tweets via celery task')
return True
这是我的twitter/tasks.py
:
from __future__ import absolute_import
import os
from django.conf import settings
from celery import Celery
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'app.settings')
app = Celery('app')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
from __future__ import absolute_import
import logging
from celery import shared_task
from twitter.views import IngestTweets
log = logging.getLogger('custom.log')
@shared_task
def get_tweets():
"""
Get tweets and save them to the DB
"""
instance = IngestTweets()
IngestTweets.get_new_tweets(instance)
log.info('Successfully ingested tweets via celery task')
return True
get_tweets
方法从未执行过,但我知道它可以正常工作,因为我可以手动执行get_tweets
,而且工作正常
我花了两天时间试图弄明白为什么它发送了到期的任务,但没有执行它们?非常感谢您的帮助。提前感谢。用户2097159感谢您为我指明了正确的方向,我不知道我还必须使用主管运行工人。我原以为这是一个工人或一个节拍,但现在我明白了,我必须有一个工人来处理这项任务,一个节拍来定期启动这项任务 下面是主管缺少的工作人员配置:
[program:nrv_celery_worker]
; Worker
command=/Users/tsantor/.virtualenvs/nrv_env/bin/celery worker -A app --loglevel=INFO
; Project dir
directory=/Users/tsantor/Projects/NRV/nrv
; Logs
stdout_logfile=/Users/tsantor/Projects/NRV/nrv/logs/celery_worker.log
redirect_stderr=true
autostart=true
autorestart=true
startsecs=10
user=tsantor
numprocs=1
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
然后重置RabbitMQ队列。现在我已经通过主管管理了beat和worker项目,所有项目都按预期运行。希望这对其他人有所帮助。您需要同时启动一个工作进程和一个beat进程。您可以创建单独的流程,也可以创建一个包含辅助工和节拍的单个流程。这在开发过程中更方便(但不建议用于生产) 发件人: 您还可以通过启用workers-B选项将beat嵌入到worker中,如果您永远不会运行多个worker节点,这会很方便,但它并不常用,因此不建议在生产中使用: $芹菜-项目工人-B
有关主管配置文件中的表达式,请参见(链接自)是否有芹菜工人在运行?是的,正如我提到的,我有一个芹菜节拍,通过主管进行管理。从上面的日志中可以看到,它每5分钟运行一次。我在上面添加了更多信息以了解更多内容。等等,你是说我不仅需要与主管一起运行芹菜节拍流程,还需要添加另一个主管程序以确保芹菜工人也在运行吗?