Python 任务中的芹菜任务
我将dropbox与芹菜结合在我的应用程序中,这样我就允许用户在连接dropbox的情况下存储自己的照片 我已经写了一段代码,但我担心这可能会导致无限循环,从而杀死系统 我正在使用的API一次只允许60张照片,然后为您提供分页 这是我的tasks.py文件的一个副本-这实际上可以正常工作,但是我想检查一下我做的事情是否正确,并且不会对系统造成太大的影响Python 任务中的芹菜任务,python,django,celery,dropbox,Python,Django,Celery,Dropbox,我将dropbox与芹菜结合在我的应用程序中,这样我就允许用户在连接dropbox的情况下存储自己的照片 我已经写了一段代码,但我担心这可能会导致无限循环,从而杀死系统 我正在使用的API一次只允许60张照片,然后为您提供分页 这是我的tasks.py文件的一个副本-这实际上可以正常工作,但是我想检查一下我做的事情是否正确,并且不会对系统造成太大的影响 class DropboxUsers(PeriodicTask): run_every = timedelta(hours=4)
class DropboxUsers(PeriodicTask):
run_every = timedelta(hours=4)
def run(self, **kwargs):
logger = self.get_logger(**kwargs)
logger.info("Collecting Dropbox users")
dropbox_users = UserSocialAuth.objects.filter(provider='dropbox')
for db in dropbox_users:
...
...
...
sync_images.delay(first, second, third_argument)
return True
@task(ignore_result=True)
def sync_images(token, secret, username):
"""docstring for sync_images"""
logger = sync_images.get_logger()
logger.info("Syncing images for %s" % username)
...
...
...
...
feed = api.user_recent_media(user_id='self', count=60)
images = feed[0]
pagination = feed[1]
for obj in images:
### STORE TO DROPBOX
...
...
...
response = dropbox.put_file(f, my_picture, overwrite=True)
### CLOSE DB SESSION
sess.unlink()
if pagination:
store_images.delay(first, second, third, fourth_argument)
@task(ignore_result=True)
def store_images(token, secret, username, max_id):
"""docstring for sync_images"""
logger = store_images.get_logger()
logger.info("Storing images for %s" % username)
...
...
...
...
feed = api.user_recent_media(user_id='self', count=60, max_id=max_id)
images = feed[0]
try:
pagination = feed[1]
except:
pagination = None
for obj in images:
### STORE TO DROPBOX
...
...
...
response = dropbox.put_file(f, my_picture, overwrite=True)
### CLOSE DB SESSION
sess.unlink()
if pagination:
### BASICALLY RESTART THE TASK WITH NEW ARGS
store_images.delay(first, second, third, fourth_argument)
return True
非常感谢您的专业知识。我看不出有什么大问题。我还实现了一个任务启动另一个任务的系统 有一段时间,我在重启服务器时遇到芹菜复制任务的问题。我编写了一个decorator来包装一个任务,该任务使用缓存后端来确保具有相同参数的相同任务不会经常运行。对于你来说,这可能是一个对冲无限循环的工具
from django.core.cache import cache as _djcache
from django.utils.functional import wraps
class cache_task(object):
""" Makes sure that a task is only run once over the course of a configurable
number of seconds. Useful for tasks that get queued multiple times by accident,
or on service restart, etc. Uses django's cache (memcache) to keep track."""
def __init__(self, seconds=120, minutes=0, hours=0):
self.cache_timeout_seconds = seconds + 60 * minutes + 60 * 60 * hours
def __call__(self, task):
task.unsynchronized_run = task.run
@wraps(task.unsynchronized_run)
def wrapper(*args, **kwargs):
key = sha1(str(task.__module__) + str(task.__name__) + str(args) + str(kwargs)).hexdigest()
is_cached = _djcache.get(key)
if not is_cached:
# store the cache BEFORE to cut down on race conditions caused by long tasks
if self.cache_timeout_seconds:
_djcache.set(key, True, self.cache_timeout_seconds)
task.unsynchronized_run(*args, **kwargs)
task.run = wrapper
return task
用法:
@cache_task(hours=2)
@task(ignore_result=True)
def store_images(token, secret, username, max_id):
...