Python 已超过延迟的任务请求截止日期，但工作从未开始

我有一个任务队列，用于处理需要实时处理的延迟任务（我听说这不是一个好主意，但找不到任何支持文档）。这个队列受到大量任务（2500/分钟）的冲击，然后通常在一两分钟内什么也没有收到

queue.yaml

- name: event-message
  rate: 200/s
  bucket_size: 200
  retry_parameters:
      task_retry_limit: 2
      min_backoff_seconds: 1

我注意到，对于一些任务，甚至没有启动，我经常会收到“进程终止，因为请求截止日期已超过。（错误代码123）”消息。任务在队列中停留10分钟而未运行，然后超时。我认为这可能与我的队列配置有关，所以我将其包括在这里

2015-06-18 10:22:45.842 /_ah/queue/deferred 500 641080ms 0kb AppEngine-Google; (+http://code.google.com/appengine) module=default version=1-0-21
0.1.0.2 - - [18/Jun/2015:07:22:45 -0700] "POST /_ah/queue/deferred HTTP/1.1" 500 0 "http://*******/_ah/queue/deferred" "AppEngine-Google; (+http://code.google.com/appengine)" "********" ms=641081 cpu_ms=0 queue_name=event-message task_name=0477163503045474472 exit_code=123 app_engine_release=1.9.22 instance=00c61b117c8af1f4df1247032afd14111a19f302
E 2015-06-18 10:22:45.842
Process terminated because the request deadline was exceeded. (Error code 123)

有人能解释一下吗？有什么办法可以让我知道为什么这些任务永远不会运行吗

因此，我更多地查看了日志，发现更多的证据支持我的理论，即任务甚至还没有开始。在任务10分钟截止日期到达后，此处会出现“此请求导致为您的应用程序启动新进程”

2015-06-24 17:16:59.554 /_ah/queue/deferred 500 641631ms 0kb AppEngine-Google; (+http://code.google.com/appengine) module=default version=1-0-27
0.1.0.2 - - [24/Jun/2015:14:16:59 -0700] "POST /_ah/queue/deferred HTTP/1.1" 500 0 "https://integrate.versature.com/api/netsapiens_events/96615760-cac7-40d6-b80d-13a1ffade2c8/" "AppEngine-Google; (+http://code.google.com/appengine)" "integrate.versature.com" ms=641632 cpu_ms=0 queue_name=event-message task_name=59067488044178506841 pending_ms=1061 exit_code=123 app_engine_release=1.9.22 instance=00c61b117c8f2db4ade6be8d30165cb84f0a17ab
E 2015-06-24 17:16:59.554
Process terminated because the request deadline was exceeded. (Error code 123)

下面是另一个：

I 2015-06-23 13:48:22.970
X-Appengine-Taskretrycount:0, X-Appengine-Tasketa:1435081701.5576639, X-Appengine-Default-Namespace:versature.com, X-Appengine-Taskname:89115825957976957021, X-Appengine-Taskexecutioncount:0, X-Appengine-Queuename:event-message, X-Appengine-Current-Namespace:, X-Appengine-Country:ZZ

E 2015-06-23 13:58:21.669
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 267, in Handle
result = handler(dict(self._environ), self._StartResponse)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in __call__
rv = self.router.dispatch(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in __call__
return handler.dispatch()
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
return method(*args, **kwargs)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 310, in post
self.run_from_request()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 305, in run_from_request
run(self.request.body)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 147, in run
return func(*args, **kwds)
File "/base/data/home/apps/s~versature-integrate-2/1-0-26.385226646541088482/app/api/resources/constructs/netsapiens_event/process_netsapiens_event.py", line 66, in process_data_for_endpoint
logging.info("Process data for specific endpoint. Endpoint Key: %s", endpoint_key_url_safe)
File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/logging/__init__.py", line 1611, in info
root.info(msg, *args, **kwargs)
File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/logging/__init__.py", line 1149, in info
self._log(INFO, msg, args, **kwargs)
File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/logging/__init__.py", line 1268, in _log
self.handle(record)
File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/logging/__init__.py", line 1278, in handle
self.callHandlers(record)
File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/logging/__init__.py", line 1318, in callHandlers
hdlr.handle(record)
File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/logging/__init__.py", line 749, in handle
self.emit(record)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/app_logging.py", line 79, in emit
self._AppLogsLocation())
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/logservice/logservice.py", line 457, in write_record
logs_buffer().write_record(level, created, message, source_location)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/logservice/logservice.py", line 270, in write_record
message = cleanup_message(message)
DeadlineExceededError

I 2015-06-23 13:58:21.684
This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application.

---

您是如何得出任务未运行的结论的？日志显示请求耗时641080ms，这比10分钟任务队列执行时间限制略高—可能是在请求终止时。也许你试图在一个任务中做太多的事情，或者某个无休止的循环潜伏在其中。我得出结论，该任务没有运行，因为与该延迟任务相关的日志消息从未出现过。对于这些任务，我得到的唯一结果是进程在10分钟后终止。我假设即使进程终止，我仍然可以看到我的日志消息。这不是真的吗？你说的东西太多是什么意思？一般来说，延迟的任务在一秒钟内执行（除了少数超时的任务）。延迟的任务被传递给一个复杂的对象，该对象延迟加载并准备结果。我想可能有什么东西导致了问题，但如果没有日志记录，很难知道是什么。缺少日志可能表明处理过程中出现了问题。可能相关-此问答建议延迟库可能不适合高吞吐量应用程序：。我只是想问问而已。好吧，我离开了推迟的图书馆。我现在正在使用taskqueue库。因此，正如谷歌所记录的那样，这应该允许更高的吞吐量。不过我也有同样的问题。我最终完成了7到8项似乎没有开始的任务。这些任务只是排在队列中，然后在10分钟后超时。有什么想法吗？