python芹菜-get（）被延迟

我正在运行下面的简单示例。 使用单个工人提交20个作业，每个作业耗时2秒：

celery -A celery_test worker --concurrency 10 -l INFO

这应该需要2*2=4秒。这对于处理数据的工作人员来说是正确的。但是，获取数据会增加6秒的额外延迟

有没有办法摆脱这种拖延

有关脚本和输出，请参见以下内容：

芹菜酱：

芹菜_test.py：

输出芹菜\u call.py->总执行时间为10秒！！！：

记录工人的输出->按预期处理数据需要4秒：

---

这是因为在循环中等待每个作业的结果。因此，您在某种程度上失去了并发的好处，因为作业结果的到达顺序与您请求结果的顺序不同。请参见下面的示例，其中添加了一些计时以获取所有时间：

from celery_test import add
import time
results = []
for i in range(20):
    results.append(add.delay(i))

allTimeStart = time.time()

for result in results:
    timeStart = time.time()
    resultValue = result.get(timeout=10)
    timePassed = time.time() - timeStart
    allTimePassed = time.time() - allTimeStart
    print(allTimePassed, timePassed, resultValue)

给予

(1.9835469722747803, 1.9835450649261475, 0)
(1.9858801364898682, 0.0022699832916259766, 1)
(1.988955020904541, 0.003039121627807617, 2)
(1.9928300380706787, 0.003849029541015625, 3)
(1.9935901165008545, 0.0007331371307373047, 4)
(1.9967319965362549, 0.0031011104583740234, 5)
(1.9973289966583252, 0.0005509853363037109, 6)
(2.0004770755767822, 0.003117084503173828, 7)
(2.0007641315460205, 0.00026702880859375, 8)
(3.00203800201416, 1.001255989074707, 9)
(3.9891350269317627, 0.9870359897613525, 10)
(3.9914891719818115, 0.0023059844970703125, 11)
(3.99283504486084, 0.001302957534790039, 12)
(3.99426007270813, 0.0013878345489501953, 13)
(3.997709035873413, 0.003403902053833008, 14)
(3.9984171390533447, 0.0006809234619140625, 15)
(4.000844955444336, 0.0024080276489257812, 16)
(4.004598140716553, 0.003731966018676758, 17)
(4.0053839683532715, 0.0007598400115966797, 18)
(5.006708145141602, 1.0012950897216797, 19)

但是，如果您查看芹菜日志中芹菜任务结果的顺序，您会发现结果并没有按照您的要求按顺序到达：

[2017-05-31 01:06:39,067: INFO/PoolWorker-2] Task celery_test.add[01fe4581-7982-40f3-92d3-9f352d0b8eca] succeeded in 2.00315466001s: 0
[2017-05-31 01:06:39,069: INFO/PoolWorker-8] Task celery_test.add[f468849c-76d9-4479-b7e2-850aab640437] succeeded in 2.003014307s: 1
[2017-05-31 01:06:39,072: INFO/PoolWorker-3] Task celery_test.add[db6a0064-0a83-49dc-a731-54264651a32f] succeeded in 2.002590772s: 3
[2017-05-31 01:06:39,072: INFO/PoolWorker-4] Task celery_test.add[421b1213-e1b7-4c73-8477-1554c53c4b14] succeeded in 2.002614007s: 2
[2017-05-31 01:06:39,076: INFO/PoolWorker-7] Task celery_test.add[90bdde7f-9740-4d18-820d-dc4c66090b2b] succeeded in 2.00297982999s: 4
[2017-05-31 01:06:39,077: INFO/PoolWorker-5] Task celery_test.add[661cba10-326a-4351-9fec-56d029847939] succeeded in 2.003134354s: 5
[2017-05-31 01:06:39,080: INFO/PoolWorker-10] Task celery_test.add[31903dfe-4b35-49b8-bc66-8c8807a1ee53] succeeded in 2.00229301301s: 6
[2017-05-31 01:06:39,080: INFO/PoolWorker-9] Task celery_test.add[60049a1b-009b-4d7b-ad4e-284f0d2e7147] succeeded in 2.00245238301s: 7
[2017-05-31 01:06:39,084: INFO/PoolWorker-1] Task celery_test.add[4e673409-af0e-4a59-8a42-38f0179b495a] succeeded in 2.00299428699s: 8
[2017-05-31 01:06:39,084: INFO/PoolWorker-6] Task celery_test.add[818bcea5-5654-4ec6-8706-1b6ca58f8735] succeeded in 2.002899974s: 9
[2017-05-31 01:06:41,072: INFO/PoolWorker-2] Task celery_test.add[4ab62e6d-ada3-4e0d-82e2-356eb054631f] succeeded in 2.00349172599s: 10
[2017-05-31 01:06:41,074: INFO/PoolWorker-8] Task celery_test.add[649c83db-a065-4cdd-9f5e-32ae1e5047f4] succeeded in 2.003091722s: 11
[2017-05-31 01:06:41,076: INFO/PoolWorker-4] Task celery_test.add[f6a6e067-7f60-4c1f-b8f4-dce40a6094c0] succeeded in 2.00157168499s: 12
[2017-05-31 01:06:41,077: INFO/PoolWorker-3] Task celery_test.add[ee7b0e01-2fa7-4bd0-b2f2-f5636155209b] succeeded in 2.00259804301s: 13
[2017-05-31 01:06:41,081: INFO/PoolWorker-7] Task celery_test.add[521f7903-3594-4aab-b4df-3a4e723347cd] succeeded in 2.002994123s: 14
[2017-05-31 01:06:41,081: INFO/PoolWorker-5] Task celery_test.add[26a3627c-7934-4613-b3c1-618784bbce26] succeeded in 2.003302467s: 15
[2017-05-31 01:06:41,084: INFO/PoolWorker-9] Task celery_test.add[8e796394-b05f-439b-b695-6d3ff3230844] succeeded in 2.00281064s: 17
[2017-05-31 01:06:41,084: INFO/PoolWorker-10] Task celery_test.add[13b40cd8-b0e4-4788-a3bb-4d050c1b6ad0] succeeded in 2.00298337401s: 16
[2017-05-31 01:06:41,088: INFO/PoolWorker-6] Task celery_test.add[cb8f1303-4d05-4eae-9b40-b2d221f20140] succeeded in 2.00274520101s: 19
[2017-05-31 01:06:41,088: INFO/PoolWorker-1] Task celery_test.add[0900bb54-8e2a-472c-99a8-ee18a8f4857c] succeeded in 2.003100015s: 18

一种解决方案：使用组获取所有结果：

from celery_test import add
from celery import group
import time
results = []
jobs = []
for i in range(20):
    jobs.append(add.s(i))
result = group(jobs).apply_async()
timeStart = time.time()
print(result.join())
timePassed = time.time() - timeStart
print(timePassed)

返回

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
4.00328302383

(1.9835469722747803, 1.9835450649261475, 0)
(1.9858801364898682, 0.0022699832916259766, 1)
(1.988955020904541, 0.003039121627807617, 2)
(1.9928300380706787, 0.003849029541015625, 3)
(1.9935901165008545, 0.0007331371307373047, 4)
(1.9967319965362549, 0.0031011104583740234, 5)
(1.9973289966583252, 0.0005509853363037109, 6)
(2.0004770755767822, 0.003117084503173828, 7)
(2.0007641315460205, 0.00026702880859375, 8)
(3.00203800201416, 1.001255989074707, 9)
(3.9891350269317627, 0.9870359897613525, 10)
(3.9914891719818115, 0.0023059844970703125, 11)
(3.99283504486084, 0.001302957534790039, 12)
(3.99426007270813, 0.0013878345489501953, 13)
(3.997709035873413, 0.003403902053833008, 14)
(3.9984171390533447, 0.0006809234619140625, 15)
(4.000844955444336, 0.0024080276489257812, 16)
(4.004598140716553, 0.003731966018676758, 17)
(4.0053839683532715, 0.0007598400115966797, 18)
(5.006708145141602, 1.0012950897216797, 19)

[2017-05-31 01:06:39,067: INFO/PoolWorker-2] Task celery_test.add[01fe4581-7982-40f3-92d3-9f352d0b8eca] succeeded in 2.00315466001s: 0
[2017-05-31 01:06:39,069: INFO/PoolWorker-8] Task celery_test.add[f468849c-76d9-4479-b7e2-850aab640437] succeeded in 2.003014307s: 1
[2017-05-31 01:06:39,072: INFO/PoolWorker-3] Task celery_test.add[db6a0064-0a83-49dc-a731-54264651a32f] succeeded in 2.002590772s: 3
[2017-05-31 01:06:39,072: INFO/PoolWorker-4] Task celery_test.add[421b1213-e1b7-4c73-8477-1554c53c4b14] succeeded in 2.002614007s: 2
[2017-05-31 01:06:39,076: INFO/PoolWorker-7] Task celery_test.add[90bdde7f-9740-4d18-820d-dc4c66090b2b] succeeded in 2.00297982999s: 4
[2017-05-31 01:06:39,077: INFO/PoolWorker-5] Task celery_test.add[661cba10-326a-4351-9fec-56d029847939] succeeded in 2.003134354s: 5
[2017-05-31 01:06:39,080: INFO/PoolWorker-10] Task celery_test.add[31903dfe-4b35-49b8-bc66-8c8807a1ee53] succeeded in 2.00229301301s: 6
[2017-05-31 01:06:39,080: INFO/PoolWorker-9] Task celery_test.add[60049a1b-009b-4d7b-ad4e-284f0d2e7147] succeeded in 2.00245238301s: 7
[2017-05-31 01:06:39,084: INFO/PoolWorker-1] Task celery_test.add[4e673409-af0e-4a59-8a42-38f0179b495a] succeeded in 2.00299428699s: 8
[2017-05-31 01:06:39,084: INFO/PoolWorker-6] Task celery_test.add[818bcea5-5654-4ec6-8706-1b6ca58f8735] succeeded in 2.002899974s: 9
[2017-05-31 01:06:41,072: INFO/PoolWorker-2] Task celery_test.add[4ab62e6d-ada3-4e0d-82e2-356eb054631f] succeeded in 2.00349172599s: 10
[2017-05-31 01:06:41,074: INFO/PoolWorker-8] Task celery_test.add[649c83db-a065-4cdd-9f5e-32ae1e5047f4] succeeded in 2.003091722s: 11
[2017-05-31 01:06:41,076: INFO/PoolWorker-4] Task celery_test.add[f6a6e067-7f60-4c1f-b8f4-dce40a6094c0] succeeded in 2.00157168499s: 12
[2017-05-31 01:06:41,077: INFO/PoolWorker-3] Task celery_test.add[ee7b0e01-2fa7-4bd0-b2f2-f5636155209b] succeeded in 2.00259804301s: 13
[2017-05-31 01:06:41,081: INFO/PoolWorker-7] Task celery_test.add[521f7903-3594-4aab-b4df-3a4e723347cd] succeeded in 2.002994123s: 14
[2017-05-31 01:06:41,081: INFO/PoolWorker-5] Task celery_test.add[26a3627c-7934-4613-b3c1-618784bbce26] succeeded in 2.003302467s: 15
[2017-05-31 01:06:41,084: INFO/PoolWorker-9] Task celery_test.add[8e796394-b05f-439b-b695-6d3ff3230844] succeeded in 2.00281064s: 17
[2017-05-31 01:06:41,084: INFO/PoolWorker-10] Task celery_test.add[13b40cd8-b0e4-4788-a3bb-4d050c1b6ad0] succeeded in 2.00298337401s: 16
[2017-05-31 01:06:41,088: INFO/PoolWorker-6] Task celery_test.add[cb8f1303-4d05-4eae-9b40-b2d221f20140] succeeded in 2.00274520101s: 19
[2017-05-31 01:06:41,088: INFO/PoolWorker-1] Task celery_test.add[0900bb54-8e2a-472c-99a8-ee18a8f4857c] succeeded in 2.003100015s: 18

from celery_test import add
from celery import group
import time
results = []
jobs = []
for i in range(20):
    jobs.append(add.s(i))
result = group(jobs).apply_async()
timeStart = time.time()
print(result.join())
timePassed = time.time() - timeStart
print(timePassed)

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
4.00328302383