Python 具有指数退避的Grequests
以下是我用于同时执行请求的模式:Python 具有指数退避的Grequests,python,python-requests,grequests,Python,Python Requests,Grequests,以下是我用于同时执行请求的模式: rs = (grequests.get(url) for url in urls) res_items = grequests.map(rs) for num, res in enumerate(res_items): json_data = json.loads(res.text) 但是,这会因以下错误而崩溃:ConnectionError:HTTPConnectionPool(host='apicache.vudu.com',port=80):ur
rs = (grequests.get(url) for url in urls)
res_items = grequests.map(rs)
for num, res in enumerate(res_items):
json_data = json.loads(res.text)
但是,这会因以下错误而崩溃:
ConnectionError:HTTPConnectionPool(host='apicache.vudu.com',port=80):url:
大约每5000个请求超过最大重试次数。执行上述操作的更可靠模式是什么?例如,如果单个请求不起作用,则最多重试url五次?这里有一个选项,使用如下所述的指数退避:
def grequester(self, url, n=1):
'''
Google exponential backoff: https://developers.google.com/drive/web/handle-errors?hl=pt-pt
'''
MAX_TRIES = 8
try:
res = grequests.get(url)
except:
if n > MAX_TRIES:
return None
n += 1
log.warning('Try #%s for %s...' % (n, url))
time.sleep((2 ** n) + (random.randint(0, 1000) / 1000.0)) # add jitter 0-1000ms
return self.grequester(url, n)
else:
return res
指数退避是一种风险solution@AliGajani你能展示一下在grequests中处理多个请求是如何做到的吗?