Python 当一行中发送多个请求时，请求停止

我正在使用scrapy和python请求解析在线商店，在获得所有信息后，我又发出一个请求，以通过python请求获取数量，几分钟后，spider停止工作 我不知道是什么引起了麻烦。有什么建议吗

刮削原木：

2014-05-08 15:27:57+0300 [scrapy] DEBUG: Start adding sku1270594 to a cart.
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): www.sds.com.au
DEBUG:requests.packages.urllib3.connectionpool:"GET /product/trefoil-tee-by-adidas-in-black-camo-grey HTTP/1.1" 200 20223
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): www.sds.com.au
DEBUG:requests.packages.urllib3.connectionpool:"POST /common/ajaxResponse.jsp;jsessionid=34E95C7662D0F5084FF971CC5693E6E8.store-node1?_DARGS=/browse/product.jsp.addToCartForm HTTP/1.1" 200 146
2014-05-08 15:27:59+0300 [scrapy] DEBUG: End adding sku1270594 to a cart.
2014-05-08 15:27:59+0300 [scrapy] DEBUG: Success. quantity of sku1270594 is 16.
2014-05-08 15:28:00+0300 [sds] DEBUG: Updating  product info sku1270594
2014-05-08 15:28:00+0300 [sds] DEBUG: Added new price sku1270594
2014-05-08 15:28:00+0300 [sds] DEBUG: Scraped from <200 http://www.sds.com.au/product/trefoil-tee-by-adidas-in-black-camo-grey>
2014-05-08 15:28:00+0300 [sds] DEBUG: Updating  product info sku901159
2014-05-08 15:28:00+0300 [sds] DEBUG: Added new price sku901159
2014-05-08 15:28:00+0300 [sds] DEBUG: Scraped from <200 http://www.sds.com.au/product/two-palm-tee-by-folke-in-chalk>
2014-05-08 15:28:00+0300 [sds] DEBUG: Updating  product info sku901163
2014-05-08 15:28:00+0300 [sds] DEBUG: Added new price sku901163
2014-05-08 15:28:00+0300 [sds] DEBUG: Scraped from <200 http://www.sds.com.au/product/two-palm-tee-by-folke-in-chalk>
2014-05-08 15:28:00+0300 [scrapy] DEBUG: Start adding sku1270591 to a cart.
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): www.sds.com.au
DEBUG:requests.packages.urllib3.connectionpool:"GET /product/trefoil-tee-by-adidas-in-black-camo-grey HTTP/1.1" 200 20225
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): www.sds.com.au

---

嗯，Jan Vlcinsky建议深入记录请求，经过一些挖掘，我决定重新组织我的代码，这给了我正确的答案，现在一切都很好

def get_qty(self, item):
    log.msg("Start adding %s to a cart." % item['internal_id'], log.DEBUG)
    logging.basicConfig(level=logging.DEBUG)
    sess = requests.Session()
    sess.keep_alive = False
    adapter = HTTPAdapter(max_retries=50)
    sess.mount('http://', adapter)
    r = sess.get(item['url'])
    cookie_cart_user = dict(r.cookies)
    sel = Selector(text=r.text, type="html")
    session = sel.xpath('//input[@name="_dynSessConf"]/@value').extract()[0]
    add_to_cart_url = 'http://www.sds.com.au/common/ajaxResponse.jsp;jsessionid=%s?_DARGS=/browse/product.jsp.addToCartForm' % cookie_cart_user['JSESSIONID']
    headers = {
        'User-Agent': USER_AGENT,
        'Accept': 'application/json, text/javascript, */*; q=0.01',
    }
    r = sess.post(add_to_cart_url,
                      data=self.generate_form_data(item, 10000, session),
                      cookies=cookie_cart_user,
                      headers=headers,
                      )
    log.msg("End adding %s to a cart." % item['internal_id'], log.DEBUG)
    try:
        response = r.json()
        r.close()
        quantity = int(re.findall(u'\d+', response['formErrors'][0]['errorMessage'])[0])
        log.msg("Success. quantity of %s is %s." % (item['internal_id'], quantity), log.DEBUG)
        return quantity
    except Exception, e:
        log.msg('Error getting data-cart-item on product %s. Error: %s' % (item['internal_id'], str(e)), log.ERROR)
        with open("log/%s.html" % item['internal_id'], "w") as myfile:
            myfile.write('%s' % r.text.encode('utf-8'))

现在，如果发生错误，日志会显示

2014-05-08 16:00:10+0300 [scrapy] DEBUG: Start adding sku1210352 to a cart.
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): www.sds.com.au
WARNING:requests.packages.urllib3.connectionpool:Retrying (50 attempts remain) after connection broken by 'error(60, 'Operation timed out')': /product/startlet-gilet-fleece-jacket-by-zoo-york-in-black
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (2): www.sds.com.au
DEBUG:requests.packages.urllib3.connectionpool:"GET /product/startlet-gilet-fleece-jacket-by-zoo-york-in-black HTTP/1.1" 200 20278
DEBUG:requests.packages.urllib3.connectionpool:"POST /common/ajaxResponse.jsp;jsessionid=EEA02CE768B288DD302896F6A8C4780F.store-node2?_DARGS=/browse/product.jsp.addToCartForm HTTP/1.1" 200 145
2014-05-08 16:01:14+0300 [scrapy] DEBUG: End adding sku1210352 to a cart.

---

在这之后，它会重试，并像什么都没有发生一样继续运行。当你重新运行脚本时，它会立即工作（至少是对站点的最初几次请求），还是暂时不工作？这是可能的，网站决定为您服务，因为您有更高的请求率。即使重新启动脚本效果很好，这也可能是真的，因为阻止请求可能与已建立的会话id有关。是的，当我重新运行它时也是一样。它会工作一段时间（最多8分钟）并停止。简，你对我该如何解决这个问题有什么建议吗？提前谢谢。@user32223824如果站点声明了一些请求率，请检查。如果是这样的话，试着遵循它们，可能会增加一些

时间。在你的请求之间
请求之间的sleep（）
现在相当稳定。无论如何，您应该启用请求本身的详细日志记录，并查看更多信息。这里的说明是：好的。现在我们可以看到，它被http请求卡住了。它启动请求，但不结束。尝试将
timeout
添加到您的请求中，如下所述。看，哦，它又停了，不再重试。。。唉，问题似乎出在服务器端，而不是客户端代码中。如果可以，请咨询web应用程序的管理员，您可能会发现，必须遵循一些规则才能按顺序访问更多请求。或者他们的服务很不稳定，你的房子就建在这个沙质地基上。
2014-05-08 16:00:10+0300 [scrapy] DEBUG: Start adding sku1210352 to a cart.
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): www.sds.com.au
WARNING:requests.packages.urllib3.connectionpool:Retrying (50 attempts remain) after connection broken by 'error(60, 'Operation timed out')': /product/startlet-gilet-fleece-jacket-by-zoo-york-in-black
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (2): www.sds.com.au
DEBUG:requests.packages.urllib3.connectionpool:"GET /product/startlet-gilet-fleece-jacket-by-zoo-york-in-black HTTP/1.1" 200 20278
DEBUG:requests.packages.urllib3.connectionpool:"POST /common/ajaxResponse.jsp;jsessionid=EEA02CE768B288DD302896F6A8C4780F.store-node2?_DARGS=/browse/product.jsp.addToCartForm HTTP/1.1" 200 145
2014-05-08 16:01:14+0300 [scrapy] DEBUG: End adding sku1210352 to a cart.