scrapy无法抓取craigslist
同样的代码在yellowbook中爬行，没有任何问题，正如预期的那样。将规则更改为CL，它会命中第一个url，然后在没有相关输出的情况下蹒跚而行scrapy无法抓取craigslist,scrapy,rules,web-crawler,Scrapy,Rules,Web Crawler,同样的代码在yellowbook中爬行，没有任何问题，正如预期的那样。将规则更改为CL，它会命中第一个url，然后在没有相关输出的情况下蹒跚而行 from scrapy.contrib.spiders import CrawlSpider, Rule from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor from scrapy.selector import HtmlXPathSelector from craigs.i
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import HtmlXPathSelector
from craigs.items import CraigsItem
class MySpider(CrawlSpider):
name = "craigs"
allowed_domains = ["craiglist.org"]
start_urls = ["http://newyork.craigslist.org/cpg/"]
rules = [Rule(SgmlLinkExtractor(restrict_xpaths=('/html/body/blockquote[3]/p/a',)), follow=True, callback='parse_profile')]
def parse_profile(self, response):
found = []
img = CraigsItem()
hxs = HtmlXPathSelector(response)
img['title'] = hxs.select('//h2[contains(@class, "postingtitle")]/text()').extract()
img['text'] = hxs.select('//section[contains(@id, "postingbody")]/text()').extract()
img['tags'] = hxs.select('//html/body/article/section/section[2]/section[2]/ul/li[1]').extract()
print found[0]
return found[0]
输出->
但在爬网中，相同的查询失败。WTF？？？如果查看文档，您将看到 允许的\u域包含以下域的字符串的可选列表： 这只蜘蛛可以爬行。请求不属于的URL 如果出现以下情况，将不遵循此列表中指定的域名 OffItemIDdleware已启用 您允许的域是
allowed_domains = ["craiglist.org"]
02-07 15:39:03+0000 [craigs] DEBUG: Filtered offsite request to 'newyork.craigslist.org': <GET http://newyork.craigslist.org/mnh/cpg/3600242403.html>
02-07 15:39:03+0000[craigs]调试：过滤到“newyork.craigslist.org”的场外请求：
从爬网程序中删除
允许的\u域