Python 不可忽略的scrapy deny规则
我从数据库中动态获取了一些规则,并将它们添加到我的spider中:Python 不可忽略的scrapy deny规则,python,web-crawler,scrapy,Python,Web Crawler,Scrapy,我从数据库中动态获取了一些规则,并将它们添加到我的spider中: self.name = exSettings['site'] self.allowed_domains = [exSettings['root']] self.start_urls = ['http://' + exSettings['root']] self.rules = [Rule(SgmlLinkExtractor(allow=(exSettings[
self.name = exSettings['site']
self.allowed_domains = [exSettings['root']]
self.start_urls = ['http://' + exSettings['root']]
self.rules = [Rule(SgmlLinkExtractor(allow=(exSettings['root'] + '$',)), follow= True)]
denyRules = []
for rule in exSettings['settings']:
linkRegex = rule['link_regex']
if rule['link_type'] == 'property_url':
propertyRule = Rule(SgmlLinkExtractor(allow=(linkRegex,)), follow=True, callback='parseProperty')
self.rules.insert(0, propertyRule)
self.listingEx.append({'link_regex': linkRegex, 'extraction': rule['extraction']})
elif rule['link_type'] == 'project_url':
projectRule = Rule(SgmlLinkExtractor(allow=(linkRegex,)), follow=True) #not set to crawl yet due to conflict if same links appear for both
self.rules.insert(0, projectRule)
elif rule['link_type'] == 'favorable_url':
favorableRule = Rule(SgmlLinkExtractor(allow=(linkRegex,)), follow=True)
self.rules.append(favorableRule)
elif rule['link_type'] == 'ignore_url':
denyRules.append(linkRegex)
#somehow all urls will get ignored if allow is empty and put as the first rule
d = Rule(SgmlLinkExtractor(allow=('testingonly',), deny=tuple(denyRules)), follow=True)
#self.rules.insert(0,d) #I have tried with both status but same results
self.rules.append(d)
我的数据库中有以下规则:
link_regex: /listing/\d+/.+ (property_url)
link_regex: /project-listings/.+ (favorable_url)
link_regex: singapore-property-listing/ (favorable_url)
link_regex: /mrt/ (ignore_url)
我在日志中看到:
http://www.propertyguru.com.sg/singapore-property-listing/property-for-sale/mrt/125/ang-mo-kio-mrt-station> (referer: http://www.propertyguru.com.sg/listing/8277630/for-sale-thomson-grand-6-star-development-)
/mrt/
不应该被拒绝吗?为什么我仍然对上面的链接进行爬网?据我所知,deny
参数必须位于相同的SgmlLinkExtractor
,它具有allow
模式
在您的案例中,您创建了SgmlLinkExtractor
,它允许有利的url
('singapore-property-listing/'
)。但是这个提取器没有任何deny
模式,所以它也提取/mrt/
要解决此问题,您应该向相应的SgmlLinkExtractor
s添加deny
模式。另外,请参见
也许有一些方法可以定义全局
deny
模式,但我没有看到它们。是的,你是对的。查看源代码后,deny只会跳过匹配的链接,但它仍会在后续规则中将跳过的链接传递给提取器。