Python 如何将两个用户定义的参数传递给scrapy spider
下面,我编写了以下简单的spider:Python 如何将两个用户定义的参数传递给scrapy spider,python,scrapy,Python,Scrapy,下面,我编写了以下简单的spider: import scrapy class Funda1Spider(scrapy.Spider): name = "funda1" allowed_domains = ["funda.nl"] def __init__(self, place='amsterdam'): self.start_urls = ["http://www.funda.nl/koop/%s/" % place] def parse
import scrapy
class Funda1Spider(scrapy.Spider):
name = "funda1"
allowed_domains = ["funda.nl"]
def __init__(self, place='amsterdam'):
self.start_urls = ["http://www.funda.nl/koop/%s/" % place]
def parse(self, response):
filename = response.url.split("/")[-2] + '.html'
with open(filename, 'wb') as f:
f.write(response.body)
这似乎有效;例如,如果我在命令行中使用
scrapy crawl funda1 -a place=rotterdam
它生成一个类似于的rotterdam.html
。接下来我想扩展它,以便可以指定一个子页面,例如。我尝试了以下方法:
import scrapy
class Funda1Spider(scrapy.Spider):
name = "funda1"
allowed_domains = ["funda.nl"]
def __init__(self, place='amsterdam', page=''):
self.start_urls = ["http://www.funda.nl/koop/%s/p%s/" % (place, page)]
def parse(self, response):
filename = response.url.split("/")[-2] + '.html'
with open(filename, 'wb') as f:
f.write(response.body)
但是,如果我尝试使用
scrapy crawl funda1 -a place=rotterdam page=2
我得到以下错误:
crawl: error: running 'scrapy crawl' with more than one spider is no longer supported
我真的不理解这个错误消息,因为我没有尝试抓取两个爬行器,只是尝试传递两个关键字参数来修改
start\u URL
。我怎样才能做到这一点呢?在提供多个参数时,您需要为每个参数添加前缀-a
适用于您的案例的正确行是:
scrapy crawl funda1-a place=rotterdam-a page=2