Python xpath有一个空值,它会把列表弄乱
我正在使用下面的代码从网页上抓取姓名、地址和汽车数量 但是,汽车数量经常有一个空值。在下面的示例中,假设第八家经销商返回的汽车数量为空,因此返回的列表如下所示: 名称=a、b、c、d、e、f、g、h、i、j 地址=aa、bb、cc、dd、ee、ff、gg、hh、ii、jj cars=1,2,3,4,5,6,7,9,10 如果地址aa处的经销商a有1辆车,地址bb处的经销商b有2辆车等,但由于地址hh处的经销商h有一个空值,因此跳过该代码,相反,该代码认为经销商h有9辆车,因此经销商i和地址ii有10辆车,因此地址jj处的经销商j被遗漏,因为车辆列表已用完 因此,如果代码返回cars的空值,如何用0替换该值?因此,在上面的示例中,经销商h和地址hh将有0辆车,因此地址ii处的经销商i有9辆车,地址jj处的经销商j有10辆车Python xpath有一个空值,它会把列表弄乱,python,web-scraping,scrapy,Python,Web Scraping,Scrapy,我正在使用下面的代码从网页上抓取姓名、地址和汽车数量 但是,汽车数量经常有一个空值。在下面的示例中,假设第八家经销商返回的汽车数量为空,因此返回的列表如下所示: 名称=a、b、c、d、e、f、g、h、i、j 地址=aa、bb、cc、dd、ee、ff、gg、hh、ii、jj cars=1,2,3,4,5,6,7,9,10 如果地址aa处的经销商a有1辆车,地址bb处的经销商b有2辆车等,但由于地址hh处的经销商h有一个空值,因此跳过该代码,相反,该代码认为经销商h有9辆车,因此经销商i和地址ii有
import scrapy
from autotrader.items import AutotraderItem
class AutotraderSpider(scrapy.Spider):
name = "autotrader"
allowed_domains = ["autotrader.co.uk"]
start_urls = ["https://www.autotrader.co.uk/car-dealers/search?advertising-location=at_cars&postcode=m43aq&radius=1500&forSale=on&toOrder=on&sort=with-retailer-reviews&page=822"]
def parse(self, response):
for sel in response.xpath('//ul[@class="dealerList__container"]'):
names = sel.xpath('.//*[@itemprop="legalName"]/text() ').extract()
names = [name.strip() for name in names]
addresses = sel.xpath('.//li/article/a/div/p[@itemprop="address"]/text()').extract()
addresses = [address.strip() for address in addresses]
carss = sel.xpath('.//li/article/a/div/p[@class="dealerList__itemCount"]/span/text()').extract()
carss = [cars.strip() for cars in carss]
result = zip(names, addresses, carss)
for name, address, cars in result:
item = AutotraderItem()
item['name'] = name
item['address'] = address
item['cars'] = cars
yield item
您的选择器循环有点混乱 在这里,您可以循环浏览未排序的列表,其中每个年龄只有一个列表:
for sel in response.xpath('//ul[@class="dealerList__container"]'):
您需要的是循环浏览所有列表项:
for sel in response.xpath('//li[@class="dealerList__itemContainer"]'):
如果以这种方式循环,则可以获得每个列表项的名称、地址:
for sel in response.xpath('//li[@class="dealerList__itemContainer"]'):
names = sel.xpath('.//*[@itemprop="legalName"]/text() ').extract()
names = [name.strip() for name in names]
addresses = sel.xpath('.//article/a/div/p[@itemprop="address"]/text()').extract()
addresses = [address.strip() for address in addresses]
carss = sel.xpath('.//article/a/div/p[@class="dealerList__itemCount"]/span/text()').extract()
carss = [cars.strip() for cars in carss]
item = AutotraderItem()
item['name'] = name
item['address'] = address
item['cars'] = cars
yield item
试试这个来得到结果。您可以在scrapy项目中使用
xpaths
,如下所示:
class AutotraderSpider(scrapy.Spider):
name = "autotrader"
allowed_domains = ["autotrader.co.uk"]
start_urls = ["https://www.autotrader.co.uk/car-dealers/search?advertising-location=at_cars&postcode=m43aq&radius=1500&forSale=on&toOrder=on&sort=with-retailer-reviews&page=822"]
def parse(self, response):
for items in response.xpath("//article[@class='dealerList__item']"):
name = items.xpath(".//span[@itemprop='legalName']/text()").extract_first()
address = ' '.join([' '.join(item.split()) for item in items.xpath(".//p[@class='dealerList__itemAddress']/text()").extract()])
cars = items.xpath(".//span[@class='dealerList__itemCountNumber']/text()").extract_first()
yield {"Name":name,"Address":address,"Cars":cars}
部分输出:
Midland Motors Leicester Street, Burton-On-Trent, Staffordshire DE14 3BA 2
Ns Cars 69 Eldon Street, Burton-On-Trent, Staffordshire DE15 0LT 1
RS Sales Nottingham Ltd Unit 1 TRINITY PARK, RANDALL PARK WAY, Retford, Nottinghamshire DN22 7WF 1
Adc Ltd Unit 3 HUCKNALL LANE, Nottingham, Nottinghamshire NG6 8AJ 5
最简单的答案是重新组织代码,这样就不用进行三次单独的xpath查询并尝试将结果压缩在一起,而是进行一次顶级查询以获取组,然后针对每个组,使用xpath从组中获取三个子字段。这很有魅力,谢谢。值得注意的是,对于阅读上述代码的人来说,解决方案是
for sel in response.xpath('//li[@class=“dealerList\uu itemContainer”]):
带有变量名称、carrs等的示例代码需要调整,方法是将名称更改为name etc,并去掉其中包含名称.strip()等的某些行。