Python xpath不使用scrapy shell提取内容
我能够在网站上使用XPath助手google chrome扩展获取网页内容,但如果我使用sxrapy shell应用,则无法使用相同的路径 使用XPath chrome扩展: webpage=anywebsite.com XPath=//section[@class='rslwrp']/div/ul/li//h2[@class='store-name']/span/a/span/text() 但当我使用scrapy shell应用程序时,它返回空值,为什么Python xpath不使用scrapy shell提取内容,python,xpath,scrapy,Python,Xpath,Scrapy,我能够在网站上使用XPath助手google chrome扩展获取网页内容,但如果我使用sxrapy shell应用,则无法使用相同的路径 使用XPath chrome扩展: webpage=anywebsite.com XPath=//section[@class='rslwrp']/div/ul/li//h2[@class='store-name']/span/a/span/text() 但当我使用scrapy shell应用程序时,它返回空值,为什么 scrapy shell "www.m
scrapy shell "www.mywebsite.com"
In [15]: response.xpath("//section[@class='rslwrp']/div/ul/li//h2[@class='store-
...: name']/span/a/span")
Out[15]: []
设置用户代理后,它似乎可以工作:
scrapy shell "www.mywebsite" -s USER_AGENT="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/53.0"
结果:
In [1]: response
Out[1]: <200 https://www.mywebsite.com>
设置用户代理后,它似乎可以工作:
scrapy shell "www.mywebsite" -s USER_AGENT="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/53.0"
结果:
In [1]: response
Out[1]: <200 https://www.mywebsite.com>