Warning: file_get_contents(/data/phpspider/zhask/data//catemap/2/python/293.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181

Warning: file_get_contents(/data/phpspider/zhask/data//catemap/3/xpath/2.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Python xpath不使用scrapy shell提取内容_Python_Xpath_Scrapy - Fatal编程技术网

Python xpath不使用scrapy shell提取内容

Python xpath不使用scrapy shell提取内容,python,xpath,scrapy,Python,Xpath,Scrapy,我能够在网站上使用XPath助手google chrome扩展获取网页内容,但如果我使用sxrapy shell应用,则无法使用相同的路径 使用XPath chrome扩展: webpage=anywebsite.com XPath=//section[@class='rslwrp']/div/ul/li//h2[@class='store-name']/span/a/span/text() 但当我使用scrapy shell应用程序时,它返回空值,为什么 scrapy shell "www.m

我能够在网站上使用XPath助手google chrome扩展获取网页内容,但如果我使用sxrapy shell应用,则无法使用相同的路径

使用XPath chrome扩展:

webpage=anywebsite.com

XPath=//section[@class='rslwrp']/div/ul/li//h2[@class='store-name']/span/a/span/text()

但当我使用scrapy shell应用程序时,它返回空值,为什么

scrapy shell "www.mywebsite.com"
In [15]: response.xpath("//section[@class='rslwrp']/div/ul/li//h2[@class='store-
    ...: name']/span/a/span")                                                   
Out[15]: []

设置用户代理后,它似乎可以工作:

scrapy shell "www.mywebsite" -s USER_AGENT="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/53.0"
结果:

In [1]: response
Out[1]: <200 https://www.mywebsite.com>

设置用户代理后,它似乎可以工作:

scrapy shell "www.mywebsite" -s USER_AGENT="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/53.0"
结果:

In [1]: response
Out[1]: <200 https://www.mywebsite.com>