Python 刮痧罐头';我没有得到正确的答复
我试图从中获取song&singerdata,但无法获得正确的响应 我查看了scrapy shell,当我请求“music.163.com/#/artist?id=16686”时,响应是“music.163.com”。我不知道原因 下面是日志Python 刮痧罐头';我没有得到正确的答复,python,scrapy,Python,Scrapy,我试图从中获取song&singerdata,但无法获得正确的响应 我查看了scrapy shell,当我请求“music.163.com/#/artist?id=16686”时,响应是“music.163.com”。我不知道原因 下面是日志 C:\Users\lszxw\PycharmProjects\untitled\scrapy\tutorial\tutorial>scrapy shell http://music.163.com/#/artist/album?id=16686 20
C:\Users\lszxw\PycharmProjects\untitled\scrapy\tutorial\tutorial>scrapy shell http://music.163.com/#/artist/album?id=16686
2017-01-16 22:47:03 [scrapy.utils.log] INFO: Scrapy 1.3.0 started (bot: tutorial)
2017-01-16 22:47:03 [scrapy.utils.log] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'tutorial.spiders', 'LOGSTATS_INTERVAL': 0, 'BOT_NAME': 'tutorial', 'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter', 'SPIDER_MODULES': ['tutorial.spiders'], 'ROBOTSTXT_OBEY': True}
2017-01-16 22:47:03 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole']
2017-01-16 22:47:03 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-01-16 22:47:03 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2017-01-16 22:47:03 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2017-01-16 22:47:03 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-01-16 22:47:03 [scrapy.core.engine] INFO: Spider opened
2017-01-16 22:47:03 [scrapy.core.engine] DEBUG: Crawled (404) <GET http://music.163.com/robots.txt> (referer: None)
2017-01-16 22:47:03 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://music.163.com/#/artist/album?id=16686> (referer: None)
2017-01-16 22:47:04 [traitlets] DEBUG: Using default logger
2017-01-16 22:47:04 [traitlets] DEBUG: Using default logger
[s] Available Scrapy objects:
[s] scrapy scrapy module (contains scrapy.Request, scrapy.Selector, etc)
[s] crawler <scrapy.crawler.Crawler object at 0x0000018893DAB9E8>
[s] item {}
[s] request <GET http://music.163.com/#/artist/album?id=16686>
[s] response <200 http://music.163.com/>
[s] settings <scrapy.settings.Settings object at 0x0000018893DCEF98>
[s] spider <DefaultSpider 'default' at 0x18893fd39b0>
[s] Useful shortcuts:
[s] fetch(url[, redirect=True]) Fetch URL and update local objects (by default, redirects are followed)
[s] fetch(req) Fetch a scrapy.Request and update local objects
[s] shelp() Shell help (print this help)
[s] view(response) View response in a browser
您的
start\u URL
似乎不正确。如果您查看“网络”选项卡和页面源,您会注意到唱片集/歌曲数据实际上包含在一个
标记中,这导致几乎相同的url,只是没有
:
之后,parse_相册
中的歌曲xpath不正确。我把它和这个一起工作:
"//ul[@class='f-hide']/li/a/@href[contains(.,'song')]"
之后一切似乎都正常。这是Ajax驱动的URL:因此,
之后的所有部分都没有得到正确处理。尝试使用AjaxCrawlMiddleware
,但是Scrapy不能很好地处理ajaxURL。Thx Granit。但我似乎无法获得“”上的评论编号。我可以通过使用“//span[@id=“cnt\u comment\u count”]/text()”获取html消息是61,但无法获取数字。这是因为“#”使用了Ajax吗?
"http://music.163.com/#/artist/album?id=16686"
# becomes:
"http://music.163.com/artist/album?id=16686"
"//ul[@class='f-hide']/li/a/@href[contains(.,'song')]"