Python 为什么我的蜘蛛不&x27；你看不到下一页吗？
我的蜘蛛没有水平爬行，我也不知道为什么Python 为什么我的蜘蛛不&x27；你看不到下一页吗？,python,scrapy,Python,Scrapy,我的蜘蛛没有水平爬行，我也不知道为什么 parse_item函数在第一页上运行良好。我在scrapy shell中检查了next\u page的xpath，它是正确的 你能查一下我的密码吗 我正在努力搜索的网站是 更新 在搜索日志时，我发现了关于水平爬网的以下内容： 2021-02-22 17:09:24 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.vivareal.com.br/venda/minas-gerais
parse_itemnext\u page2021-02-22 17:09:24 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.vivareal.com.br/venda/minas-gerais/belo-horizonte/#pagina=2> (referer: https://www.vivareal.com.br/venda/minas-gerais/belo-horizonte/)
2021-02-22 17:09:24 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET https://www.vivareal.com.br/venda/minas-gerais/belo-horizonte/#pagina=2> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)
2021-02-22 17:09:24[scrapy.core.engine]调试：爬网（200）（参考：https://www.vivareal.com.br/venda/minas-gerais/belo-horizonte/)
2021-02-22 17:09:24[scrapy.dupefilters]调试：已筛选的重复请求：-将不再显示重复项（请参阅DUPEFILTER\u调试以显示所有重复项）
href#pagina=2？pagina=2有什么提示吗？事实上，你的蜘蛛连第一页都没有爬行 问题存在于允许的\u域中。换成
allowed_domains=['www.vivareal.com.br']
2021-02-22 13:36:19 [scrapy.spidermiddlewares.offsite] DEBUG: Filtered offsite request to 'www.vivareal.com.br': <GET https://www.vivareal.com.br/venda/minas-gerais/belo-horizonte/#pagina=2>
2021-02-22 13:36:19[scrapy.spidermiddleware.offsite]调试：过滤到“www.vivareal.com.br”的异地请求：
2021-02-22 13:29:18 [scrapy.utils.log] INFO: Scrapy 2.4.1 started (bot: properties)
2021-02-22 13:29:18 [scrapy.utils.log] INFO: Versions: lxml 4.5.2.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 20.3.0, Python 3.8.1 (default, Feb 9 2020, 21:34:32) - [GCC 7.4.0], pyOpenSSL 20.0.1 (OpenSSL 1.1.1i 8 Dec 2020), cryptography 3.3.1, Platform Linux-4.15.0-135-generic-x86_64-with-glibc2.27
2021-02-22 13:29:18 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.epollreactor.EPollReactor
2021-02-22 13:29:18 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'properties',
'NEWSPIDER_MODULE': 'properties.spiders',
'ROBOTSTXT_OBEY': True,
'SPIDER_MODULES': ['properties.spiders']}
2021-02-22 13:29:18 [scrapy.extensions.telnet] INFO: Telnet Password: 3790c3525890efea
2021-02-22 13:29:18 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.memusage.MemoryUsage',
'scrapy.extensions.logstats.LogStats']
2021-02-22 13:29:18 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2021-02-22 13:29:18 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2021-02-22 13:29:18 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2021-02-22 13:29:18 [scrapy.core.engine] INFO: Spider opened
2021-02-22 13:29:18 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2021-02-22 13:29:18 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2021-02-22 13:29:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.vivareal.com.br/robots.txt> (referer: None)
2021-02-22 13:29:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.vivareal.com.br/venda/minas-gerais/belo-horizonte/> (referer: None)
2021-02-22 13:29:20 [scrapy.spidermiddlewares.offsite] DEBUG: Filtered offsite request to 'www.vivareal.com.br': <GET https://www.vivareal.com.br/venda/minas-gerais/belo-horizonte/#pagina=2>
2021-02-22 13:29:20 [scrapy.core.engine] INFO: Closing spider (finished)
2021-02-22 13:29:20 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 606,
'downloader/request_count': 2,
'downloader/request_method_count/GET': 2,
'downloader/response_bytes': 156997,
'downloader/response_count': 2,
'downloader/response_status_count/200': 2,
'elapsed_time_seconds': 1.87473,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2021, 2, 22, 16, 29, 20, 372722),
'log_count/DEBUG': 3,
'log_count/INFO': 10,
'memusage/max': 54456320,
'memusage/startup': 54456320,
'offsite/domains': 1,
'offsite/filtered': 34,
'request_depth_max': 1,
'response_received_count': 2,
'robotstxt/request_count': 1,
'robotstxt/response_count': 1,
'robotstxt/response_status_count/200': 1,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'start_time': datetime.datetime(2021, 2, 22, 16, 29, 18, 497992)}
2021-02-22 13:29:20 [scrapy.core.engine] INFO: Spider closed (finished)
2021-02-22 13:29:18[scrapy.utils.log]信息：scrapy 2.4.1已启动（bot:properties）
2021-02-22 13:29:18[scrapy.utils.log]信息：版本：lxml 4.5.2.0，libxml2.9.10，cssselect 1.1.0，parsel 1.6.0，w3lib 1.22.0，Twisted 20.3.0，Python 3.8.1（默认，2020年2月9日，21:34:32）-[GCC 7.4.0]，pyOpenSSL 20.0.1（OpenSSL 1.1.1.1i 8 2020年12月8日），密码学3.1，平台Linux-4.15.0-135-x86-64-BCU-27
2021-02-22 13:29:18[scrapy.utils.log]调试：使用reactor:twisted.internet.epollreactor.epollreactor
2021-02-22 13:29:18[刮屑爬虫]信息：覆盖设置：
{'BOT_NAME'：'properties'，
“NEWSPIDER_模块”：“properties.spider”，
“机器人服从”：没错，
'SPIDER_MODULES'：['properties.SPIDER']}
2021-02-22 13:29:18[scrapy.extensions.telnet]信息：telnet密码：3790c3525890efea
2021-02-22 13:29:18[scrapy.middleware]信息：启用的扩展：
['scrapy.extensions.corestats.corestats'，
'scrapy.extensions.telnet.TelnetConsole'，
'scrapy.extensions.memusage.MemoryUsage'，
'scrapy.extensions.logstats.logstats']
2021-02-22 13:29:18[scrapy.middleware]信息：启用的下载程序中间件：
['scrapy.downloaderMiddleware.robotstxt.RobotsTxtMiddleware'，
'scrapy.downloaderMiddleware.httpauth.HttpAuthMiddleware'，
'scrapy.downloadermiddleware.downloadtimeout.DownloadTimeoutMiddleware'，
'scrapy.downloaderMiddleware.defaultheaders.DefaultHeadersMiddleware'，
'scrapy.DownloaderMiddleware.useragent.UserAgentMiddleware'，
'scrapy.DownloaderMiddleware.retry.RetryMiddleware'，
'scrapy.DownloaderMiddleware.redirect.MetaRefreshMiddleware'，
'scrapy.DownloaderMiddleware.httpcompression.HttpCompressionMiddleware'，
'scrapy.DownloaderMiddleware.redirect.RedirectMiddleware'，
“scrapy.DownloaderMiddleware.cookies.CookiesMiddleware”，
'scrapy.downloadermiddleware.httpproxy.HttpProxyMiddleware'，
'scrapy.downloadermiddleware.stats.DownloaderStats']
2021-02-22 13:29:18[scrapy.middleware]信息：启用的蜘蛛中间件：
['scrapy.spidermiddleware.httperror.httperror中间件'，
'刮皮.SpiderMiddleware.场外.场外Iddleware'，
“scrapy.Spidermiddleware.referer.RefererMiddleware”，
'scrapy.spiderMiddleware.urllength.UrlLengthMiddleware'，
'scrapy.spidermiddleware.depth.DepthMiddleware']
2021-02-22 13:29:18[scrapy.middleware]信息：启用的项目管道：
[]
2021-02-22 13:29:18[刮屑.堆芯.发动机]信息：十字轴已打开
2021-02-22 13:29:18[scrapy.extensions.logstats]信息：爬网0页（0页/分钟），爬网0项（0项/分钟）
2021-02-22 13:29:18[scrapy.extensions.telnet]信息：telnet控制台监听127.0.0.1:6023
2021-02-22 13:29:19[scrapy.core.engine]调试：爬网（200）（参考：无）
2021-02-22 13:29:20[scrapy.core.engine]调试：爬网（200）（参考：无）
2021-02-22 13:29:20[scrapy.spidermiddleware.offsite]调试：过滤到“www.vivareal.com.br”的非现场请求：
2021-02-22 13:29:20[刮屑芯发动机]信息：关闭卡盘（完成）
2021-02-22 13:29:20[scrapy.statscollectors]信息：转储scrapy统计信息：
{'downloader/request_bytes'：606，
“下载程序/请求计数”：2，
“下载器/请求\方法\计数/获取”：2，
“downloader/response_字节”：156997，
“下载程序/响应计数”：2，
“下载程序/响应状态\计数/200”：2，
“已用时间秒”：1.87473，
“完成原因”：“完成”，
“完成时间”：datetime.datetime（2021,2,22,16,29,20,372722），
“日志计数/调试”：3，
“日志计数/信息”：10，
“memusage/max”：54456320，
“memusage/startup”：54456320，
“异地/域”：1，
“场外/过滤”：34，
“请求深度最大值”：1，
“响应\u已收到\u计数”：2，
“机器人文本/请求计数”：1，
“机器人文本/响应计数”：1，
“robotstxt/response\u status\u count/200”：1，
“调度程序/出列”：1，
“调度程序/出列/内存”：1，
“调度程序/排队”：1，
“调度程序/排队/内存”：1，
“开始时间”：datetime.datetime（2021,2,22,16,29,18,497992）}
2021-02-22 13:29:20[刮屑芯发动机]信息：十字轴关闭（完成）
2021-02-22 13:31:47[scrapy.utils.log]信息：scrapy 2.4.1已启动（bot:property）
2021-02-22 13:29:18 [scrapy.utils.log] INFO: Scrapy 2.4.1 started (bot: properties)
2021-02-22 13:29:18 [scrapy.utils.log] INFO: Versions: lxml 4.5.2.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 20.3.0, Python 3.8.1 (default, Feb 9 2020, 21:34:32) - [GCC 7.4.0], pyOpenSSL 20.0.1 (OpenSSL 1.1.1i 8 Dec 2020), cryptography 3.3.1, Platform Linux-4.15.0-135-generic-x86_64-with-glibc2.27
2021-02-22 13:29:18 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.epollreactor.EPollReactor
2021-02-22 13:29:18 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'properties',
'NEWSPIDER_MODULE': 'properties.spiders',
'ROBOTSTXT_OBEY': True,
'SPIDER_MODULES': ['properties.spiders']}
2021-02-22 13:29:18 [scrapy.extensions.telnet] INFO: Telnet Password: 3790c3525890efea
2021-02-22 13:29:18 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.memusage.MemoryUsage',
'scrapy.extensions.logstats.LogStats']
2021-02-22 13:29:18 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2021-02-22 13:29:18 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2021-02-22 13:29:18 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2021-02-22 13:29:18 [scrapy.core.engine] INFO: Spider opened
2021-02-22 13:29:18 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2021-02-22 13:29:18 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2021-02-22 13:29:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.vivareal.com.br/robots.txt> (referer: None)
2021-02-22 13:29:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.vivareal.com.br/venda/minas-gerais/belo-horizonte/> (referer: None)
2021-02-22 13:29:20 [scrapy.spidermiddlewares.offsite] DEBUG: Filtered offsite request to 'www.vivareal.com.br': <GET https://www.vivareal.com.br/venda/minas-gerais/belo-horizonte/#pagina=2>
2021-02-22 13:29:20 [scrapy.core.engine] INFO: Closing spider (finished)
2021-02-22 13:29:20 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 606,
'downloader/request_count': 2,
'downloader/request_method_count/GET': 2,
'downloader/response_bytes': 156997,
'downloader/response_count': 2,
'downloader/response_status_count/200': 2,
'elapsed_time_seconds': 1.87473,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2021, 2, 22, 16, 29, 20, 372722),
'log_count/DEBUG': 3,
'log_count/INFO': 10,
'memusage/max': 54456320,
'memusage/startup': 54456320,
'offsite/domains': 1,
'offsite/filtered': 34,
'request_depth_max': 1,
'response_received_count': 2,
'robotstxt/request_count': 1,
'robotstxt/response_count': 1,
'robotstxt/response_status_count/200': 1,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'start_time': datetime.datetime(2021, 2, 22, 16, 29, 18, 497992)}
2021-02-22 13:29:20 [scrapy.core.engine] INFO: Spider closed (finished)
2021-02-22 13:31:47 [scrapy.utils.log] INFO: Scrapy 2.4.1 started (bot: properties)
2021-02-22 13:31:47 [scrapy.utils.log] INFO: Versions: lxml 4.5.2.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 20.3.0, Python 3.8.1 (default, Feb 9 2020, 21:34:32) - [GCC 7.4.0], pyOpenSSL 20.0.1 (OpenSSL 1.1.1i 8 Dec 2020), cryptography 3.3.1, Platform Linux-4.15.0-135-generic-x86_64-with-glibc2.27
2021-02-22 13:31:47 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.epollreactor.EPollReactor
2021-02-22 13:31:47 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'properties',
'NEWSPIDER_MODULE': 'properties.spiders',
'ROBOTSTXT_OBEY': True,
'SPIDER_MODULES': ['properties.spiders']}
2021-02-22 13:31:47 [scrapy.extensions.telnet] INFO: Telnet Password: 65a5f31c8dda80fa
2021-02-22 13:31:47 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.memusage.MemoryUsage',
'scrapy.extensions.logstats.LogStats']
2021-02-22 13:31:47 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2021-02-22 13:31:47 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2021-02-22 13:31:47 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2021-02-22 13:31:47 [scrapy.core.engine] INFO: Spider opened
2021-02-22 13:31:47 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2021-02-22 13:31:47 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2021-02-22 13:31:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.vivareal.com.br/robots.txt> (referer: None)
2021-02-22 13:31:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.vivareal.com.br/venda/minas-gerais/belo-horizonte/> (referer: None)
2021-02-22 13:31:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.vivareal.com.br/venda/minas-gerais/belo-horizonte/#pagina=2> (referer: https://www.vivareal.com.br/venda/minas-gerais/belo-horizonte/)
2021-02-22 13:31:50 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET https://www.vivareal.com.br/venda/minas-gerais/belo-horizonte/#pagina=2> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)
2021-02-22 13:31:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.vivareal.com.br/imovel/apartamento-1-quartos-funcionarios-bairros-belo-horizonte-com-garagem-41m2-venda-RS330000-id-2510414426/> (referer: https://www.vivareal.com.br/venda/minas-gerais/belo-horizonte/)
2021-02-22 13:31:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.vivareal.com.br/imovel/apartamento-3-quartos-nova-granada-bairros-belo-horizonte-com-garagem-74m2-venda-RS499000-id-2509923918/> (referer: https://www.vivareal.com.br/venda/minas-gerais/belo-horizonte/)
2021-02-22 13:31:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.vivareal.com.br/imovel/apartamento-4-quartos-serra-bairros-belo-horizonte-com-garagem-246m2-venda-RS1950000-id-2510579983/> (referer: https://www.vivareal.com.br/venda/minas-gerais/belo-horizonte/)
2021-02-22 13:31:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.vivareal.com.br/imovel/casa-3-quartos-sao-geraldo-bairros-belo-horizonte-com-garagem-120m2-venda-RS460000-id-2484383176/> (referer: https://www.vivareal.com.br/venda/minas-gerais/belo-horizonte/)
2021-02-22 13:31:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.vivareal.com.br/imovel/apartamento-4-quartos-savassi-bairros-belo-horizonte-com-garagem-206m2-venda-RS1790000-id-2503711314/> (referer: https://www.vivareal.com.br/venda/minas-gerais/belo-horizonte/)
2021-02-22 13:31:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.vivareal.com.br/imovel/apartamento-2-quartos-paqueta-bairros-belo-horizonte-com-garagem-60m2-venda-RS260000-id-2479637684/> (referer: https://www.vivareal.com.br/venda/minas-gerais/belo-horizonte/)
2021-02-22 13:31:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.vivareal.com.br/imovel/apartamento-3-quartos-savassi-bairros-belo-horizonte-com-garagem-107m2-venda-RS1250000-id-2506122689/> (referer: https://www.vivareal.com.br/venda/minas-gerais/belo-horizonte/)
2021-02-22 13:31:50 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.vivareal.com.br/imovel/apartamento-1-quartos-funcionarios-bairros-belo-horizonte-com-garagem-41m2-venda-RS330000-id-2510414426/> (referer: https://www.vivareal.com.br/venda/minas-gerais/belo-horizonte/)
Traceback (most recent call last):
File "/usr/lib/python3.8/site-packages/scrapy/utils/defer.py", line 120, in iter_errback
yield next(it)
File "/usr/lib/python3.8/site-packages/scrapy/utils/python.py", line 353, in __next__
return next(self.data)
File "/usr/lib/python3.8/site-packages/scrapy/utils/python.py", line 353, in __next__
return next(self.data)
File "/usr/lib/python3.8/site-packages/scrapy/core/spidermw.py", line 62, in _evaluate_iterable
for r in iterable:
File "/usr/lib/python3.8/site-packages/scrapy/spidermiddlewares/offsite.py", line 29, in process_spider_output
for x in result:
File "/usr/lib/python3.8/site-packages/scrapy/core/spidermw.py", line 62, in _evaluate_iterable
for r in iterable:
File "/usr/lib/python3.8/site-packages/scrapy/spidermiddlewares/referer.py", line 340, in <genexpr>
return (_set_referer(r) for r in result or ())
File "/usr/lib/python3.8/site-packages/scrapy/core/spidermw.py", line 62, in _evaluate_iterable
for r in iterable:
File "/usr/lib/python3.8/site-packages/scrapy/spidermiddlewares/urllength.py", line 37, in <genexpr>
return (r for r in result or () if _filter(r))
File "/usr/lib/python3.8/site-packages/scrapy/core/spidermw.py", line 62, in _evaluate_iterable
for r in iterable:
File "/usr/lib/python3.8/site-packages/scrapy/spidermiddlewares/depth.py", line 58, in <genexpr>
return (r for r in result or () if _filter(r))
File "/usr/lib/python3.8/site-packages/scrapy/core/spidermw.py", line 62, in _evaluate_iterable
for r in iterable:
File "/usr/lib/python3.8/site-packages/scrapy/spiders/crawl.py", line 114, in _parse_response
cb_res = callback(response, **cb_kwargs) or ()
File "/home/leomaffei/properties/properties/spiders/spider.py", line 28, in parse_item
l.add_xpath('url', 'a/@href', )
File "/usr/lib/python3.8/site-packages/itemloaders/__init__.py", line 350, in add_xpath
self.add_value(field_name, values, *processors, **kw)
File "/usr/lib/python3.8/site-packages/itemloaders/__init__.py", line 190, in add_value
self._add_value(field_name, value)
File "/usr/lib/python3.8/site-packages/itemloaders/__init__.py", line 208, in _add_value
processed_value = self._process_input_value(field_name, value)
File "/usr/lib/python3.8/site-packages/itemloaders/__init__.py", line 312, in _process_input_value
proc = self.get_input_processor(field_name)
File "/usr/lib/python3.8/site-packages/itemloaders/__init__.py", line 290, in get_input_processor
proc = self._get_item_field_attr(
File "/usr/lib/python3.8/site-packages/itemloaders/__init__.py", line 308, in _get_item_field_attr
field_meta = ItemAdapter(self.item).get_field_meta(field_name)
File "/usr/lib/python3.8/site-packages/itemadapter/adapter.py", line 235, in get_field_meta
return self.adapter.get_field_meta(field_name)
File "/usr/lib/python3.8/site-packages/itemadapter/adapter.py", line 161, in get_field_meta
return MappingProxyType(self.item.fields[field_name])
KeyError: 'url'
2021-02-22 13:31:50 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.vivareal.com.br/imovel/apartamento-3-quartos-nova-granada-bairros-belo-horizonte-com-garagem-74m2-venda-RS499000-id-2509923918/> (referer: https://www.vivareal.com.br/venda/minas-gerais/belo-horizonte/)
Traceback (most recent call last):
File "/usr/lib/python3.8/site-packages/scrapy/utils/defer.py", line 120, in iter_errback
yield next(it)
File "/usr/lib/python3.8/site-packages/scrapy/utils/python.py", line 353, in __next__
return next(self.data)
File "/usr/lib/python3.8/site-packages/scrapy/utils/python.py", line 353, in __next__
return next(self.data)
File "/usr/lib/python3.8/site-packages/scrapy/core/spidermw.py", line 62, in _evaluate_iterable
for r in iterable:
File "/usr/lib/python3.8/site-packages/scrapy/spidermiddlewares/offsite.py", line 29, in process_spider_output
for x in result:
File "/usr/lib/python3.8/site-packages/scrapy/core/spidermw.py", line 62, in _evaluate_iterable
for r in iterable:
File "/usr/lib/python3.8/site-packages/scrapy/spidermiddlewares/referer.py", line 340, in <genexpr>
return (_set_referer(r) for r in result or ())
File "/usr/lib/python3.8/site-packages/scrapy/core/spidermw.py", line 62, in _evaluate_iterable
for r in iterable:
File "/usr/lib/python3.8/site-packages/scrapy/spidermiddlewares/urllength.py", line 37, in <genexpr>
return (r for r in result or () if _filte
...