Python 如何避免刮痧症
在我的Python 如何避免刮痧症,python,unicode,scrapy,Python,Unicode,Scrapy,在我的parse_项回调中有以下代码: sel = Selector(response) item['name'] = sel.xpath('//div[@class="productDescriptionBlock"]/h2/text()').extract()[0] return item 但是我得到了UnicodeEncodeError: exceptions.UnicodeEncodeError: 'charmap' codec can't encode character u'\uf
parse_项
回调中有以下代码:
sel = Selector(response)
item['name'] = sel.xpath('//div[@class="productDescriptionBlock"]/h2/text()').extract()[0]
return item
但是我得到了UnicodeEncodeError:
exceptions.UnicodeEncodeError: 'charmap' codec can't encode character u'\uff01' in position 271761: character maps to <undefined>
我以前见过这个。如果我没有错,您正在使用规则的链接提取器中的
restrict\xpaths
参数
可能的解决办法是:
restrict\u xpath
。这是因为页面内容包含声明的编码中未定义的字符异常的完整回溯是什么?您更新的错误看起来好像您正在抓取的站点报告了错误的编码(响应中的HTTP头告诉您要使用什么编解码器,而
response.encoding
就是该值)。我们必须看看是否有人有特定的知识,可以提供一个这样的网站变通。该网站的元标签中有这一点。当然可以,但是我认为response.encoding
属性反映了内容类型:text/html;字符集=..
值。
Traceback (most recent call last):
File "/home/scraper/.fakeroot/lib/python2.7/site-packages/twisted/internet/base.py", line 824, in runUntilCurrent
call.func(*call.args, **call.kw)
File "/home/scraper/.fakeroot/lib/python2.7/site-packages/twisted/internet/task.py", line 638, in _tick
taskObj._oneWorkUnit()
File "/home/scraper/.fakeroot/lib/python2.7/site-packages/twisted/internet/task.py", line 484, in _oneWorkUnit
result = next(self._iterator)
File "/home/scraper/.fakeroot/lib/python2.7/site-packages/scrapy/utils/defer.py", line 57, in <genexpr>
work = (callable(elem, *args, **named) for elem in iterable)
--- <exception caught here> ---
File "/home/scraper/.fakeroot/lib/python2.7/site-packages/scrapy/utils/defer.py", line 96, in iter_errback
yield next(it)
File "/home/scraper/.fakeroot/lib/python2.7/site-packages/scrapy/contrib/spidermiddleware/offsite.py", line 23, in process_spider_output
for x in result:
File "/home/scraper/.fakeroot/lib/python2.7/site-packages/scrapy/contrib/spidermiddleware/referer.py", line 22, in <genexpr>
return (_set_referer(r) for r in result or ())
File "/home/scraper/.fakeroot/lib/python2.7/site-packages/scrapy/contrib/spidermiddleware/urllength.py", line 33, in <genexpr>
return (r for r in result or () if _filter(r))
File "/home/scraper/.fakeroot/lib/python2.7/site-packages/scrapy/contrib/spidermiddleware/depth.py", line 50, in <genexpr>
return (r for r in result or () if _filter(r))
File "/home/scraper/.fakeroot/lib/python2.7/site-packages/scrapy/contrib/spiders/crawl.py", line 73, in _parse_response
for request_or_item in self._requests_to_follow(response):
File "/home/scraper/.fakeroot/lib/python2.7/site-packages/scrapy/contrib/spiders/crawl.py", line 52, in _requests_to_follow
links = [l for l in rule.link_extractor.extract_links(response) if l not in seen]
File "/home/scraper/.fakeroot/lib/python2.7/site-packages/scrapy/contrib/linkextractors/sgml.py", line 124, in extract_links
).encode(response.encoding)
File "/home/scraper/.fakeroot/lib/python2.7/encodings/cp1252.py", line 12, in encode
return codecs.charmap_encode(input,errors,encoding_table)
exceptions.UnicodeEncodeError: 'charmap' codec can't encode character u'\x99' in position 349751: character maps to <undefined>