Python 使用pandas读取csv文件时出现的编码问题
我试图在一个大的csv中阅读,其中包含一些不规则的行。我一直收到一些错误,说明“预期113个字符,遇到114个”。我尝试丢弃定界字符数大于预期的行,但随后遇到另一个错误。代码和错误如下所示:Python 使用pandas读取csv文件时出现的编码问题,python,pandas,character-encoding,Python,Pandas,Character Encoding,我试图在一个大的csv中阅读,其中包含一些不规则的行。我一直收到一些错误,说明“预期113个字符,遇到114个”。我尝试丢弃定界字符数大于预期的行,但随后遇到另一个错误。代码和错误如下所示: import pandas as pd import csv f_name = 'some_file.txt' with open(f_name,'r') as file: lines = file.readlines() bad_lines= [] for i,line in enum
import pandas as pd
import csv
f_name = 'some_file.txt'
with open(f_name,'r') as file:
lines = file.readlines()
bad_lines= []
for i,line in enumerate(lines):
if line.count('\t') != 112:
bad_lines.append(i)
df = pd.read_csv(f_name,header=0,sep='\t',skiprows=bad_lines,error_bad_lines=False)
df.head()
这将产生以下警告和错误:
Skipping line 3332366: expected 113 fields, saw 168
Skipping line 3829308: expected 113 fields, saw 114
Skipping line 4819090: expected 113 fields, saw 114
Skipping line 4832856: expected 113 fields, saw 114
Skipping line 5170065: expected 113 fields, saw 114
Skipping line 5198811: expected 113 fields, saw 168
---------------------------------------------------------------------------
UnicodeDecodeError Traceback (most recent call last)
<ipython-input-19-ea2fca7ce11a> in <module>()
1 df = pd.read_csv(f_name,header=0,sep='\t',skiprows=bad_lines,error_bad_lines=False)
----> 2 df.head()
/Library/Python/2.7/site-packages/IPython/core/displayhook.pyc in __call__(self, result)
236 self.write_format_data(format_dict, md_dict)
237 self.log_output(format_dict)
--> 238 self.finish_displayhook()
239
240 def cull_cache(self):
/Library/Python/2.7/site-packages/IPython/kernel/zmq/displayhook.pyc in finish_displayhook(self)
70 sys.stderr.flush()
71 if self.msg['content']['data']:
---> 72 self.session.send(self.pub_socket, self.msg, ident=self.topic)
73 self.msg = None
74
/Library/Python/2.7/site-packages/IPython/kernel/zmq/session.pyc in send(self, stream, msg_or_type, content, parent, ident, buffers, track, header, metadata)
647 if self.adapt_version:
648 msg = adapt(msg, self.adapt_version)
--> 649 to_send = self.serialize(msg, ident)
650 to_send.extend(buffers)
651 longest = max([ len(s) for s in to_send ])
/Library/Python/2.7/site-packages/IPython/kernel/zmq/session.pyc in serialize(self, msg, ident)
551 content = self.none
552 elif isinstance(content, dict):
--> 553 content = self.pack(content)
554 elif isinstance(content, bytes):
555 # content is already packed, as in a relayed message
/Library/Python/2.7/site-packages/IPython/kernel/zmq/session.pyc in <lambda>(obj)
83 # disallow nan, because it's not actually valid JSON
84 json_packer = lambda obj: jsonapi.dumps(obj, default=date_default,
---> 85 ensure_ascii=False, allow_nan=False,
86 )
87 json_unpacker = lambda s: jsonapi.loads(s)
/Library/Python/2.7/site-packages/zmq/utils/jsonapi.pyc in dumps(o, **kwargs)
38 kwargs['separators'] = (',', ':')
39
---> 40 s = jsonmod.dumps(o, **kwargs)
41
42 if isinstance(s, unicode):
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.pyc in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, encoding, default, sort_keys, **kw)
248 check_circular=check_circular, allow_nan=allow_nan, indent=indent,
249 separators=separators, encoding=encoding, default=default,
--> 250 sort_keys=sort_keys, **kw).encode(obj)
251
252
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.pyc in encode(self, o)
208 if not isinstance(chunks, (list, tuple)):
209 chunks = list(chunks)
--> 210 return ''.join(chunks)
211
212 def iterencode(self, o, _one_shot=False):
UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 4: ordinal not in range(128)
我本以为我的预防措施会删除这些行,但我也不完全理解如何解决编码问题。完全有可能是奇怪的字符进入了数据,但我没有处理编码问题的经验,我也无法适应我在网上找到的任何其他解决方案
提前感谢您的帮助 您必须发布原始输入数据,理想情况下是有问题的行来重现您的问题,您可以尝试:df=pd.read_csvf_name,header=0,sep='\s+',skiprows=bad_行,error_bad_行=False,以防您的分隔符不仅仅是制表符,您还想用header=0做什么?如果您的csv没有标题,那么您希望header=None,您所做的是指定第一行是headerOne解决方案,它将切换到Python 3,这将帮助您消除95%的编码问题。我不认为这是编码问题,而不是错误的csv,或者错误的解析参数被传递到read_csvI之前实际上没有注意到这一点,但是错误不在read语句中,而是在df.head语句中-但是,如果我写print df.head,它工作得很好。我相信我会继续了解更多,但我不完全理解: