Python 读取pandas中的csv文件时出错[CParserError:错误标记化数据。C错误:捕获到缓冲区溢出-可能是输入文件格式错误。]
所以我尝试从一个文件夹中读取所有csv文件,然后将它们连接起来创建一个大的csv(所有文件的结构都是相同的),保存它,然后再次读取。所有这些都是用熊猫做的。读取时出错。我附上下面的代码和错误Python 读取pandas中的csv文件时出错[CParserError:错误标记化数据。C错误:捕获到缓冲区溢出-可能是输入文件格式错误。],python,csv,pandas,Python,Csv,Pandas,所以我尝试从一个文件夹中读取所有csv文件,然后将它们连接起来创建一个大的csv(所有文件的结构都是相同的),保存它,然后再次读取。所有这些都是用熊猫做的。读取时出错。我附上下面的代码和错误 import pandas as pd import numpy as np import glob path =r'somePath' # use your path allFiles = glob.glob(path + "/*.csv") frame = pd.DataFrame() list_ =
import pandas as pd
import numpy as np
import glob
path =r'somePath' # use your path
allFiles = glob.glob(path + "/*.csv")
frame = pd.DataFrame()
list_ = []
for file_ in allFiles:
df = pd.read_csv(file_,index_col=None, header=0)
list_.append(df)
store = pd.concat(list_)
store.to_csv("C:\work\DATA\Raw_data\\store.csv", sep=',', index= False)
store1 = pd.read_csv("C:\work\DATA\Raw_data\\store.csv", sep=',')
错误:-
CParserError Traceback (most recent call last)
<ipython-input-48-2983d97ccca6> in <module>()
----> 1 store1 = pd.read_csv("C:\work\DATA\Raw_data\\store.csv", sep=',')
C:\Users\armsharm\AppData\Local\Continuum\Anaconda\lib\site-packages\pandas\io\parsers.pyc in parser_f(filepath_or_buffer, sep, dialect, compression, doublequote, escapechar, quotechar, quoting, skipinitialspace, lineterminator, header, index_col, names, prefix, skiprows, skipfooter, skip_footer, na_values, na_fvalues, true_values, false_values, delimiter, converters, dtype, usecols, engine, delim_whitespace, as_recarray, na_filter, compact_ints, use_unsigned, low_memory, buffer_lines, warn_bad_lines, error_bad_lines, keep_default_na, thousands, comment, decimal, parse_dates, keep_date_col, dayfirst, date_parser, memory_map, float_precision, nrows, iterator, chunksize, verbose, encoding, squeeze, mangle_dupe_cols, tupleize_cols, infer_datetime_format, skip_blank_lines)
472 skip_blank_lines=skip_blank_lines)
473
--> 474 return _read(filepath_or_buffer, kwds)
475
476 parser_f.__name__ = name
C:\Users\armsharm\AppData\Local\Continuum\Anaconda\lib\site-packages\pandas\io\parsers.pyc in _read(filepath_or_buffer, kwds)
258 return parser
259
--> 260 return parser.read()
261
262 _parser_defaults = {
C:\Users\armsharm\AppData\Local\Continuum\Anaconda\lib\site-packages\pandas\io\parsers.pyc in read(self, nrows)
719 raise ValueError('skip_footer not supported for iteration')
720
--> 721 ret = self._engine.read(nrows)
722
723 if self.options.get('as_recarray'):
C:\Users\armsharm\AppData\Local\Continuum\Anaconda\lib\site-packages\pandas\io\parsers.pyc in read(self, nrows)
1168
1169 try:
-> 1170 data = self._reader.read(nrows)
1171 except StopIteration:
1172 if nrows is None:
pandas\parser.pyx in pandas.parser.TextReader.read (pandas\parser.c:7544)()
pandas\parser.pyx in pandas.parser.TextReader._read_low_memory (pandas\parser.c:7784)()
pandas\parser.pyx in pandas.parser.TextReader._read_rows (pandas\parser.c:8401)()
pandas\parser.pyx in pandas.parser.TextReader._tokenize_rows (pandas\parser.c:8275)()
pandas\parser.pyx in pandas.parser.raise_parser_error (pandas\parser.c:20691)()
CParserError: Error tokenizing data. C error: Buffer overflow caught - possible malformed input file.
Error Traceback (most recent call last)
<ipython-input-36-9249469f31a6> in <module>()
1 with open('C:\work\DATA\Raw_data\\store.csv', 'rb') as f:
2 reader = csv.reader(f)
----> 3 l = list(reader)
Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?
错误:-
CParserError Traceback (most recent call last)
<ipython-input-48-2983d97ccca6> in <module>()
----> 1 store1 = pd.read_csv("C:\work\DATA\Raw_data\\store.csv", sep=',')
C:\Users\armsharm\AppData\Local\Continuum\Anaconda\lib\site-packages\pandas\io\parsers.pyc in parser_f(filepath_or_buffer, sep, dialect, compression, doublequote, escapechar, quotechar, quoting, skipinitialspace, lineterminator, header, index_col, names, prefix, skiprows, skipfooter, skip_footer, na_values, na_fvalues, true_values, false_values, delimiter, converters, dtype, usecols, engine, delim_whitespace, as_recarray, na_filter, compact_ints, use_unsigned, low_memory, buffer_lines, warn_bad_lines, error_bad_lines, keep_default_na, thousands, comment, decimal, parse_dates, keep_date_col, dayfirst, date_parser, memory_map, float_precision, nrows, iterator, chunksize, verbose, encoding, squeeze, mangle_dupe_cols, tupleize_cols, infer_datetime_format, skip_blank_lines)
472 skip_blank_lines=skip_blank_lines)
473
--> 474 return _read(filepath_or_buffer, kwds)
475
476 parser_f.__name__ = name
C:\Users\armsharm\AppData\Local\Continuum\Anaconda\lib\site-packages\pandas\io\parsers.pyc in _read(filepath_or_buffer, kwds)
258 return parser
259
--> 260 return parser.read()
261
262 _parser_defaults = {
C:\Users\armsharm\AppData\Local\Continuum\Anaconda\lib\site-packages\pandas\io\parsers.pyc in read(self, nrows)
719 raise ValueError('skip_footer not supported for iteration')
720
--> 721 ret = self._engine.read(nrows)
722
723 if self.options.get('as_recarray'):
C:\Users\armsharm\AppData\Local\Continuum\Anaconda\lib\site-packages\pandas\io\parsers.pyc in read(self, nrows)
1168
1169 try:
-> 1170 data = self._reader.read(nrows)
1171 except StopIteration:
1172 if nrows is None:
pandas\parser.pyx in pandas.parser.TextReader.read (pandas\parser.c:7544)()
pandas\parser.pyx in pandas.parser.TextReader._read_low_memory (pandas\parser.c:7784)()
pandas\parser.pyx in pandas.parser.TextReader._read_rows (pandas\parser.c:8401)()
pandas\parser.pyx in pandas.parser.TextReader._tokenize_rows (pandas\parser.c:8275)()
pandas\parser.pyx in pandas.parser.raise_parser_error (pandas\parser.c:20691)()
CParserError: Error tokenizing data. C error: Buffer overflow caught - possible malformed input file.
Error Traceback (most recent call last)
<ipython-input-36-9249469f31a6> in <module>()
1 with open('C:\work\DATA\Raw_data\\store.csv', 'rb') as f:
2 reader = csv.reader(f)
----> 3 l = list(reader)
Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?
错误回溯(最近一次调用上次)
在()
1打开('C:\work\DATA\Raw\u DATA\\store.csv','rb')作为f:
2读卡器=csv读卡器(f)
---->3 l=列表(读卡器)
错误:在不带引号的字段中看到新行字符-是否需要以通用新行模式打开文件?
不是答案,但注释太长(不包括代码格式)
当您在csv模块中读取它时,它会断开,您至少可以找到发生错误的行:
import csv
with open(r"C:\work\DATA\Raw_data\store.csv", 'rb') as f:
reader = csv.reader(f)
linenumber = 1
try:
for row in reader:
linenumber += 1
except Exception as e:
print (("Error line %d: %s %s" % (linenumber, str(type(e)), e.message)))
然后在store.csv中查看该行发生的情况。我发现此错误,原因是pandas用作行终止符的数据中有一些回车“\r”,就好像它是“\n”一样。我想我会在这里发布,因为这可能是出现此错误的常见原因 我找到的解决方案是将lineterminator='\n'添加到read_csv函数中,如下所示:
df_clean = pd.read_csv('test_error.csv',
lineterminator='\n')
如果您使用的是python,并且它是一个大文件,那么您可以使用它
engine='python'
如下所示,应该可以正常工作
df=pd.read\u csv(文件,索引\u col=None,header=0,engine='python')
将目录更改为csv
Corpus = pd.read_csv(r"C:\Users\Dell\Desktop\Dataset.csv",encoding='latin-1')
很抱歉响应太晚,请查看csv。有一些unicode字符,如\r、->等,导致意外转义。在源代码中替换它们就成功了。您的答案在可视化它们方面帮助很大。对于Python 3,只需使用“r”打开文件,即
open('file.csv',r')
,为了解决此问题,只需将文件从excel工作表菜单选项保存为“csv”格式,即可解决此问题。将来,请使用标记来格式化您的帖子和回复。非常贴切,很难从错误信息中找到答案这是一个非常棒的答案!!节省了我很多时间,谢谢!你救了我的命:-)谢谢,这成功了。简单的解决方案,太棒了!这对我有用。