select查询中的HDF5存储unicode错误

我有从该文件读取的unicode数据：

Mdt,Doccompra,OrgC,Cen,NumP,Criadopor,Dtcriacao,Fornecedor,P,Fun
400,8751215432,2581,,1,MIGRAÇÃO,01.10.2004,75852214,,TD
400,5464282154,9874,,1,MIGRAÇÃO,01.10.2004,78995411,,FO

我有两个问题：

当我试图查询这个unicode数据时，我得到一个

UnicodeDecodeError

：

Traceback (most recent call last):
  File "<ipython-input-1-4423dceb2b1d>", line 1, in <module>
    runfile('C:/Users/u5en/Documents/SAP/Programação/Problema HDF.py', wdir='C:/Users/u5en/Documents/SAP/Programação')

  File "C:\Users\u5en\AppData\Local\Continuum\Anaconda3\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 580, in runfile
    execfile(filename, namespace)

  File "C:\Users\u5en\AppData\Local\Continuum\Anaconda3\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 48, in execfile
    exec(compile(open(filename, 'rb').read(), filename, 'exec'), namespace)

  File "C:/Users/u5en/Documents/SAP/Programação/Problema HDF.py", line 15, in <module>
    store.select("EKKA", "columns=['Mdt', 'Fornecedor']")

  File "C:\Users\u5en\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\io\pytables.py", line 665, in select
    return it.get_result()

  File "C:\Users\u5en\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\io\pytables.py", line 1359, in get_result
    results = self.func(self.start, self.stop, where)

  File "C:\Users\u5en\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\io\pytables.py", line 658, in func
    columns=columns, **kwargs)

  File "C:\Users\u5en\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\io\pytables.py", line 3968, in read
    if not self.read_axes(where=where, **kwargs):

  File "C:\Users\u5en\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\io\pytables.py", line 3201, in read_axes
    a.convert(values, nan_rep=self.nan_rep, encoding=self.encoding)

  File "C:\Users\u5en\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\io\pytables.py", line 2058, in convert
    self.data, nan_rep=nan_rep, encoding=encoding)

  File "C:\Users\u5en\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\io\pytables.py", line 4359, in _unconvert_string_array
    data = f(data)

  File "C:\Users\u5en\AppData\Local\Continuum\Anaconda3\lib\site-packages\numpy\lib\function_base.py", line 1700, in __call__
    return self._vectorize_call(func=func, args=vargs)

  File "C:\Users\u5en\AppData\Local\Continuum\Anaconda3\lib\site-packages\numpy\lib\function_base.py", line 1769, in _vectorize_call
    outputs = ufunc(*inputs)

  File "C:\Users\u5en\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\io\pytables.py", line 4358, in <lambda>
    f = np.vectorize(lambda x: x.decode(encoding), otypes=[np.object])

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 7: unexpected end of data

在sqlite中执行此操作会更好吗

环境：

• Windows 7 64位
• 熊猫15.2
• NumPy 1.9.2

---

因此，在Windows7上的Python2.7和0.15.2下，一切都按预期进行，无需编码。然而，在Python3.4上，以下内容对我来说很有用。显然，有些字符在“utf-8”中不可表示拉丁1'编码通常可以解决这些问题。注意，我必须首先用这种编码读取csv

>>> df = pd.read_csv('../../test.csv',encoding='latin1')
>>> df
   Mdt   Doccompra  OrgC  Cen  NumP Criadopor   Dtcriacao  Fornecedor   P Fun
0  400  8751215432  2581  NaN     1  MIGRAÇ\xc3O  01.10.2004    75852214 NaN  TD
1  400  5464282154  9874  NaN     1  MIGRAÇ\xc3O  01.10.2004    78995411 NaN  FO

此外，必须在

append/put

调用中指定编码，而不是在打开存储时指定编码

>>> df.to_hdf('test.h5','df',format='table',mode='w',data_columns=True,encoding='latin1')

>>> pd.read_hdf('test.h5','df')
   Mdt   Doccompra  OrgC  Cen  NumP Criadopor   Dtcriacao  Fornecedor   P Fun
0  400  8751215432  2581  NaN     1  MIGRAÇ\xc3O  01.10.2004    75852214 NaN  TD
1  400  5464282154  9874  NaN     1  MIGRAÇ\xc3O  01.10.2004    78995411 NaN  FO

一旦写入编码，则在读取时无需指定编码

>>> df.to_hdf('test.h5','df',format='table',mode='w',data_columns=True,encoding='latin1')

>>> pd.read_hdf('test.h5','df')
   Mdt   Doccompra  OrgC  Cen  NumP Criadopor   Dtcriacao  Fornecedor   P Fun
0  400  8751215432  2581  NaN     1  MIGRAÇ\xc3O  01.10.2004    75852214 NaN  TD
1  400  5464282154  9874  NaN     1  MIGRAÇ\xc3O  01.10.2004    78995411 NaN  FO