Pandas 将csv文件转换为数据帧
我有以下格式的CSV文件：Pandas 将csv文件转换为数据帧,pandas,Pandas,我有以下格式的CSV文件： DATES, 01-12-2010, 01-12-2010, 01-12-2010, 02-12-2010, 02-12-2010, 02-12-2010 UNITS, Hz, kV, MW, Hz, kV, MW Interval, , , , , , 00:15, 49.82, 33.73755, 34.65, 49.92, 33.9009, 36.33, 00:30, 49.
DATES, 01-12-2010, 01-12-2010, 01-12-2010, 02-12-2010, 02-12-2010, 02-12-2010
UNITS, Hz, kV, MW, Hz, kV, MW
Interval, , , , , ,
00:15, 49.82, 33.73755, 34.65, 49.92, 33.9009, 36.33,
00:30, 49.9, 33.7722, 35.34, 49.89, 33.8382, 37.65,
00:45, 49.94, 33.8316, 33.5, 50.09, 34.07745, 37.41,
01:00, 49.86, 33.94875, 30.91, 50.18, 34.20945, 36.11,
01:15, 49.97, 34.2243, 27.28, 50.11, 34.3596, 33.24,
01:30, 50.02, 34.3332, 26.91, 50.12, 34.452, 31.03,
01:45, 50.01, 34.1286, 31.26, 50, 33.9306, 38.86,
02:00, 50.08, 33.9141, 34.96, 50.14, 33.99165, 38.31,
02:15, 50.07, 33.84975, 35.33, 50.01, 33.9537, 39.78,
02:30, 49.97, 34.0263, 33.63, 50.07, 33.8547, 41.48,
Hz kV MW
DATES_Interval
01-12-2010 00:15 49.82 33.73755 34.65
01-12-2010 00:30 49.9 33.7722 35.34
01-12-2010 00:45 49.94 33.8316 33.5
01-12-2010 01:00 49.86 33.94875 30.91
01-12-2010 01:15 49.97 34.2243 27.28
01-12-2010 01:30 50.02 34.3332 26.91
01-12-2010 01:45 50.01 34.1286 31.26
01-12-2010 02:00 50.08 33.9141 34.96
01-12-2010 02:15 50.07 33.84975 35.33
01-12-2010 02:30 49.97 34.0263 33.63
02-12-2010 00:15 49.92 33.9009 36.33
02-12-2010 00:30 49.89 33.8382 37.65
02-12-2010 00:45 50.09 34.07745 37.41
02-12-2010 01:00 50.09 34.07745 37.41
02-12-2010 01:15 50.11 34.3596 33.24
02-12-2010 01:30 50.12 34.452 31.03
02-12-2010 01:45 50 33.9306 38.86
02-12-2010 02:00 50.14 33.99165 38.31
02-12-2010 02:15 50.01 33.9537 39.78
02-12-2010 02:30 50.07 33.8547 41.48
DATES, 01-12-2010, 01-12-2010, 01-12-2010, 02-12-2010, 02-12-2010, 02-12-2010
UNITS, Hz, kV, MW, Hz, kV, MW
Interval, , , , , ,
00:15, 49.82, 33.73755, 34.65, 49.92, 33.9009, 36.33,
00:30, 49.9, 33.7722, 35.34, 49.89, 33.8382, 37.65,
00:45, 49.94, 33.8316, 33.5, 50.09, 34.07745, 37.41,
01:00, 49.86, 33.94875, 30.91, 50.18, 34.20945, 36.11,
01:15, 49.97, 34.2243, 27.28, 50.11, 34.3596, 33.24,
01:30, 50.02, 34.3332, 26.91, 50.12, 34.452, 31.03,
01:45, 50.01, 34.1286, 31.26, 50, 33.9306, 38.86,
02:00, 50.08, 33.9141, 34.96, 50.14, 33.99165, 38.31,
02:15, 50.07, 33.84975, 35.33, 50.01, 33.9537, 39.78,
02:30, 49.97, 34.0263, 33.63, 50.07, 33.8547, 41.48,
Hz kV MW
DATES_Interval
01-12-2010 00:15 49.82 33.73755 34.65
01-12-2010 00:30 49.9 33.7722 35.34
01-12-2010 00:45 49.94 33.8316 33.5
01-12-2010 01:00 49.86 33.94875 30.91
01-12-2010 01:15 49.97 34.2243 27.28
01-12-2010 01:30 50.02 34.3332 26.91
01-12-2010 01:45 50.01 34.1286 31.26
01-12-2010 02:00 50.08 33.9141 34.96
01-12-2010 02:15 50.07 33.84975 35.33
01-12-2010 02:30 49.97 34.0263 33.63
02-12-2010 00:15 49.92 33.9009 36.33
02-12-2010 00:30 49.89 33.8382 37.65
02-12-2010 00:45 50.09 34.07745 37.41
02-12-2010 01:00 50.09 34.07745 37.41
02-12-2010 01:15 50.11 34.3596 33.24
02-12-2010 01:30 50.12 34.452 31.03
02-12-2010 01:45 50 33.9306 38.86
02-12-2010 02:00 50.14 33.99165 38.31
02-12-2010 02:15 50.01 33.9537 39.78
02-12-2010 02:30 50.07 33.8547 41.48
如何在熊猫身上做到这一点？在熊猫身上做到这一点的关键是方法： 然而，我发现要找到一个你可以使用这个的地方，至少特定的csv是很棘手的。。至少可以说，几乎可以肯定有一种更好的方法可以做到这一点
df_data = pd.read_csv('e.csv', sep=',\s+', header=None, skiprows=3)[range(7)].set_index(0)
df_cols = pd.read_csv('e.csv', sep=',\s+', header=None, nrows=2).set_index(0)[:2] #interval causing problems
df_ = df_cols.append(df_data).T.set_index(['DATES','UNITS','Interval']).T
df = df_.stack(level=0)
df_dates = map(lambda x: pd.to_datetime(' '.join(x[::-1])), df.index)
df.index = df_dates
In [7]: df
Out[7]:
UNITS Hz MW kV
2010-01-12 00:15:00 49.82 34.65 33.73755
2010-02-12 00:15:00 49.92 36.33, 33.9009
2010-01-12 00:30:00 49.9 35.34 33.7722
2010-02-12 00:30:00 49.89 37.65, 33.8382
2010-01-12 00:45:00 49.94 33.5 33.8316
2010-02-12 00:45:00 50.09 37.41, 34.07745
2010-01-12 01:00:00 49.86 30.91 33.94875
2010-02-12 01:00:00 50.18 36.11, 34.20945
2010-01-12 01:15:00 49.97 27.28 34.2243
2010-02-12 01:15:00 50.11 33.24, 34.3596
2010-01-12 01:30:00 50.02 26.91 34.3332
2010-02-12 01:30:00 50.12 31.03, 34.452
2010-01-12 01:45:00 50.01 31.26 34.1286
2010-02-12 01:45:00 50 38.86, 33.9306
2010-01-12 02:00:00 50.08 34.96 33.9141
2010-02-12 02:00:00 50.14 38.31, 33.99165
2010-01-12 02:15:00 50.07 35.33 33.84975
2010-02-12 02:15:00 50.01 39.78, 33.9537
2010-01-12 02:30:00 49.97 33.63 34.0263
2010-02-12 02:30:00 50.07 41.48, 33.8547
def clean(s):
try: return float(s.strip(','))
except: return s
In [9]: df.applymap(clean)
Out[9]:
Hz MW kV
2010-01-12 00:15:00 49.82 34.65 33.73755
2010-02-12 00:15:00 49.92 36.33 33.90090
2010-01-12 00:30:00 49.90 35.34 33.77220
2010-02-12 00:30:00 49.89 37.65 33.83820
2010-01-12 00:45:00 49.94 33.50 33.83160
2010-02-12 00:45:00 50.09 37.41 34.07745
2010-01-12 01:00:00 49.86 30.91 33.94875
2010-02-12 01:00:00 50.18 36.11 34.20945
2010-01-12 01:15:00 49.97 27.28 34.22430
2010-02-12 01:15:00 50.11 33.24 34.35960
2010-01-12 01:30:00 50.02 26.91 34.33320
2010-02-12 01:30:00 50.12 31.03 34.45200
2010-01-12 01:45:00 50.01 31.26 34.12860
2010-02-12 01:45:00 50.00 38.86 33.93060
2010-01-12 02:00:00 50.08 34.96 33.91410
2010-02-12 02:00:00 50.14 38.31 33.99165
2010-01-12 02:15:00 50.07 35.33 33.84975
2010-02-12 02:15:00 50.01 39.78 33.95370
2010-01-12 02:30:00 49.97 33.63 34.02630
2010-02-12 02:30:00 50.07 41.48 33.85470
在熊猫身上做这种事情的关键是方法： 然而，我发现要找到一个你可以使用这个的地方，至少特定的csv是很棘手的。。至少可以说，几乎可以肯定有一种更好的方法可以做到这一点
df_data = pd.read_csv('e.csv', sep=',\s+', header=None, skiprows=3)[range(7)].set_index(0)
df_cols = pd.read_csv('e.csv', sep=',\s+', header=None, nrows=2).set_index(0)[:2] #interval causing problems
df_ = df_cols.append(df_data).T.set_index(['DATES','UNITS','Interval']).T
df = df_.stack(level=0)
df_dates = map(lambda x: pd.to_datetime(' '.join(x[::-1])), df.index)
df.index = df_dates
In [7]: df
Out[7]:
UNITS Hz MW kV
2010-01-12 00:15:00 49.82 34.65 33.73755
2010-02-12 00:15:00 49.92 36.33, 33.9009
2010-01-12 00:30:00 49.9 35.34 33.7722
2010-02-12 00:30:00 49.89 37.65, 33.8382
2010-01-12 00:45:00 49.94 33.5 33.8316
2010-02-12 00:45:00 50.09 37.41, 34.07745
2010-01-12 01:00:00 49.86 30.91 33.94875
2010-02-12 01:00:00 50.18 36.11, 34.20945
2010-01-12 01:15:00 49.97 27.28 34.2243
2010-02-12 01:15:00 50.11 33.24, 34.3596
2010-01-12 01:30:00 50.02 26.91 34.3332
2010-02-12 01:30:00 50.12 31.03, 34.452
2010-01-12 01:45:00 50.01 31.26 34.1286
2010-02-12 01:45:00 50 38.86, 33.9306
2010-01-12 02:00:00 50.08 34.96 33.9141
2010-02-12 02:00:00 50.14 38.31, 33.99165
2010-01-12 02:15:00 50.07 35.33 33.84975
2010-02-12 02:15:00 50.01 39.78, 33.9537
2010-01-12 02:30:00 49.97 33.63 34.0263
2010-02-12 02:30:00 50.07 41.48, 33.8547
def clean(s):
try: return float(s.strip(','))
except: return s
In [9]: df.applymap(clean)
Out[9]:
Hz MW kV
2010-01-12 00:15:00 49.82 34.65 33.73755
2010-02-12 00:15:00 49.92 36.33 33.90090
2010-01-12 00:30:00 49.90 35.34 33.77220
2010-02-12 00:30:00 49.89 37.65 33.83820
2010-01-12 00:45:00 49.94 33.50 33.83160
2010-02-12 00:45:00 50.09 37.41 34.07745
2010-01-12 01:00:00 49.86 30.91 33.94875
2010-02-12 01:00:00 50.18 36.11 34.20945
2010-01-12 01:15:00 49.97 27.28 34.22430
2010-02-12 01:15:00 50.11 33.24 34.35960
2010-01-12 01:30:00 50.02 26.91 34.33320
2010-02-12 01:30:00 50.12 31.03 34.45200
2010-01-12 01:45:00 50.01 31.26 34.12860
2010-02-12 01:45:00 50.00 38.86 33.93060
2010-01-12 02:00:00 50.08 34.96 33.91410
2010-02-12 02:00:00 50.14 38.31 33.99165
2010-01-12 02:15:00 50.07 35.33 33.84975
2010-02-12 02:15:00 50.01 39.78 33.95370
2010-01-12 02:30:00 49.97 33.63 34.02630
2010-02-12 02:30:00 50.07 41.48 33.85470
另一个解决办法是 读取csv第一行的日期 读入其余数据，包括间隔 根据需要构造索引并将其应用于数据帧 下面是一些示例代码：
In [1]: from StringIO import StringIO
In [2]: import pandas as pd
In [3]: pd.__version__
Out[3]: '0.10.1'
In [4]: CSV_SAMPLE = """
DATES, 01-12-2010, 01-12-2010, 01-12-2010, 02-12-2010, 02-12-2010, 02-12-2010
UNITS, Hz, kV, MW, Hz, kV, MW
Interval, , , , , ,
00:15, 49.82, 33.73755, 34.65, 49.92, 33.9009, 36.33,
00:30, 49.9, 33.7722, 35.34, 49.89, 33.8382, 37.65,
00:45, 49.94, 33.8316, 33.5, 50.09, 34.07745, 37.41,
01:00, 49.86, 33.94875, 30.91, 50.18, 34.20945, 36.11,
01:15, 49.97, 34.2243, 27.28, 50.11, 34.3596, 33.24,
01:30, 50.02, 34.3332, 26.91, 50.12, 34.452, 31.03,
"""
#Create one dataframe from just the dates (and we'll grab the units, too)
In [6]: datesdf = pd.read_csv(StringIO(CSV_SAMPLE), nrows= 2)
In [7]: dates, units = datesdf.index.droplevel()
In [9]: dates, units
Out[9]:
((' 01-12-2010',
' 01-12-2010',
' 01-12-2010',
' 02-12-2010',
' 02-12-2010',
' 02-12-2010'),
(' Hz', ' kV', ' MW', ' Hz', ' kV', ' MW'))
#Create a second dataframe from the rest of the data
In [11]: data = pd.read_csv(StringIO(CSV_SAMPLE), skiprows=3)
In [12]: data = data.icol([0,1,2])
#Note: Instead, in pandas 0.10, you can use the usecols paramater in read_csv()
# to combine the above two steps into one.
In [14]: data.columns = units[:3]
In [15]: print data
Hz kV MW
00:15 49.82 33.73755 34.65
00:30 49.90 33.77220 35.34
00:45 49.94 33.83160 33.50
01:00 49.86 33.94875 30.91
01:15 49.97 34.22430 27.28
01:30 50.02 34.33320 26.91
#We'll need to grab the intervals from this data df
In [16]: intervals = data.index.tolist()
In [17]: index1 = pd.MultiIndex.from_arrays([dates,intervals])
#This is a multi-index
In [18]: print index1
MultiIndex
[( 01-12-2010, 00:15), ( 01-12-2010, 00:30), ( 01-12-2010, 00:45), ( 02-12-2010, 01:00), ( 02-12-2010, 01:15), ( 02-12-2
010, 01:30)]
#This index is a tuple of date,interval
In [21]: index2 = pd.Index(zip(dates, intervals))
In [22]: print index2
Index([( 01-12-2010, 00:15), ( 01-12-2010, 00:30), ( 01-12-2010, 00:45), ( 02-12-2010, 01:00), ( 02-12-2010, 01:15), ( 0
2-12-2010, 01:30)], dtype=object)
#This index is based on a string concat of date and interval
In [23]: def list_join(x,y):
....: joined = x + ' ' + y
....: return joined.strip()
....:
In [24]: index3 = pd.Index(map(list_join, dates, intervals))
In [25]: print index3
Simple index:
Index([01-12-2010 00:15, 01-12-2010 00:30, 01-12-2010 00:45, 02-12-2010 01:00, 02-12-2010 01:15, 02-12-2010 01:30], dtyp
e=object)
In [26]: data.index = index3
In [27]: print data
Hz kV MW
01-12-2010 00:15 49.82 33.73755 34.65
01-12-2010 00:30 49.90 33.77220 35.34
01-12-2010 00:45 49.94 33.83160 33.50
02-12-2010 01:00 49.86 33.94875 30.91
02-12-2010 01:15 49.97 34.22430 27.28
02-12-2010 01:30 50.02 34.33320 26.91
如果完整的数据集抱怨非唯一索引值，您可能必须修改上面的代码来处理它。在这种情况下，将Interval csv列作为数据列而不是索引读入，并将其作为数组取出以创建所需的索引，如上所述 另一个解决方案是 读取csv第一行的日期 读入其余数据，包括间隔 根据需要构造索引并将其应用于数据帧 下面是一些示例代码：
In [1]: from StringIO import StringIO
In [2]: import pandas as pd
In [3]: pd.__version__
Out[3]: '0.10.1'
In [4]: CSV_SAMPLE = """
DATES, 01-12-2010, 01-12-2010, 01-12-2010, 02-12-2010, 02-12-2010, 02-12-2010
UNITS, Hz, kV, MW, Hz, kV, MW
Interval, , , , , ,
00:15, 49.82, 33.73755, 34.65, 49.92, 33.9009, 36.33,
00:30, 49.9, 33.7722, 35.34, 49.89, 33.8382, 37.65,
00:45, 49.94, 33.8316, 33.5, 50.09, 34.07745, 37.41,
01:00, 49.86, 33.94875, 30.91, 50.18, 34.20945, 36.11,
01:15, 49.97, 34.2243, 27.28, 50.11, 34.3596, 33.24,
01:30, 50.02, 34.3332, 26.91, 50.12, 34.452, 31.03,
"""
#Create one dataframe from just the dates (and we'll grab the units, too)
In [6]: datesdf = pd.read_csv(StringIO(CSV_SAMPLE), nrows= 2)
In [7]: dates, units = datesdf.index.droplevel()
In [9]: dates, units
Out[9]:
((' 01-12-2010',
' 01-12-2010',
' 01-12-2010',
' 02-12-2010',
' 02-12-2010',
' 02-12-2010'),
(' Hz', ' kV', ' MW', ' Hz', ' kV', ' MW'))
#Create a second dataframe from the rest of the data
In [11]: data = pd.read_csv(StringIO(CSV_SAMPLE), skiprows=3)
In [12]: data = data.icol([0,1,2])
#Note: Instead, in pandas 0.10, you can use the usecols paramater in read_csv()
# to combine the above two steps into one.
In [14]: data.columns = units[:3]
In [15]: print data
Hz kV MW
00:15 49.82 33.73755 34.65
00:30 49.90 33.77220 35.34
00:45 49.94 33.83160 33.50
01:00 49.86 33.94875 30.91
01:15 49.97 34.22430 27.28
01:30 50.02 34.33320 26.91
#We'll need to grab the intervals from this data df
In [16]: intervals = data.index.tolist()
In [17]: index1 = pd.MultiIndex.from_arrays([dates,intervals])
#This is a multi-index
In [18]: print index1
MultiIndex
[( 01-12-2010, 00:15), ( 01-12-2010, 00:30), ( 01-12-2010, 00:45), ( 02-12-2010, 01:00), ( 02-12-2010, 01:15), ( 02-12-2
010, 01:30)]
#This index is a tuple of date,interval
In [21]: index2 = pd.Index(zip(dates, intervals))
In [22]: print index2
Index([( 01-12-2010, 00:15), ( 01-12-2010, 00:30), ( 01-12-2010, 00:45), ( 02-12-2010, 01:00), ( 02-12-2010, 01:15), ( 0
2-12-2010, 01:30)], dtype=object)
#This index is based on a string concat of date and interval
In [23]: def list_join(x,y):
....: joined = x + ' ' + y
....: return joined.strip()
....:
In [24]: index3 = pd.Index(map(list_join, dates, intervals))
In [25]: print index3
Simple index:
Index([01-12-2010 00:15, 01-12-2010 00:30, 01-12-2010 00:45, 02-12-2010 01:00, 02-12-2010 01:15, 02-12-2010 01:30], dtyp
e=object)
In [26]: data.index = index3
In [27]: print data
Hz kV MW
01-12-2010 00:15 49.82 33.73755 34.65
01-12-2010 00:30 49.90 33.77220 35.34
01-12-2010 00:45 49.94 33.83160 33.50
02-12-2010 01:00 49.86 33.94875 30.91
02-12-2010 01:15 49.97 34.22430 27.28
02-12-2010 01:30 50.02 34.33320 26.91
如果完整的数据集抱怨非唯一索引值，您可能必须修改上面的代码来处理它。在这种情况下，将Interval csv列作为数据列而不是索引读入，并将其作为数组取出以创建所需的索引，如上所述 抱歉@AndyHayden。我没有正确地使用它。我仍然在努力获得groupby和stack的组合来重铸数据帧。嗨@Andy！我如何将2010年1月2日24:00第一天转换为2010年1月3日00:00？@amaity嗨，我不确定atm、pd.to_datetime似乎不这样做，我认为你把这个问题作为一个单独的问题问是有意义的，也许可以作为一个开始。@Andy我如何对上述时间序列进行排序，以在matplotlib中绘制日期-时间与赫兹的关系？抱歉@AndyHayden。我没有正确地使用它。我仍然在努力获得groupby和stack的组合来重铸数据帧。嗨@Andy！我怎样才能将2010年1月2日24:00第一天转换为2010年1月3日00:00？@amaity嗨，我不确定atm、pd.to_datetime似乎没有这样做，我认为你可以把这个问题作为一个单独的问题来问，也许可以作为一个开始。@Andy我如何对上述时间序列进行排序，以在matplotlib中绘制日期-时间与赫兹的关系？感谢@Aman提供了不同的方法。您的输出日期时间对象中是否包含日期？使用read_csv时，pandas可以将这些日期转换为日期时间对象，这将很容易在索引类型1和索引类型2编号中维护，如我的示例所示。对于索引类型3，您需要做更多的工作来创建一个更好的函数来replcae list_join，它为您提供了所需的datetime对象。感谢@Aman提供了不同的方法。您的输出日期时间对象中是否包含日期？使用read_csv时，pandas可以将这些日期转换为日期时间对象，这将很容易在索引类型1和索引类型2编号中维护，如我的示例所示。对于索引类型3，您需要做更多的工作来创建一个更好的函数来replcae list_join，该函数为您提供所需的datetime对象。