Python 熊猫:基于局部极小极大值的锯齿形数据分割
我有一个timeseries数据。生成数据Python 熊猫:基于局部极小极大值的锯齿形数据分割,python,pandas,time-series,technical-indicator,Python,Pandas,Time Series,Technical Indicator,我有一个timeseries数据。生成数据 date_rng = pd.date_range('2019-01-01', freq='s', periods=400) df = pd.DataFrame(np.random.lognormal(.005, .5,size=(len(date_rng), 3)), columns=['data1', 'data2', 'data3'], index= date_rng) s =
date_rng = pd.date_range('2019-01-01', freq='s', periods=400)
df = pd.DataFrame(np.random.lognormal(.005, .5,size=(len(date_rng), 3)),
columns=['data1', 'data2', 'data3'],
index= date_rng)
s = df['data1']
我想在局部极大值和局部极小值之间创建一条锯齿线,它满足以下条件:在y轴上,每条锯齿线的|最高-最低值|
必须超过前一条锯齿线距离的百分比(比如20%)和预先规定的值k(比如1.2)
我可以使用以下代码找到局部极值:
# Find peaks(max).
peak_indexes = signal.argrelextrema(s.values, np.greater)
peak_indexes = peak_indexes[0]
# Find valleys(min).
valley_indexes = signal.argrelextrema(s.values, np.less)
valley_indexes = valley_indexes[0]
# Merge peaks and valleys data points using pandas.
df_peaks = pd.DataFrame({'date': s.index[peak_indexes], 'zigzag_y': s[peak_indexes]})
df_valleys = pd.DataFrame({'date': s.index[valley_indexes], 'zigzag_y': s[valley_indexes]})
df_peaks_valleys = pd.concat([df_peaks, df_valleys], axis=0, ignore_index=True, sort=True)
# Sort peak and valley datapoints by date.
df_peaks_valleys = df_peaks_valleys.sort_values(by=['date'])
但我不知道如何对其应用阈值条件。
请告诉我如何应用这些条件
由于数据可能包含一百万个时间戳,因此强烈建议进行有效的计算
为了更清楚地描述:
示例输出,来自我的数据:
# Instantiate axes.
(fig, ax) = plt.subplots()
# Plot zigzag trendline.
ax.plot(df_peaks_valleys['date'].values, df_peaks_valleys['zigzag_y'].values,
color='red', label="Zigzag")
# Plot original line.
ax.plot(s.index, s, linestyle='dashed', color='black', label="Org. line", linewidth=1)
# Format time.
ax.xaxis_date()
ax.xaxis.set_major_formatter(mdates.DateFormatter("%Y-%m-%d"))
plt.gcf().autofmt_xdate() # Beautify the x-labels
plt.autoscale(tight=True)
plt.legend(loc='best')
plt.grid(True, linestyle='dashed')
我想要的输出(类似于此,锯齿形只连接重要的部分)
我对这个问题的回答已经达到了我的最佳理解。然而,变量K如何影响滤波器尚不清楚 您希望根据运行条件过滤极值。我假设您想要标记与最后标记的极值的相对距离大于p%的所有极值。我进一步假设,你总是认为时间序列的第一个元素是一个有效的/相关的点。 我通过以下过滤器功能实现了这一点:
def filter(values, percentage):
previous = values[0]
mask = [True]
for value in values[1:]:
relative_difference = np.abs(value - previous)/previous
if relative_difference > percentage:
previous = value
mask.append(True)
else:
mask.append(False)
return mask
要运行代码,我首先导入依赖项:
from scipy import signal
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
为了使代码可复制,我修复了随机种子:
np.random.seed(0)
剩下的是意大利面。请注意,我减少了样本量以使结果更清晰
date_rng = pd.date_range('2019-01-01', freq='s', periods=30)
df = pd.DataFrame(np.random.lognormal(.005, .5,size=(len(date_rng), 3)),
columns=['data1', 'data2', 'data3'],
index= date_rng)
s = df['data1']
# Find peaks(max).
peak_indexes = signal.argrelextrema(s.values, np.greater)
peak_indexes = peak_indexes[0]
# Find valleys(min).
valley_indexes = signal.argrelextrema(s.values, np.less)
valley_indexes = valley_indexes[0]
# Merge peaks and valleys data points using pandas.
df_peaks = pd.DataFrame({'date': s.index[peak_indexes], 'zigzag_y': s[peak_indexes]})
df_valleys = pd.DataFrame({'date': s.index[valley_indexes], 'zigzag_y': s[valley_indexes]})
df_peaks_valleys = pd.concat([df_peaks, df_valleys], axis=0, ignore_index=True, sort=True)
# Sort peak and valley datapoints by date.
df_peaks_valleys = df_peaks_valleys.sort_values(by=['date'])
然后我们使用过滤函数:
p = 0.2 # 20%
filter_mask = filter(df_peaks_valleys.zigzag_y, p)
filtered = df_peaks_valleys[filter_mask]
与之前的绘图以及新过滤的极值一样进行绘图:
# Instantiate axes.
(fig, ax) = plt.subplots(figsize=(10,10))
# Plot zigzag trendline.
ax.plot(df_peaks_valleys['date'].values, df_peaks_valleys['zigzag_y'].values,
color='red', label="Extrema")
# Plot zigzag trendline.
ax.plot(filtered['date'].values, filtered['zigzag_y'].values,
color='blue', label="ZigZag")
# Plot original line.
ax.plot(s.index, s, linestyle='dashed', color='black', label="Org. line", linewidth=1)
# Format time.
ax.xaxis_date()
ax.xaxis.set_major_formatter(mdates.DateFormatter("%Y-%m-%d"))
plt.gcf().autofmt_xdate() # Beautify the x-labels
plt.autoscale(tight=True)
plt.legend(loc='best')
plt.grid(True, linestyle='dashed')
编辑:
如果要同时考虑第一个和最后一个点一样有效,则可以将过滤函数调整为:
def filter(values, percentage):
# the first value is always valid
previous = values[0]
mask = [True]
# evaluate all points from the second to (n-1)th
for value in values[1:-1]:
relative_difference = np.abs(value - previous)/previous
if relative_difference > percentage:
previous = value
mask.append(True)
else:
mask.append(False)
# the last value is always valid
mask.append(True)
return mask
可以使用滚动功能创建局部极值。与Scipy方法相比,这稍微简化了代码 用于查找极值的函数:
def islocalmax(x):
"""Both neighbors are lower,
assumes a centered window of size 3"""
return (x[0] < x[1]) & (x[2] < x[1])
def islocalmin(x):
"""Both neighbors are higher,
assumes a centered window of size 3"""
return (x[0] > x[1]) & (x[2] > x[1])
def isextrema(x):
return islocalmax(x) or islocalmin(x)
生成一些示例数据:
date_rng = pd.date_range('2019-01-01', freq='s', periods=35)
df = pd.DataFrame(np.random.randn(len(date_rng), 3),
columns=['data1', 'data2', 'data3'],
index= date_rng)
df = df.cumsum()
应用函数并提取“data1”列的结果:
dfzigzag = df.apply(create_zigzag)
data1_zigzag = dfzigzag['data1'].dropna()
将结果可视化:
fig, axs = plt.subplots(figsize=(10, 3))
axs.plot(df.data1, 'ko-', ms=4, label='original')
axs.plot(data1_zigzag, 'ro-', ms=4, label='zigzag')
axs.legend()
嗨,谢谢你的回答。是的,您的假设是正确的“标记与最后标记的极值的相对距离大于p%的所有极值”,并且始终应考虑第一点和最后一点。我已经检查了你的答案,有时会漏掉最后一点,你能帮我吗?谢谢你的回答。我想问一下这一行
(ext_val.diff().abs()>(ext_val.shift(-1.abs()*p))
,据我所知,你是在比较两点之间的距离和最后一点的p%
,对吗?因为我想将每个锯齿形线段与前一个线段进行比较,然后重复,直到满足条件为止。
fig, axs = plt.subplots(figsize=(10, 3))
axs.plot(df.data1, 'ko-', ms=4, label='original')
axs.plot(data1_zigzag, 'ro-', ms=4, label='zigzag')
axs.legend()