Python 如何使用groupby执行引用数据帧中数据子集上的前一行的函数

我有一些日志数据，这些数据表示一个项（id）和一个时间戳，表明某个操作已启动，我想确定每个项上的操作之间的时间间隔

例如，我有一些数据如下所示：

data = [{"timestamp":"2019-05-21T14:17:29.265Z","id":"ff9dad92-e7c1-47a5-93a7-6e49533a6e25"},{"timestamp":"2019-05-21T14:21:49.722Z","id":"ff9dad92-e7c1-47a5-93a7-6e49533a6e25"},{"timestamp":"2019-05-21T15:16:25.695Z","id":"ff9dad92-e7c1-47a5-93a7-6e49533a6e25"},{"timestamp":"2019-05-21T15:16:25.696Z","id":"ff9dad92-e7c1-47a5-93a7-6e49533a6e25"},{"timestamp":"2019-05-22T07:51:17.49Z","id":"ff12891e-5786-438b-891c-abd4244723b4"},{"timestamp":"2019-05-22T08:11:13.948Z","id":"ff12891e-5786-438b-891c-abd4244723b4"},{"timestamp":"2019-05-22T11:52:59.897Z","id":"ff12891e-5786-438b-891c-abd4244723b4"},{"timestamp":"2019-05-22T11:53:03.406Z","id":"ff12891e-5786-438b-891c-abd4244723b4"},{"timestamp":"2019-05-22T11:53:03.481Z","id":"ff12891e-5786-438b-891c-abd4244723b4"},{"timestamp":"2019-05-21T14:23:08.147Z","id":"fe55bb22-fe5b-4b12-8aaf-d5f0320ac7fa"},{"timestamp":"2019-05-21T14:29:18.228Z","id":"fe55bb22-fe5b-4b12-8aaf-d5f0320ac7fa"},{"timestamp":"2019-05-21T15:17:09.831Z","id":"fe55bb22-fe5b-4b12-8aaf-d5f0320ac7fa"},{"timestamp":"2019-05-21T15:17:09.834Z","id":"fe55bb22-fe5b-4b12-8aaf-d5f0320ac7fa"},{"timestamp":"2019-05-21T14:02:19.072Z","id":"fd3554cd-b83d-49af-a8e6-7bf41c741cd0"},{"timestamp":"2019-05-21T14:02:34.867Z","id":"fd3554cd-b83d-49af-a8e6-7bf41c741cd0"},{"timestamp":"2019-05-21T14:12:28.877Z","id":"fd3554cd-b83d-49af-a8e6-7bf41c741cd0"},{"timestamp":"2019-05-21T15:19:19.567Z","id":"fd3554cd-b83d-49af-a8e6-7bf41c741cd0"},{"timestamp":"2019-05-21T15:19:19.582Z","id":"fd3554cd-b83d-49af-a8e6-7bf41c741cd0"},{"timestamp":"2019-05-21T09:58:02.185Z","id":"f89c2e3e-06dc-467b-813b-dc92f2692f63"},{"timestamp":"2019-05-21T10:07:24.044Z","id":"f89c2e3e-06dc-467b-813b-dc92f2692f63"}]
stack = pd.DataFrame(data)
stack.head()

我尝试过获取所有唯一ID来分割数据帧，然后获取索引与原始集重新组合所需的时间，如，但在大型数据集上，该函数速度非常慢，并且会将索引都弄乱 和时间戳顺序，导致结果未匹配

import ciso8601 as time
records = []
for i in list(stack.id.unique()):
    dff = stack[stack.id == i]
    time_taken = []
    times = []
    i = 0
    for _, row in dff.iterrows():
        if bool(times):
            print(_)
            current_time = time.parse_datetime(row.timestamp)
            prev_time = times[i]
            time_taken = current_time - prev_time
            times.append(current_time)
            i+=1
            records.append(dict(index = _, time_taken = time_taken.seconds))
        else:
            records.append(dict(index = _, time_taken = 0))
            times.append(time.parse_datetime(row.timestamp))

x = pd.DataFrame(records).set_index('index')
stack.merge(x, left_index=True, right_index=True, how='inner')

是否有一种整洁的pandas groupby和apply方法来执行此操作，这样我就不必拆分框架并将其存储在内存中，以便可以引用子集中的前一行

谢谢

您可以使用：

---

如果希望生成的数据帧按日期排序，请改为执行以下操作：

stack['timestamp'] = pd.to_datetime(stack['timestamp']) 
stack = stack.sort_values(['id','timestamp'])
stack['time_taken'] = (stack.groupby('id')
                            .diff()['timestamp'] 
                            .dt.total_seconds() 
                            .round()
                            .fillna(0))

---

如果datetimes不需要替换时间戳来创建由datetimes填充的序列，并将其传递给，然后将转换为秒，如有必要，将其舍入，并将缺少的值替换为

0

：

t = pd.to_datetime(stack['timestamp'])
stack['time_taken'] = t.groupby(stack['id']).diff().dt.total_seconds().round().fillna(0)

print (stack)
                                      id                 timestamp  time_taken
0   ff9dad92-e7c1-47a5-93a7-6e49533a6e25  2019-05-21T14:17:29.265Z         0.0
1   ff9dad92-e7c1-47a5-93a7-6e49533a6e25  2019-05-21T14:21:49.722Z       260.0
2   ff9dad92-e7c1-47a5-93a7-6e49533a6e25  2019-05-21T15:16:25.695Z      3276.0
3   ff9dad92-e7c1-47a5-93a7-6e49533a6e25  2019-05-21T15:16:25.696Z         0.0
4   ff12891e-5786-438b-891c-abd4244723b4   2019-05-22T07:51:17.49Z         0.0
5   ff12891e-5786-438b-891c-abd4244723b4  2019-05-22T08:11:13.948Z      1196.0
6   ff12891e-5786-438b-891c-abd4244723b4  2019-05-22T11:52:59.897Z     13306.0
7   ff12891e-5786-438b-891c-abd4244723b4  2019-05-22T11:53:03.406Z         4.0
8   ff12891e-5786-438b-891c-abd4244723b4  2019-05-22T11:53:03.481Z         0.0
9   fe55bb22-fe5b-4b12-8aaf-d5f0320ac7fa  2019-05-21T14:23:08.147Z         0.0
10  fe55bb22-fe5b-4b12-8aaf-d5f0320ac7fa  2019-05-21T14:29:18.228Z       370.0
11  fe55bb22-fe5b-4b12-8aaf-d5f0320ac7fa  2019-05-21T15:17:09.831Z      2872.0
12  fe55bb22-fe5b-4b12-8aaf-d5f0320ac7fa  2019-05-21T15:17:09.834Z         0.0
13  fd3554cd-b83d-49af-a8e6-7bf41c741cd0  2019-05-21T14:02:19.072Z         0.0
14  fd3554cd-b83d-49af-a8e6-7bf41c741cd0  2019-05-21T14:02:34.867Z        16.0
15  fd3554cd-b83d-49af-a8e6-7bf41c741cd0  2019-05-21T14:12:28.877Z       594.0
16  fd3554cd-b83d-49af-a8e6-7bf41c741cd0  2019-05-21T15:19:19.567Z      4011.0
17  fd3554cd-b83d-49af-a8e6-7bf41c741cd0  2019-05-21T15:19:19.582Z         0.0
18  f89c2e3e-06dc-467b-813b-dc92f2692f63  2019-05-21T09:58:02.185Z         0.0
19  f89c2e3e-06dc-467b-813b-dc92f2692f63  2019-05-21T10:07:24.044Z       562.0

---

或者，如果需要替换时间戳以更新日期，请使用@yatu-answer。

groupby的结果是否会自然地按时间戳排序？这些答案假设日期时间已排序检查我的答案@JohnyMudly中的更新。如果顺序没有保证，首先必须对值进行排序。嗯，我认为应该是相同的，但在我的情况下，最终的df没有顺序。但是，是的，我想有一个有序的结果会更可取@jez。但它也不符合预期的产出

t = pd.to_datetime(stack['timestamp'])
stack['time_taken'] = t.groupby(stack['id']).diff().dt.total_seconds().round().fillna(0)

print (stack)
                                      id                 timestamp  time_taken
0   ff9dad92-e7c1-47a5-93a7-6e49533a6e25  2019-05-21T14:17:29.265Z         0.0
1   ff9dad92-e7c1-47a5-93a7-6e49533a6e25  2019-05-21T14:21:49.722Z       260.0
2   ff9dad92-e7c1-47a5-93a7-6e49533a6e25  2019-05-21T15:16:25.695Z      3276.0
3   ff9dad92-e7c1-47a5-93a7-6e49533a6e25  2019-05-21T15:16:25.696Z         0.0
4   ff12891e-5786-438b-891c-abd4244723b4   2019-05-22T07:51:17.49Z         0.0
5   ff12891e-5786-438b-891c-abd4244723b4  2019-05-22T08:11:13.948Z      1196.0
6   ff12891e-5786-438b-891c-abd4244723b4  2019-05-22T11:52:59.897Z     13306.0
7   ff12891e-5786-438b-891c-abd4244723b4  2019-05-22T11:53:03.406Z         4.0
8   ff12891e-5786-438b-891c-abd4244723b4  2019-05-22T11:53:03.481Z         0.0
9   fe55bb22-fe5b-4b12-8aaf-d5f0320ac7fa  2019-05-21T14:23:08.147Z         0.0
10  fe55bb22-fe5b-4b12-8aaf-d5f0320ac7fa  2019-05-21T14:29:18.228Z       370.0
11  fe55bb22-fe5b-4b12-8aaf-d5f0320ac7fa  2019-05-21T15:17:09.831Z      2872.0
12  fe55bb22-fe5b-4b12-8aaf-d5f0320ac7fa  2019-05-21T15:17:09.834Z         0.0
13  fd3554cd-b83d-49af-a8e6-7bf41c741cd0  2019-05-21T14:02:19.072Z         0.0
14  fd3554cd-b83d-49af-a8e6-7bf41c741cd0  2019-05-21T14:02:34.867Z        16.0
15  fd3554cd-b83d-49af-a8e6-7bf41c741cd0  2019-05-21T14:12:28.877Z       594.0
16  fd3554cd-b83d-49af-a8e6-7bf41c741cd0  2019-05-21T15:19:19.567Z      4011.0
17  fd3554cd-b83d-49af-a8e6-7bf41c741cd0  2019-05-21T15:19:19.582Z         0.0
18  f89c2e3e-06dc-467b-813b-dc92f2692f63  2019-05-21T09:58:02.185Z         0.0
19  f89c2e3e-06dc-467b-813b-dc92f2692f63  2019-05-21T10:07:24.044Z       562.0