Sql GroupBy在Google BigQuery中获取不同的行需要更长的时间

我正在对Google BigQuery中的publicdata:samples.github_timeline数据集进行漏斗分析。我想按时间顺序提取所有执行了一系列三个事件的唯一用户

事件及其顺序：

监视事件 推送事件 CreateEvent 以下是查询：

    select user from (
    SELECT user1 as user,
        ts1 as eventDate1,
        ts2 as eventDate2,
        IF(ts2 < ts3, ts3, NULL) as eventDate3
     FROM
        (SELECT user1,
        ts1,
            ts2,
            ts3
            FROM (SELECT user1,
    ts1,
    IF(ts1 < ts2, ts2, NULL) as ts2
    FROM
    (SELECT user1,
    ts1,
    ts2
    FROM (SELECT repository_owner as user1,
    created_at as ts1
    FROM [publicdata:samples.github_timeline]
    WHERE type = "WatchEvent") as step1
    LEFT JOIN EACH (SELECT repository_owner as user2,
    created_at as ts2
    FROM [publicdata:samples.github_timeline]
    WHERE type = "PushEvent") as step2
    ON user1 = user2 where ts1 is not NULL)

    ) as steps1_2
           LEFT JOIN (SELECT repository_owner as user3,
            created_at as ts3
     FROM [publicdata:samples.github_timeline]
     WHERE type = "CreateEvent") as step3
           ON user1 = user3 
           where ts2 is not NULL
           )
    )
    where eventDate3 is not null
    group by user
    limit 100

如果最后没有按用户分组，速度相当快，只有10秒。但是当我加上它的时候，需要花很多时间才能在20分钟内完成

这个问题怎么了？

---

您可以在此处测试查询：

如果在非分组查询中使用limit 100，orchestrator将在获取前100个数据行后中断执行

---

group by user limit 100要求在分组之前必须计算所有数据行。然后执行分组。最后，所有限制100生效。

你有一个加入爆炸；也就是说，如果用户A有20个WatchEvents、20个PushEvents和20个CreateEvents，那么您的查询可以在这60行中生成8000行。这是因为当连接的两侧都有多个匹配关键点时，它会生成两侧的笛卡尔积。您可以通过使用最短匹配时间来解决此问题，因此您只需查看用户的最短WatchEvent时间以查找后续PushEvent时间，然后查看晚于WatchEvent时间的最短PushEvent时间以查找匹配的CreateEvent时间

下面是一个大约20秒钟内运行的查询：

SELECT user 
FROM (
  SELECT step2_2.user1 as user,
    MIN(step2_2.ts1) as eventDate1,
    MIN(step2_2.ts2) as eventDate2,
    MIN(step3.ts3) as eventDate3
  FROM (
   SELECT user1, MIN(ts1) as ts1, MIN(ts2) as ts2
   FROM (
    SELECT repository_owner as user1,
    MIN(created_at) as ts1
    FROM [publicdata:samples.github_timeline]
    WHERE type = "WatchEvent"
    GROUP EACH BY user1) as step1
   JOIN EACH (
    SELECT repository_owner as user2,
      created_at as ts2
    FROM [publicdata:samples.github_timeline]
    WHERE type = "PushEvent") as step2
   ON user1 = user2
   WHERE ts1 < ts2
   GROUP EACH BY user1
  ) as step2_2
  JOIN EACH (
    SELECT repository_owner as user3,
      created_at as ts3
    FROM [publicdata:samples.github_timeline]
    WHERE type = "CreateEvent") as step3
  ON user1 = user3 
  WHERE step2_2.ts2 < step3.ts3
  GROUP EACH BY user
  ) 
GROUP BY user  
LIMIT 100

---

如果您的数据集不是太大，您可以使用lead window函数来查找序列并完全避免连接

Select repository_owner 
FROM
(
Select repository_owner,type as Event0, 
LEAD(x,1) OVER(Partition by repository_owner order by ts) as Event1, 
LEAD(x,2) OVER(Partition by repository_owner order by ts) as Event2, 

FROM
(
SELECT repository_owner as user,created_at as ts,type as x
from [publicdata:samples.github_timeline]
where type in ("WatchEvent","PushEvent","CreateEvent")
))
where Event0="WatchEvent"
and Event1="PushEvent"
and Event2="CreateEvent"

Group by repository_owner

7秒

如果事件的顺序与约旦的评论不一致，则需要将其变得更加复杂：

    Select repository_owner from
(
Select repository_owner,Event0,Event1,
Lead(Event0,1) OVER (Partition by repository_owner order by ts) as Event2,
Lead(Event1,1) OVER (Partition by repository_owner order by ts) as Event3,
FROM
(Select * from
(Select repository_owner,type as Event0,ts, 
LEAD(x,1) OVER(Partition by repository_owner order by ts) as Event1, 
FROM
(
SELECT repository_owner as user,created_at as ts,type as x
from [publicdata:samples.github_timeline]
where type in ("WatchEvent","PushEvent","CreateEvent")
))
where (Event0="WatchEvent" and
     Event1 in("PushEvent" ,"CreateEvent"))
OR ( Event1="CreateEvent" and
   Event0 in("PushEvent" ,"WatchEvent")))
)
   Where Event0="WatchEvent" and 
      (Event1="PushEvent" Or Event2="PushEvent") and
      Event3="CreateEvent"
Group by repository_owner

如果数据集太大，则会遇到以下问题：

---

希望它能有所帮助

我已经尝试了100次，用了23分钟。我认为这个解决方案比我建议的解决方案更优雅，但这是否只有在这些事件以这种顺序连续发生时才有效？例如，如果有一个WatchEvent/PushEvent/WatchEvent/CreateEvent，这是否会失败？编辑了我的答案，为您提到的场景添加了一个解决方案。