Google bigquery 如何从嵌套记录中查询字段而不引用BigQuery中的父记录？_Google Bigquery

Google bigquery 如何从嵌套记录中查询字段而不引用BigQuery中的父记录？

google-bigquery

Google bigquery 如何从嵌套记录中查询字段而不引用BigQuery中的父记录？,google-bigquery,Google Bigquery,我的数据结构如下： { "results": { "A": {"first": 1, "second": 2, "third": 3}, "B": {"first": 4, "second": 5, "third": 6}, "C": {"first"

我的数据结构如下：

{ 
  "results": {
    "A": {"first": 1, "second": 2, "third": 3},
    "B": {"first": 4, "second": 5, "third": 6},
    "C": {"first": 7, "second": 8, "third": 9},
    "D": {"first": 1, "second": 2, "third": 3}, 
   ... },
  ...
}

i、 e.嵌套记录，其中最低级别对上面级别中的所有记录具有相同的模式。模式类似于此：

results              RECORD    NULLABLE
results.A            RECORD    NULLABLE
results.A.first      INTEGER   NULLABLE
results.A.second     INTEGER   NULLABLE
results.A.third      INTEGER   NULLABLE
results.B            RECORD    NULLABLE
results.B.first      INTEGER   NULLABLE
...

是否有一种方法，例如在不知道直接父级上的键的情况下，在BigQuery中对最低级别的字段进行聚合查询？换言之，我可以先查询结果中的所有记录，而不必指定a、B、。。。在我的问题中

例如，我想实现以下目标：

SELECT SUM(results.*.first) FROM table

为了得到1+4+7+1=13，但选择结果。*.first不受支持

我已经尝试过使用结构，但还没走多远。

下面的技巧是针对BigQuery标准SQL的

#standardSQL
SELECT id, ( 
    SELECT AS STRUCT 
      SUM(first) AS sum_first, 
      SUM(second) AS sum_second, 
      SUM(third) AS sum_third 
    FROM UNNEST([a]||[b]||[c]||[d])
  ).*
FROM `project.dataset.table`,
UNNEST([results])

您可以使用问题中的虚拟/示例数据测试、播放上述内容，如下例所示

#standardSQL
WITH `project.dataset.table` AS (
  SELECT 1 AS id, STRUCT(
    STRUCT(1 AS first, 2 AS second, 3 AS third) AS A,
    STRUCT(4 AS first, 5 AS second, 6 AS third) AS B,
    STRUCT(7 AS first, 8 AS second, 9 AS third) AS C,
    STRUCT(1 AS first, 2 AS second, 3 AS third) AS D
  ) AS results
)
SELECT id, ( 
    SELECT AS STRUCT 
      SUM(first) AS sum_first, 
      SUM(second) AS sum_second, 
      SUM(third) AS sum_third 
    FROM UNNEST([a]||[b]||[c]||[d])
  ).*
FROM `project.dataset.table`,
UNNEST([results])

#standardSQL
CREATE TEMP FUNCTION Nested_SUM(entries ANY TYPE, field_name STRING) AS ((
  SELECT SUM(CAST(SPLIT(kv, ':')[OFFSET(1)] AS INT64))
  FROM UNNEST(REGEXP_EXTRACT_ALL(TO_JSON_STRING(entries), r'":{(.*?)}')) entry,
  UNNEST(SPLIT(entry)) kv
  WHERE TRIM(SPLIT(kv, ':')[OFFSET(0)], '"') = field_name
));
WITH `project.dataset.table` AS (
  SELECT 1 AS id, STRUCT(
    STRUCT(1 AS first, 2 AS second, 3 AS third) AS A,
    STRUCT(4 AS first, 5 AS second, 6 AS third) AS B,
    STRUCT(7 AS first, 8 AS second, 9 AS third) AS C,
    STRUCT(1 AS first, 2 AS second, 3 AS third) AS D
  ) AS results
)
SELECT id, 
  Nested_SUM(results, 'first') AS first_sum,
  Nested_SUM(results, 'second') AS second_sum,
  Nested_SUM(results, 'third') AS third_sum,
  Nested_SUM(results, 'forth') AS forth_sum
FROM `project.dataset.table`

有输出

Row id  sum_first   sum_second  sum_third    
1   1   13          17          21

以下技巧适用于BigQuery标准SQL

#standardSQL
SELECT id, ( 
    SELECT AS STRUCT 
      SUM(first) AS sum_first, 
      SUM(second) AS sum_second, 
      SUM(third) AS sum_third 
    FROM UNNEST([a]||[b]||[c]||[d])
  ).*
FROM `project.dataset.table`,
UNNEST([results])

您可以使用问题中的虚拟/示例数据测试、播放上述内容，如下例所示

#standardSQL
WITH `project.dataset.table` AS (
  SELECT 1 AS id, STRUCT(
    STRUCT(1 AS first, 2 AS second, 3 AS third) AS A,
    STRUCT(4 AS first, 5 AS second, 6 AS third) AS B,
    STRUCT(7 AS first, 8 AS second, 9 AS third) AS C,
    STRUCT(1 AS first, 2 AS second, 3 AS third) AS D
  ) AS results
)
SELECT id, ( 
    SELECT AS STRUCT 
      SUM(first) AS sum_first, 
      SUM(second) AS sum_second, 
      SUM(third) AS sum_third 
    FROM UNNEST([a]||[b]||[c]||[d])
  ).*
FROM `project.dataset.table`,
UNNEST([results])

#standardSQL
CREATE TEMP FUNCTION Nested_SUM(entries ANY TYPE, field_name STRING) AS ((
  SELECT SUM(CAST(SPLIT(kv, ':')[OFFSET(1)] AS INT64))
  FROM UNNEST(REGEXP_EXTRACT_ALL(TO_JSON_STRING(entries), r'":{(.*?)}')) entry,
  UNNEST(SPLIT(entry)) kv
  WHERE TRIM(SPLIT(kv, ':')[OFFSET(0)], '"') = field_name
));
WITH `project.dataset.table` AS (
  SELECT 1 AS id, STRUCT(
    STRUCT(1 AS first, 2 AS second, 3 AS third) AS A,
    STRUCT(4 AS first, 5 AS second, 6 AS third) AS B,
    STRUCT(7 AS first, 8 AS second, 9 AS third) AS C,
    STRUCT(1 AS first, 2 AS second, 3 AS third) AS D
  ) AS results
)
SELECT id, 
  Nested_SUM(results, 'first') AS first_sum,
  Nested_SUM(results, 'second') AS second_sum,
  Nested_SUM(results, 'third') AS third_sum,
  Nested_SUM(results, 'forth') AS forth_sum
FROM `project.dataset.table`

有输出

Row id  sum_first   sum_second  sum_third    
1   1   13          17          21

是否有一种方法，例如在不知道直接父级上的键的情况下，在BigQuery中对最低级别的字段进行聚合查询

下面是BigQuery标准SQL，完全避免引用父记录A、B、C、D等

#standardSQL
CREATE TEMP FUNCTION Nested_SUM(entries ANY TYPE, field_name STRING) AS ((
  SELECT SUM(CAST(SPLIT(kv, ':')[OFFSET(1)] AS INT64))
  FROM UNNEST(REGEXP_EXTRACT_ALL(TO_JSON_STRING(entries), r'":{(.*?)}')) entry,
  UNNEST(SPLIT(entry)) kv
  WHERE TRIM(SPLIT(kv, ':')[OFFSET(0)], '"') = field_name
));
SELECT id, 
  Nested_SUM(results, 'first') AS first_sum,
  Nested_SUM(results, 'second') AS second_sum,
  Nested_SUM(results, 'third') AS third_sum,
  Nested_SUM(results, 'forth') AS forth_sum
FROM `project.dataset.table`

如果要应用于问题中的样本数据，如下例所示

#standardSQL
WITH `project.dataset.table` AS (
  SELECT 1 AS id, STRUCT(
    STRUCT(1 AS first, 2 AS second, 3 AS third) AS A,
    STRUCT(4 AS first, 5 AS second, 6 AS third) AS B,
    STRUCT(7 AS first, 8 AS second, 9 AS third) AS C,
    STRUCT(1 AS first, 2 AS second, 3 AS third) AS D
  ) AS results
)
SELECT id, ( 
    SELECT AS STRUCT 
      SUM(first) AS sum_first, 
      SUM(second) AS sum_second, 
      SUM(third) AS sum_third 
    FROM UNNEST([a]||[b]||[c]||[d])
  ).*
FROM `project.dataset.table`,
UNNEST([results])

#standardSQL
CREATE TEMP FUNCTION Nested_SUM(entries ANY TYPE, field_name STRING) AS ((
  SELECT SUM(CAST(SPLIT(kv, ':')[OFFSET(1)] AS INT64))
  FROM UNNEST(REGEXP_EXTRACT_ALL(TO_JSON_STRING(entries), r'":{(.*?)}')) entry,
  UNNEST(SPLIT(entry)) kv
  WHERE TRIM(SPLIT(kv, ':')[OFFSET(0)], '"') = field_name
));
WITH `project.dataset.table` AS (
  SELECT 1 AS id, STRUCT(
    STRUCT(1 AS first, 2 AS second, 3 AS third) AS A,
    STRUCT(4 AS first, 5 AS second, 6 AS third) AS B,
    STRUCT(7 AS first, 8 AS second, 9 AS third) AS C,
    STRUCT(1 AS first, 2 AS second, 3 AS third) AS D
  ) AS results
)
SELECT id, 
  Nested_SUM(results, 'first') AS first_sum,
  Nested_SUM(results, 'second') AS second_sum,
  Nested_SUM(results, 'third') AS third_sum,
  Nested_SUM(results, 'forth') AS forth_sum
FROM `project.dataset.table`

输出为

Row id  first_sum   second_sum  third_sum   forth_sum    
1   1   13          17          21          null

是否有一种方法，例如在不知道直接父级上的键的情况下，在BigQuery中对最低级别的字段进行聚合查询

下面是BigQuery标准SQL，完全避免引用父记录A、B、C、D等

#standardSQL
CREATE TEMP FUNCTION Nested_SUM(entries ANY TYPE, field_name STRING) AS ((
  SELECT SUM(CAST(SPLIT(kv, ':')[OFFSET(1)] AS INT64))
  FROM UNNEST(REGEXP_EXTRACT_ALL(TO_JSON_STRING(entries), r'":{(.*?)}')) entry,
  UNNEST(SPLIT(entry)) kv
  WHERE TRIM(SPLIT(kv, ':')[OFFSET(0)], '"') = field_name
));
SELECT id, 
  Nested_SUM(results, 'first') AS first_sum,
  Nested_SUM(results, 'second') AS second_sum,
  Nested_SUM(results, 'third') AS third_sum,
  Nested_SUM(results, 'forth') AS forth_sum
FROM `project.dataset.table`

如果要应用于问题中的样本数据，如下例所示

#standardSQL
WITH `project.dataset.table` AS (
  SELECT 1 AS id, STRUCT(
    STRUCT(1 AS first, 2 AS second, 3 AS third) AS A,
    STRUCT(4 AS first, 5 AS second, 6 AS third) AS B,
    STRUCT(7 AS first, 8 AS second, 9 AS third) AS C,
    STRUCT(1 AS first, 2 AS second, 3 AS third) AS D
  ) AS results
)
SELECT id, ( 
    SELECT AS STRUCT 
      SUM(first) AS sum_first, 
      SUM(second) AS sum_second, 
      SUM(third) AS sum_third 
    FROM UNNEST([a]||[b]||[c]||[d])
  ).*
FROM `project.dataset.table`,
UNNEST([results])

#standardSQL
CREATE TEMP FUNCTION Nested_SUM(entries ANY TYPE, field_name STRING) AS ((
  SELECT SUM(CAST(SPLIT(kv, ':')[OFFSET(1)] AS INT64))
  FROM UNNEST(REGEXP_EXTRACT_ALL(TO_JSON_STRING(entries), r'":{(.*?)}')) entry,
  UNNEST(SPLIT(entry)) kv
  WHERE TRIM(SPLIT(kv, ':')[OFFSET(0)], '"') = field_name
));
WITH `project.dataset.table` AS (
  SELECT 1 AS id, STRUCT(
    STRUCT(1 AS first, 2 AS second, 3 AS third) AS A,
    STRUCT(4 AS first, 5 AS second, 6 AS third) AS B,
    STRUCT(7 AS first, 8 AS second, 9 AS third) AS C,
    STRUCT(1 AS first, 2 AS second, 3 AS third) AS D
  ) AS results
)
SELECT id, 
  Nested_SUM(results, 'first') AS first_sum,
  Nested_SUM(results, 'second') AS second_sum,
  Nested_SUM(results, 'third') AS third_sum,
  Nested_SUM(results, 'forth') AS forth_sum
FROM `project.dataset.table`

输出为

Row id  first_sum   second_sum  third_sum   forth_sum    
1   1   13          17          21          null

为了支持对最低级别字段的值进行分组，我进行了调整：

#standardSQL
CREATE TEMP FUNCTION Nested_AGGREGATE(entries ANY TYPE, field_name STRING) AS ((
  SELECT ARRAY(
    SELECT AS STRUCT TRIM(SPLIT(kv, ':')[OFFSET(1)], '"') AS value, COUNT(SPLIT(kv, ':')[OFFSET(1)]) AS count
    FROM UNNEST(REGEXP_EXTRACT_ALL(TO_JSON_STRING(entries), r'":{(.*?)}')) entry,
    UNNEST(SPLIT(entry)) kv
    WHERE TRIM(SPLIT(kv, ':')[OFFSET(0)], '"') = field_name
    GROUP BY TRIM(SPLIT(kv, ':')[OFFSET(1)], '"')
 )
));
SELECT id, 
  Nested_AGGREGATE(results, 'first') AS first_agg,
  Nested_AGGREGATE(results, 'second') AS second_agg,
  Nested_AGGREGATE(results, 'third') AS third_agg,
FROM `project.dataset.table`

使用“project.dataset.table”作为的输出选择1作为id，结构1作为第一个，2作为第二个，3作为第三个作为A，结构4作为第一个，5作为第二个，6作为第三个作为B，结构7作为第一个，8作为第二个，9作为第三个作为C，结构1作为第一个，2作为第二个，3作为第三个作为D作为结果：

为了支持对最低级别字段的值进行分组，我进行了调整：

#standardSQL
CREATE TEMP FUNCTION Nested_AGGREGATE(entries ANY TYPE, field_name STRING) AS ((
  SELECT ARRAY(
    SELECT AS STRUCT TRIM(SPLIT(kv, ':')[OFFSET(1)], '"') AS value, COUNT(SPLIT(kv, ':')[OFFSET(1)]) AS count
    FROM UNNEST(REGEXP_EXTRACT_ALL(TO_JSON_STRING(entries), r'":{(.*?)}')) entry,
    UNNEST(SPLIT(entry)) kv
    WHERE TRIM(SPLIT(kv, ':')[OFFSET(0)], '"') = field_name
    GROUP BY TRIM(SPLIT(kv, ':')[OFFSET(1)], '"')
 )
));
SELECT id, 
  Nested_AGGREGATE(results, 'first') AS first_agg,
  Nested_AGGREGATE(results, 'second') AS second_agg,
  Nested_AGGREGATE(results, 'third') AS third_agg,
FROM `project.dataset.table`

表的架构仍然不清楚！是包含json的字符串字段吗？还是重复记录？请提供架构。最好的方法也是提供语句来重现您的数据，以便我们能够有效地帮助表的模式仍然不清楚！是包含json的字符串字段吗？还是重复记录？请提供架构。最好的方法也是提供声明来重现您的数据，以便我们能够有效地帮助您。您可能希望接受我各自的答案，因为它看起来确实有帮助！你可能想接受我各自的答案，因为它看起来真的很有帮助！