Hive 他使用Glue crawler为uf列创建了相同的模式,并且能够unest。我的文件中的一行看起来像{“prop1”:1,“uf”:[{“type”:“1browserId”,“name”:“value”:“1unknown”},{“type”:“2bro
Hive 他使用Glue crawler为uf列创建了相同的模式,并且能够unest。我的文件中的一行看起来像{“prop1”:1,“uf”:[{“type”:“1browserId”,“name”:“value”:“1unknown”},{“type”:“2bro,hive,aws-glue,presto,amazon-athena,aws-glue-data-catalog,Hive,Aws Glue,Presto,Amazon Athena,Aws Glue Data Catalog,他使用Glue crawler为uf列创建了相同的模式,并且能够unest。我的文件中的一行看起来像{“prop1”:1,“uf”:[{“type”:“1browserId”,“name”:“value”:“1unknown”},{“type”:“2browserId”,“name”:“value”:“2unknown”}为什么不通过spark展平这个数组,然后在athena中使用它呢?@Ilykisil我添加了示例CSVline@syadav我还不知道Spark,因此,尝试使用我所知道的东西来
他使用Glue crawler为
uf
列创建了相同的模式,并且能够unest
。我的文件中的一行看起来像{“prop1”:1,“uf”:[{“type”:“1browserId”,“name”:“value”:“1unknown”},{“type”:“2browserId”,“name”:“value”:“2unknown”}
为什么不通过spark展平这个数组,然后在athena中使用它呢?@Ilykisil我添加了示例CSVline@syadav我还不知道Spark,因此,尝试使用我所知道的东西来实现这一点,您能否提供完整的表DDL定义,即SHOW CREATE table
,以及整个JSON文件的示例,而不仅仅是列uf
?当我试图复制您的JSON文件时,我使用Glue crawler为uf
列获得了相同的模式,并且能够unest
。我的文件中的一行看起来像{“prop1”:1,“uf”:[{“type”:“1browserId”,“name”:“value”:“1unknown”},{“type”:“2browserId”,“name”:“value”:“2unknown”}
为什么不通过spark展平这个数组,然后在athena中使用它呢?@Ilykisil我添加了示例CSVline@syadav我还不知道Spark,因此,出于某种原因,我尝试使用我所知道的东西来执行转换,转换为json在select中可以工作,但如果是视图的一部分,则无法工作。例如:selectcast(5asjson)“test”
起作用,而| create view bla as selectcast(5asjson)“test”由于某种原因不起作用,cast to json在select中起作用,但如果是视图的一部分则不起作用。例如:selectcast(5作为json)“test”可以工作,而| createview bla作为selectcast(5作为json)“test”不能工作
[
{"type": "browserId", "name": "", "value": "unknown"},
{"type": "campaign", "name": "", "value": "om_227dec0082a5"},
{"type": "custom", "name": "2351350529", "value": "10148"},
{"type": "custom", "name": "9501713387", "value": "true"},
{"type": "custom", "name": "9517735577", "value": "true"},
{"type": "custom", "name": "9507402548", "value": "true"},
{"type": "custom", "name": "9733902068", "value": "true"}
]
[{type=[{"type": "browserId", "name": "", "value": "ff"}, name=null, value=null}]
{
"Name": "uf",
"Type": "array<struct<type:string,name:string,value:string>>"
}
timestamp project_id campaign_id experiment_id variation_id layer_holdback audience_names end_user_id uuid session_id snippet_revision user_ip user_agent user_engine user_engine_version referer global_holdback event_type event_name uf active_views event_features event_metrics event_uuid
1570326511 74971132 11089500404 11097730080 11078120202 false [] oeu1535997971348r0.4399811351004357 AUTO 6540 5.91.170.0 Mozilla/5.0 (Linux; Android 7.0; SAMSUNG SM-G925F Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/9.2 Chrome/67.0.3396.87 Mobile Safari/537.36 js 0.128.0 https://www.zavamed.com/uk/account/ false view_activated 10832783364 [{"type": "browserId", "name": "", "value": "unknown"}, {"type": "device", "name": "", "value": "mobile"}, {"type": "device_type", "name": "", "value": "phone"}, {"type": "referrer", "name": "", "value": "https:\/\/www.google.co.uk\/"}, {"type": "source_type", "name": "", "value": "campaign"}, {"type": "currentTimestamp", "name": "", "value": "-1631518596"}, {"type": "offset", "name": "", "value": "-60"}] [] [] [] 4926a5f1-bbb5-4553-9d0b-b26f773fa0f4
timestamp _unnested_column
1 1570326511 {"name":"","type":"browserId","value":"unknown"}
2 1570326511 {"name":"","type":"device","value":"mobile"}
3 1570326511 {"name":"","type":"device_type","value":"phone"}
4 1570326511 {"name":"","type":"referrer","value":"https://www.google.co.uk/"}
5 1570326511 {"name":"","type":"source_type","value":"campaign"}
6 1570326511 {"name":"","type":"currentTimestamp","value":"-1631518596"}
7 1570326511 {"name":"","type":"offset","value":"-60"}