Hive 蜂箱罐';t使用嵌套的avro模式创建表
我正在尝试使用嵌套的avro模式来创建配置单元表。但它不起作用。我正在使用cdh5.7.2中的Hive1.1 以下是我的嵌套avro模式:Hive 蜂箱罐';t使用嵌套的avro模式创建表,hive,nested,schema,avro,Hive,Nested,Schema,Avro,我正在尝试使用嵌套的avro模式来创建配置单元表。但它不起作用。我正在使用cdh5.7.2中的Hive1.1 以下是我的嵌套avro模式: [ { "type": "record", "name": "Id", "namespace": "com.test.app_list", "doc": "Device ID", "fields": [ { "nam
[
{
"type": "record",
"name": "Id",
"namespace": "com.test.app_list",
"doc": "Device ID",
"fields": [
{
"name": "idType",
"type": "int"
},{
"name": "id",
"type": "string"
}
]
},
{
"type": "record",
"name": "AppList",
"namespace": "com.test.app_list",
"doc": "",
"fields": [
{
"name": "appId",
"type": "string",
"avro.java.string": "String"
},
{
"name": "timestamp",
"type": "long"
},
{
"name": "idList",
"type": [{"type": "array", "items": "com.test.app_list.Id"}]
}
]
}
]
和我的sql来创建表:
CREATE EXTERNAL TABLE app_list
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
TBLPROPERTIES (
'avro.schema.url'='/hive/schema/test_app_list.avsc');
但蜂巢给了我:
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: MetaException(message:org.apache.hadoop.hive.serde2.avro.AvroSerdeException Schema for table must be of type RECORD. Received type: UNION)
hive
doc显示:支持任意嵌套的架构。
from:
数据样本:
{
"appId":{"string":"com.test.app"},
"timestamp":{"long":1495893601606},
"idList":{
"array":[
{"idType":15,"id":"6c:5c:14:c3:a5:39"},
{"idType":13,"id":"eb297afe56ff340b6bb7de5c5ab09193"}
]
}
}
但我不知道怎么做。我需要一些帮助来解决这个问题。谢谢 avro模式的顶层应该是记录类型,这就是为什么Hive不允许这样做的原因。解决方法可以是将顶层创建为记录,内部创建两个字段作为记录类型
{
"type": "record",
"name": "myRecord",
"namespace": "com.test.app_list"
"fields": [
{
"type": "record",
"name": "Id",
"doc": "Device ID",
"fields": [
{
"name": "idType",
"type": "int"
},{
"name": "id",
"type": "string"
}
]
},
{
"type": "record",
"name": "AppList",
"doc": "",
"fields": [
{
"name": "appId",
"type": "string",
"avro.java.string": "String"
},
{
"name": "timestamp",
"type": "long"
},
{
"name": "idList",
"type": [{"type": "array", "items": "com.test.app_list.Id"}]
}
]
}
]
}
已经添加了数据样本。谢谢你。我根据你的提示更改了模式。它是有效的。