Hive 使用Apache配置单元对日志数据进行会话的更好方法?
有没有更好的方法使用Apache配置单元对日志数据进行会话化?我不确定我是否以最佳方式在下面这样做: 日志数据存储在序列文件中;单个日志条目是一个JSON字符串;例如:Hive 使用Apache配置单元对日志数据进行会话的更好方法?,hive,Hive,有没有更好的方法使用Apache配置单元对日志数据进行会话化?我不确定我是否以最佳方式在下面这样做: 日志数据存储在序列文件中;单个日志条目是一个JSON字符串;例如: {“source”:{“api_key”:“app_key_1”,“user_id”:“user0”},“events”:[{“timestamp”:1330988326,“event_type”:“high_score”,“event_params”:{“score”:“1123”,“level”:“9”},{“timesta
{“source”:{“api_key”:“app_key_1”,“user_id”:“user0”},“events”:[{“timestamp”:1330988326,“event_type”:“high_score”,“event_params”:{“score”:“1123”,“level”:“9”},{“timestamp”:1330987183,“event_type”:“some_event_event_0”,“event_params”:{“some_param”:“val”,“some_param”:“val”,“some_param”:100},{“timestamp”;“eventͫ133;“一些事件1”,“事件参数”:{“一些参数11”:100,“一些参数10”:“val”}]}
格式化后,这看起来像:
{'source': {'api_key': 'app_key_1', 'user_id': 'user0'},
'events': [{'event_params': {'level': '9', 'score': '1123'},
'event_type': 'high_score',
'timestamp': 1330988326},
{'event_params': {'some_param_00': 'val', 'some_param_01': 100},
'event_type': 'some_event_0',
'timestamp': 1330987183},
{'event_params': {'some_param_10': 'val', 'some_param_11': 100},
'event_type': 'some_event_1',
'timestamp': 1330987775}]
}
“源”包含有关“事件”中包含的事件源的一些信息(用户id和api密钥);“事件”包含由源生成的事件列表;每个事件都有“事件参数”、“事件类型”和“时间戳”(时间戳是GMT中的Unix时间戳)。请注意,单个日志条目内和跨日志条目的时间戳可能不符合顺序
请注意,我受到限制,无法更改日志格式,无法将数据最初记录到分区的单独文件中(尽管我可以在记录数据后使用配置单元来完成此操作),等等
最后,我想要一个会话表,其中会话与应用程序(api_k)和用户关联,并且具有开始时间和会话长度(或结束时间);会话被拆分,对于给定的应用程序和用户,事件之间的间隔为30分钟或更长
我的解决方案执行以下操作(配置单元脚本和python转换脚本如下;显示SerDe源代码似乎没有什么用处,但请告诉我是否有用):
[1] 以非规范化格式将数据加载到log_entry_tmp中
[2] 将数据分解为log_条目,这样,例如,上面的单个条目现在将有多个条目:
{"source_api_key":"app_key_1","source_user_id":"user0","event_type":"high_score","event_params":{"score":"1123","level":"9"},"event_timestamp":1330988326}
{"source_api_key":"app_key_1","source_user_id":"user0","event_type":"some_event_0","event_params":{"some_param_00":"val","some_param_01":"100"},"event_timestamp":1330987183}
{"source_api_key":"app_key_1","source_user_id":"user0","event_type":"some_event_1","event_params":{"some_param_11":"100","some_param_10":"val"},"event_timestamp":1330987775}
[3] 将数据转换并写入会话信息0,其中每个条目包含事件的应用程序id、用户id和时间戳
[4] 转换并将数据写入会话\信息\ 1,其中条目按应用\ id、用户\ id、事件\时间戳排序;每个条目都包含一个会话\ id;python转换脚本查找拆分,并将数据分组到会话中
[5] 转换并将最终会话数据写入会话信息2;会话的应用程序+用户、开始时间和长度(以秒为单位)
[蜂巢脚本]
drop table if exists app_info;
create external table app_info ( app_id int, app_name string, api_k string )
location '${WORK}/hive_tables/app_info';
add jar ../build/our-serdes.jar;
-- [1] load the data into log_entry_tmp, in a denormalized format
drop table if exists log_entry_tmp;
create external table log_entry_tmp
row format serde 'com.company.TestLogSerde'
location '${WORK}/hive_tables/test_logs';
drop table if exists log_entry;
create table log_entry (
entry struct<source_api_key:string,
source_user_id:string,
event_type:string,
event_params:map<string,string>,
event_timestamp:bigint>);
-- [2] explode the data into log_entry
insert overwrite table log_entry
select explode (trans0_list) t
from log_entry_tmp;
drop table if exists session_info_0;
create table session_info_0 (
app_id string,
user_id string,
event_timestamp bigint
);
-- [3] transform and write data into session_info_0, where each entry contains events' app_id, user_id, and timestamp
insert overwrite table session_info_0
select ai.app_id, le.entry.source_user_id, le.entry.event_timestamp
from log_entry le
join app_info ai on (le.entry.source_api_key = ai.api_k);
add file ./TestLogTrans.py;
drop table if exists session_info_1;
create table session_info_1 (
session_id string,
app_id string,
user_id string,
event_timestamp bigint,
session_start_datetime string,
session_start_timestamp bigint,
gap_secs int
);
-- [4] tranform and write data into session_info_1, where entries are ordered by app_id, user_id, event_timestamp ; and each entry contains a session_id ; the python tranform script finds the splits, and groups the data into sessions
insert overwrite table session_info_1
select
transform (t.app_id, t.user_id, t.event_timestamp)
using './TestLogTrans.py'
as (session_id, app_id, user_id, event_timestamp, session_start_datetime, session_start_timestamp, gap_secs)
from
(select app_id as app_id, user_id as user_id, event_timestamp as event_timestamp from session_info_0 order by app_id, user_id, event_timestamp ) t;
drop table if exists session_info_2;
create table session_info_2 (
session_id string,
app_id string,
user_id string,
session_start_datetime string,
session_start_timestamp bigint,
len_secs int
);
-- [5] transform and write final session data to session_info_2 ; the sessions' app + user, start time, and length in seconds
insert overwrite table session_info_2
select session_id, app_id, user_id, session_start_datetime, session_start_timestamp, sum(gap_secs)
from session_info_1
group by session_id, app_id, user_id, session_start_datetime, session_start_timestamp;
我认为您可以轻松地删除第3步,并将其中使用的查询作为第4步from子句的子查询。物理化该转换似乎不会给您带来任何好处 否则,我认为对于你在这里试图实现的目标来说,这似乎是一个合理的方法 步骤2可以使用自定义映射器实现,将输出作为custome reducer传递到步骤4(步骤3内置为子查询)。这将使mapreduce作业减少1,因此可以显著节省时间
#!/usr/bin/python
import sys, time
def buildDateTime(ts):
return time.strftime('%Y-%m-%d %H:%M:%S', time.gmtime(ts))
curGroup = None
prevGroup = None
curSessionStartTimestamp = None
curSessionStartDatetime = None
prevTimestamp = None
for line in sys.stdin.readlines():
fields = line.split('\t')
if len(fields) != 3:
raise Exception('fields = %s', fields)
app_id = fields[0]
user_id = fields[1]
event_timestamp = int(fields[2].strip())
curGroup = '%s-%s' % (app_id, user_id)
curTimestamp = event_timestamp
if prevGroup == None:
prevGroup = curGroup
curSessionStartTimestamp = curTimestamp
curSessionStartDatetime = buildDateTime(curSessionStartTimestamp)
prevTimestamp = curTimestamp
isNewGroup = (curGroup != prevGroup)
gapSecs = 0 if isNewGroup else (curTimestamp - prevTimestamp)
isSessionSplit = (gapSecs >= 1800)
if isNewGroup or isSessionSplit:
curSessionStartTimestamp = curTimestamp
curSessionStartDatetime = buildDateTime(curSessionStartTimestamp)
session_id = '%s-%s-%d' % (app_id, user_id, curSessionStartTimestamp)
print '%s\t%s\t%s\t%d\t%s\t%d\t%d' % (session_id, app_id, user_id, curTimestamp, curSessionStartDatetime, curSessionStartTimestamp, gapSecs)
prevGroup = curGroup
prevTimestamp = curTimestamp