Performance 蜂巢:有没有更好的方法对一列进行百分位排序?
目前,为了在配置单元中对列进行百分位排序,我使用了如下方法。我试图按照项目所属的百分比对列中的项目进行排序,为每个项目分配一个0到1的值。下面的代码指定了一个从0到9的值,基本上表示Performance 蜂巢:有没有更好的方法对一列进行百分位排序?,performance,hadoop,hive,rank,percentile,Performance,Hadoop,Hive,Rank,Percentile,目前,为了在配置单元中对列进行百分位排序,我使用了如下方法。我试图按照项目所属的百分比对列中的项目进行排序,为每个项目分配一个0到1的值。下面的代码指定了一个从0到9的值,基本上表示char\u percentile\u rank为0的项目位于项目的底部10%,值9位于项目的顶部10%。有更好的方法吗 select item , characteristic , case when characteristic <= char_perc[0] then 0
char\u percentile\u rank
为0的项目位于项目的底部10%,值9位于项目的顶部10%。有更好的方法吗
select item
, characteristic
, case when characteristic <= char_perc[0] then 0
when characteristic <= char_perc[1] then 1
when characteristic <= char_perc[2] then 2
when characteristic <= char_perc[3] then 3
when characteristic <= char_perc[4] then 4
when characteristic <= char_perc[5] then 5
when characteristic <= char_perc[6] then 6
when characteristic <= char_perc[7] then 7
when characteristic <= char_perc[8] then 8
else 9
end as char_percentile_rank
from (
select split(item_id,'-')[0] as item
, split(item_id,'-')[1] as characteristic
, char_perc
from (
select collect_set(concat_ws('-',item,characteristic)) as item_set
, PERCENTILE(BIGINT(characteristic),array(0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9)) as char_perc
from(
select item
, sum(characteristic) as characteristic
from table
group by item
) t1
) t2
lateral view explode(item_set) explodetable as item_id
) t3
选择项目
特征
,case when characteristic尝试删除一个派生表
select item
, characteristic
, case when characteristic <= char_perc[0] then 0
when characteristic <= char_perc[1] then 1
when characteristic <= char_perc[2] then 2
when characteristic <= char_perc[3] then 3
when characteristic <= char_perc[4] then 4
when characteristic <= char_perc[5] then 5
when characteristic <= char_perc[6] then 6
when characteristic <= char_perc[7] then 7
when characteristic <= char_perc[8] then 8
else 9
end as char_percentile_rank
from (
select item, characteristic,
, PERCENTILE(BIGINT(characteristic),array(0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9)) over () as char_perc
from (
select item
, sum(characteristic) as characteristic
from table
group by item
) t1
) t2
选择项目
特征
,case-when-characteristic百分位函数需要一个group-by-这就是为什么我做了看起来不必要的collect_集(concat_-ws(…)-以便使它成为一个组。如果我改为按项目和特征分组,那么它将按项目和特征分组,这是我不想要的。有什么办法可以避免吗?我几乎是在睡了一觉后马上想出来的。只需在百分位数上加上over()。所以这行应该是:PERCENTILE(BIGINT(特征),array(0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9))over()作为char_perc。但我的原始查询和新查询的性能似乎非常相似。@CharlieHaley是否可以引用PERCENTILE(BIGINT…[i]
,其中i=0-8
?如果是这样,您可以移动case语句并删除1table@CharlieHaley另一个建议是,是否可以将百分位数(BIGINT(sum(characteristic),…)作为characteristic移到()上方的最里面的查询中?因为我认为窗口函数在聚合之后运行它看起来像是在两个方向上移动百分位数(向内或向外)不影响语法树,因此也不应该影响性能。但它确实使查询看起来更美观,所以就是这样。