Kdb 多列上的慢速聚合
这个聚合多个列的kdb查询大约需要31秒，而使用J需要3秒 有没有更快的方法用kdb求和 最终，这将针对32位版本上的分区数据库运行Kdb 多列上的慢速聚合,kdb,Kdb,这个聚合多个列的kdb查询大约需要31秒，而使用J需要3秒 有没有更快的方法用kdb求和 最终，这将针对32位版本上的分区数据库运行 /test 1 - using symbols n: 13000000; cust: n?`8; prod: n?`8; v: n?100 a:([]cust:cust; prod:prod ;v:v) /query 1 - using simple by q)\t select sum(v) by cust, prod from a 31058 /quer
/test 1 - using symbols
n: 13000000;
cust: n?`8;
prod: n?`8;
v: n?100
a:([]cust:cust; prod:prod ;v:v)
/query 1 - using simple by
q)\t select sum(v) by cust, prod from a
31058
/query 2 - grouping manually
\t {sum each x[`v][group[flip (x[`cust]; x[`prod])]]}(select v, cust, prod from a)
12887
/query 3 - simpler method of accessing
\t {sum each a.v[group x]} (flip (a.cust;a.prod))
11576
/test 2 - using strings, very slow
n: 13000000;
cust: string n?`8;
prod: string n?`8;
v: n?100
a:([]cust:cust; prod:prod ;v:v)
q)\t select sum(v) by cust, prod from a
116745
n=:13000000
cust=: _8[\ a. {~ (65+?(8*n)#26)
prod=: _8[\ a. {~ (65+?(8*n)#26)
v=: ?.n#100
agg=: 3 : 0
keys=:i.~ |: i.~ every (cust;prod)
c=.((~.keys) { cust)
p=.((~.keys) { prod)
s=.keys +//. v
c;p;s
)
NB. 3.57 seconds
6!:2 'r=.agg 0'
3.57139
({.@$) every r
13000000 13000000 13000000
q)\t r:(`cust`prod xkey a inds) + select sum v by cust,prod from a til[count a] except inds:(select cust,prod from a) ? d:distinct select cust,prod from a
6809
n: 2500000
g: 4
v: (g*n)?100
cust: (g*n)#(n?`8)
prod: (g*n)#(n?`8)
b:([]cust:cust; prod:prod ;v:v)
q)\ts select sum v by cust, prod from b
9737 838861968
q)\ts r:(`cust`prod xkey b inds) + select sum v by cust,prod from a til[count b] except inds:(select cust,prod from b) ? d:distinct select cust,prod from b
17181 671090384
如果您使用的是病理数据集，那么一组长度为8的随机符号将几乎没有重复项，这使得分组变得多余
q)n:13000000; (count distinct n?`8)%n
0.9984848
q)n:1000000
q)
q)a:([]cust:n?`8; prod:n?`8; v:n?100)
q)b:([]cust:n?`3; prod:n?`3; v:n?100)
q)
q)\ts select sum v by cust, prod from a
3779 92275568
q)
q)\ts select sum v by cust, prod from b
762 58786352
如果更新此数据的频率低于查询此数据的频率，那么预计算组索引如何？创建一个查询的成本与创建一个查询的成本差不多，并且它允许以大约30倍的速度进行查询
q)\ts select sum v by cust,prod from b
14014 838861360
q)\ts update g:`g#{(key group x)?x}flip(cust;prod)from`b
14934 1058198384
q)\ts select first cust,first prod,sum v by g from b
473 201327488
q)
q)(select sum v by cust,prod from b)~`cust`prod xasc 2!delete g from select first cust,first prod,sum v by g from b
1b
q)
（顺便说一句，我对J基本一无所知，但我猜测它正在计算一个类似的多列组索引仅限于纯向量数据，如果可以以某种方式将其应用于
custprodpg