Python 从反向索引Elasticsearch按高频项顺序排序字符串
我是Elasticsearch的新手,我想知道这样做是否可行: 我有一堆地址字符串,我想根据字符串中最重复的术语进行排序 例如:Python 从反向索引Elasticsearch按高频项顺序排序字符串,python,elasticsearch,Python,elasticsearch,我是Elasticsearch的新手,我想知道这样做是否可行: 我有一堆地址字符串,我想根据字符串中最重复的术语进行排序 例如: 1. Shop no 1 ABC Lane City1 - Zipcode1 2. Shop no 2 EFG Lane City1 - Zipcode2 3. Shop no 1 XYZ Lane City2 - Zipcode3 4. Shop no 3 ABC Lane City1 - Zipcode1 我真正需要的是根据字符串中最常见的术语将它们组合在一起
1. Shop no 1 ABC Lane City1 - Zipcode1
2. Shop no 2 EFG Lane City1 - Zipcode2
3. Shop no 1 XYZ Lane City2 - Zipcode3
4. Shop no 3 ABC Lane City1 - Zipcode1
我真正需要的是根据字符串中最常见的术语将它们组合在一起
因此,对于前面的示例,排序输出应该是:
1. Shop no 1 ABC Lane City1 - Zipcode1
4. Shop no 3 ABC Lane City1 - Zipcode1 # Because 1 and 2 have the most common words in them.
2. Shop no 2 EFG Lane City1 - Zipcode2 # Second most common words with 1 and 4.
3. Shop no 1 XYZ Lane City2 - Zipcode3 # Not all that many common terms amongst them.
我不知道该怎么办。
我知道我可以将每个字符串作为一个查询激发,以获得最接近被激发查询的结果。
但我有十万行,这似乎根本不是一个有效的选择
如果我可以使用term
过滤器对每个字符串中重复出现的术语数量最多的matchall()
和sort
,那将非常有用
是否可以对反向索引中包含大多数相似单词的文档进行排序
以下是我的数据外观的示例粘贴箱:
解决方案 我曾经计算两个字符串的长度(归功于@vpekar),作为相似度的基本算法。一般来说,我把所有的字符串都放在一个列表中。然后我将索引参数I设置为0,并循环I,只要它在列表长度的范围内。在该循环中,我将位置p从I+1迭代到length(列表)。然后我找到列表[I]和列表[p]之间的最大余弦值。这两个文本字符串将被放入一个输出列表中,以便在以后的相似性计算中不考虑它们。两个文本字符串将与余弦值一起放入结果列表,数据结构为VectorResult 之后,列表按余弦值排序。现在我们有了唯一的字符串对,具有递减余弦,也称为相似值。嗯
import re
import math
import timeit
from collections import Counter
WORD = re.compile(r'\w+')
def get_cosine(vec1, vec2):
intersection = set(vec1.keys()) & set(vec2.keys())
numerator = sum([vec1[x] * vec2[x] for x in intersection])
sum1 = sum([vec1[x] ** 2 for x in vec1.keys()])
sum2 = sum([vec2[x] ** 2 for x in vec2.keys()])
denominator = math.sqrt(sum1) * math.sqrt(sum2)
if not denominator:
return 0.0
else:
return float(numerator) / denominator
def text_to_vector(text):
words = WORD.findall(text)
return Counter(words)
class VectorResult(object):
def __init__(self, cosine, text_1, text_2):
self.cosine = cosine
self.text_1 = text_1
self.text_2 = text_2
def __eq__(self, other):
if self.cosine == other.cosine:
return True
return False
def __le__(self, other):
if self.cosine <= other.cosine:
return True
return False
def __ge__(self, other):
if self.cosine >= other.cosine:
return True
return False
def __lt__(self, other):
if self.cosine < other.cosine:
return True
return False
def __gt__(self, other):
if self.cosine > other.cosine:
return True
return False
def main():
start = timeit.default_timer()
texts = []
with open('data.txt', 'r') as f:
texts = f.readlines()
cosmap = []
i = 0
out = []
while i < len(texts):
max_cosine = 0.0
current = None
for p in range(i + 1, len(texts)):
if texts[i] in out or texts[p] in out:
continue
vector1 = text_to_vector(texts[i])
vector2 = text_to_vector(texts[p])
cosine = get_cosine(vector1, vector2)
if cosine > max_cosine:
current = VectorResult(cosine, texts[i], texts[p])
max_cosine = cosine
if current:
out.extend([current.text_1, current.text_2])
cosmap.append(current)
i += 1
cosmap = sorted(cosmap)
for item in reversed(cosmap):
print(item.cosine, item.text_1, item.text_2)
end = timeit.default_timer()
print("Similarity Sorting of {} strings lasted {} s.".format(len(texts), end - start))
if __name__ == '__main__':
main()
余弦相似性绝对是一条出路 Igor Motov创建了一个Elasticsearch原生脚本来计算多个文档中某个字段的相似性值
您可以在内部或外部使用此脚本 我不知道elasticsearch,但python至少可以做一个单词计数器,然后使用它构建一个键函数来进行排序,比如说,每个词中每个单词的计数之和,我已经回答了,并展示了如何使用简单(简单)的python方法来使用余弦相似性进行排序。但是,如果您依赖于发出弹性搜索查询,那么您可能应该看看MLT查询()。
1.0000000000000002 NO 15& 16 1ST FLOOR,2ND MAIN ROAD,KHB COLONY,GANDINAGAR YELAHANKA
NO 15& 16 1ST FLOOR,2ND MAIN ROAD,KHB COLONY,GANDINAGAR YELAHANKA
1.0 # 51/3 AGRAHARA YELAHANKA
#51/3 AGRAHARA YELAHANKA
0.9999999999999999 # C M C ROAD,YALAHANKA
# C M C ROAD,YALAHANKA
0.8728715609439696 # 1002/B B B ROAD,YELAHANKA
0,B B ROAD,YELAHANKA
0.8432740427115678 # LAKSHMI COMPLEX C M C ROAD,YALAHANKA
# SRI LAKSHMAN COMPLEX C M C ROAD,YALAHANKA
0.8333333333333335 # 85/1 B B M P OFFICE ROAD,KOGILU YELAHANKA
#85/1 B B M P OFFICE NEAR KOGILU YALAHANKA
0.8249579113843053 # 689 3RD A CROSS SHESHADRIPURAM CALLEGE OPP YELAHANKA
# 715 3RD CROSS A SECTUR SHESHADRIPURAM CALLEGE OPP YELAHANKA
0.8249579113843053 # 10 RAMAIAIA COMPLEX B B ROAD,YALAHANKA
# JAMATI COMPLEX B B ROAD,YALAHANKA
[ SNIPPED ]
Similarity Sorting of 702 strings lasted 8.955146235887025 s.