使用python在句子列表中形成单词的双字组并计算双字组
我需要: 1.形成双RAM对并将其存储在列表中 2.找到id的总和,其中有频率最高的前3个二元图 我有一个句子列表:使用python在句子列表中形成单词的双字组并计算双字组,python,python-3.x,list-comprehension,word-frequency,Python,Python 3.x,List Comprehension,Word Frequency,我需要: 1.形成双RAM对并将其存储在列表中 2.找到id的总和,其中有频率最高的前3个二元图 我有一个句子列表: [['22574999', 'your message communication sent'] , ['22582857', 'your message be delivered'] , ['22585166', 'message has be delivered'] , ['22585424', 'message originated communication sent']
[['22574999', 'your message communication sent']
, ['22582857', 'your message be delivered']
, ['22585166', 'message has be delivered']
, ['22585424', 'message originated communication sent']]
以下是我所做的:
for row in messages:
sstrm = list(row)
bigrams=[b for l in sstrm for b in zip(l.split(" ")[:1], l.split(" ")[1:])]
print(sstrm[0],bigrams)
这将产生:
22574999 [('your', 'message')]
22582857 [('[your', 'message')]
22585166 [('message', 'has')]
22585424 [('message', 'originated')]
我想要的是:
22574999 [('your', 'message'),('communication','sent')]
22582857 [('[your', 'message'),('be','delivered')]
22585166 [('message', 'has'),('be','delivered')]
22585424 [('message', 'originated'),('communication','sent')]
我想得到以下结果
结果:
频率最高的前3个大字:
('your', 'message') :2
('communication','sent'):2
('be','delivered'):2
('your', 'message'):2 Is included (22574999,22582857)
('communication','sent'):2 Is included(22574999,22585424)
('be','delivered'):2 Is included (22582857,22585166)
具有最高频率的前3个二元图的id总和:
('your', 'message') :2
('communication','sent'):2
('be','delivered'):2
('your', 'message'):2 Is included (22574999,22582857)
('communication','sent'):2 Is included(22574999,22585424)
('be','delivered'):2 Is included (22582857,22585166)
谢谢你的帮助 这一行中有一个错误:
bigrams=[b for l in sstrm for b in zip(l.split(" ")[:1], l.split(" ")[1:])]
在zip中的第一个参数中,您将使用[:1]
在列表的第一个元素处停止。您希望获取除最后一个元素之外的所有元素,该元素对应于[:-1]
所以这条线应该是这样的:
bigrams=[b for l in sstrm for b in zip(l.split(" ")[:-1], l.split(" ")[1:])]
我想指出的第一件事是,bigram是两个相邻元素的序列。
例如,“狐狸跳过了懒狗”的大公羊是:
这个问题可以使用一个模型来建模,其中bigram是posting,id集是posting列表
def bigrams(line):
tokens = line.split(" ")
return [(tokens[i], tokens[i+1]) for i in range(0, len(tokens)-1)]
if __name__ == "__main__":
messages = [['22574999', 'your message communication sent'], ['22582857', 'your message be delivered'], ['22585166', 'message has be delivered'], ['22585424', 'message originated communication sent']]
bigrams_set = set()
for row in messages:
l_bigrams = bigrams(row[1])
for bigram in l_bigrams:
bigrams_set.add(bigram)
inverted_idx = dict((b,[]) for b in bigrams_set)
for row in messages:
l_bigrams = bigrams(row[1])
for bigram in l_bigrams:
inverted_idx[bigram].append(row[0])
freq_bigrams = dict((b,len(ids)) for b,ids in inverted_idx.items())
import operator
top3_bigrams = sorted(freq_bigrams.iteritems(), key=operator.itemgetter(1), reverse=True)[:3]
输出
[(('communication', 'sent'), 2), (('your', 'message'), 2), (('be', 'delivered'), 2)]
虽然这段代码可以进行大量优化,但它提供了一个想法