Nlp 从三元图列表生成单元图和双元图

我正在寻找一种可能的方法，只需将三元图频率存储在内存中，并通过以下方式动态计算单元图和双元图频率：

给定一个三元组u，v，w：

计数（v，w）=总和（v，w），即所有u的总和

类似地，计数（w）=和（，w）

---

这确实会导致缺少一些单字，例如句子开始标记，但这听起来像是生成单字和双字的有效方法吗？

是的。那就行了。你可以通过自己制作一个小语料库并手动进行计数来检查它，以确保结果相同

from collections import Counter

corpus = [['the','dog','walks'], ['the','dog','runs'], ['the','cat','runs']]
corpus_with_ends = [['<s>','<s>'] + s + ['<e>'] for s in corpus]

trigram_counts = Counter(trigram for s in corpus_with_ends for trigram in zip(s,s[1:],s[2:]))

unique_bigrams = set((b,c) for a,b,c in trigram_counts)
bigram_counts = dict((bigram,sum(count for trigram,count in trigram_counts.iteritems() if trigram[1:] == bigram)) for bigram in unique_bigrams)

unique_unigrams = set((c,) for a,b,c in trigram_counts if c != '<e>')
unigram_counts = dict((unigram,sum(count for trigram,count in trigram_counts.iteritems() if trigram[2:] == unigram)) for unigram in unique_unigrams)

>>> true_bigrams = [bigram for s in corpus_with_ends for bigram in zip(s[1:],s[2:])]
>>> true_bigram_counts = Counter(true_bigrams)
>>> bigram_counts == true_bigram_counts
True

>>> true_unigrams = [(unigram,) for s in corpus_with_ends for unigram in s[2:-1]]
>>> true_unigram_counts = Counter(true_unigrams)
>>> unigram_counts == true_unigram_counts
True