Python 词频计数奇怪的结果

在对我的语料库进行词频统计时，结果似乎不准确（我感觉不是最频繁的词，而且频率统计只有一两个），一些结果显示“as over\xe2”和“\xad”。有人能帮忙吗

def toptenwords(mycorpus):
  mywords = mycorpus.words()
  nocapitals = [word.lower() for word in mywords] 
  filtered = [word for word in nocapitals if word not in stoplist]
  nopunctuation= [s.translate(None, string.punctuation) for s in filtered] 
  woordcounter = {}
     for word in nopunctuation:
        if word in wordcounter:
          woordcounter[word] += 1
        else:
          woordcounter[word] = 1
    frequentwords = sorted(wordcounter.iteritems(), key = itemgetter(1), reverse = True)
    top10 = frequentwords[:10]
    woord1 = frequentwords[1]
    woord2 = frequentwords[2]
    woord3 = frequentwords[3]
    woord4 = frequentwords[4]
    woord5 = frequentwords[5]
    woord6 = frequentwords[6]
    woord7 = frequentwords[7]
    woord8 = frequentwords[8]
    woord9 = frequentwords[9]
    woord10 = frequentwords[10]
    print "De 10 meest frequente woorden zijn: ", woord1, ",", woord2, ",", woord3, ",",    woord4, ",", woord5, ",", woord6, ",", woord7, ",", woord8, ",", woord9, "en", woord10

代码最初是荷兰语，这是未翻译的代码：

def toptienwoorden(mycorpus):
   woorden = mycorpus.words()
   zonderhoofdletters = [word.lower() for word in woorden] 
   gefiltered = [word for word in zonderhoofdletters if word not in stoplijst] 
   geenleestekens = [s.translate(None, string.punctuation) for s in gefiltered] 
   woordteller = {}
   for word in geenleestekens:
      if word in woordteller:
         woordteller[word] += 1
      else:
         woordteller[word] = 1
  frequentewoorden = sorted(woordteller.iteritems(), key = itemgetter(1), reverse = True)
  top10 = frequentewoorden[:10]
  woord1 = frequentewoorden[1]
  woord2 = frequentewoorden[2]
  woord3 = frequentewoorden[3]
  woord4 = frequentewoorden[4]
  woord5 = frequentewoorden[5]
  woord6 = frequentewoorden[6]
  woord7 = frequentewoorden[7]
  woord8 = frequentewoorden[8]
  woord9 = frequentewoorden[9]
  woord10 = frequentewoorden[10]
  print "De 10 meest frequente woorden zijn: ", woord1, ",", woord2, ",", woord3, ",",    woord4, ",", woord5, ",", woord6, ",", woord7, ",", woord8, ",", woord9, "en", woord10

使用一个。它非常适合计算（散列）项的频率，它有一个

最常用的
方法，可以返回前十个最频繁的项，而无需您自己编写逻辑代码：

import string
import collections


def topNwords(mywords, N = 10, stoplist = set(), filtered = set()):
    # mywords = mycorpus.words()
    nocapitals = [word.lower() for word in mywords]
    filtered = [word for word in nocapitals if word not in stoplist]
    nopunctuation = [s.translate(None, string.punctuation) for s in filtered]
    woordcounter = collections.Counter(nopunctuation)
    top_ten = [word for word, freq in woordcounter.most_common(N)]
    return top_ten


top_ten = topNwords('This is a test. It is only a test. In case of a real emergency'.split(), N = 10)
print("De 10 meest frequente woorden zijn: {w}".format(w = ', '.join(top_ten)))

你必须给我们看一些代码。给我们看代码和数据：Pand wierd字母是unicode字符，
“带扬抑符的拉丁小写字母A”
和
“软连字符”
：PI将其从荷兰语翻译过来，这样你就知道了参数的含义。我现在就用荷兰语发。最明显的问题是，最常用的单词是
frequentewoorden[0]
，你会默默地忽略它。虽然这是最好的方式，但我还是忍不住觉得你只是对一个家庭作业问题给出了一个完整的答案……不是真的，因为我没有最新的python版本，因为我使用的是NLTK。因此，“counter”函数在这个版本的Python上不起作用。不过，对于其他成员来说，anwser可能仍然是一个不错的工具。@JohnDoe:如果您使用的是
NLTK
，请查看。