Ruby 如何清理和改进关键字列表？
在使用关键字API获取流行关键字和短语之后，我还得到了很多带有太多额外单词的“肮脏”术语（“the”、“a”等） 我还想在搜索词中隔离名称 是否有一个Ruby库来清理关键字列表？这样的算法存在吗？你说的是“stopwords”，它是演讲稿，如“the”和“a”，加上经常遇到的毫无价值的单词 存在停止词列表；如果我没记错的话，它有一个，在Ruby或模块中可能有一个，但实际上它们很容易自己生成。而且，你可能需要这样做，因为垃圾词因特定主题而异 最简单的方法是使用几个示例文档运行一个初步过程，将文本拆分为单词，然后循环它们，每增加一个计数器。当你写完的时候，寻找那些两到四个字母长的单词，它们的数量要多得不成比例。这些都是很好的停止语候选词 然后，run遍历目标文档，像以前一样分割文本，边运行边计算出现次数。您可以忽略停止字列表中的字，而不将其添加到哈希中，或者处理所有内容，然后删除停止字Ruby 如何清理和改进关键字列表？,ruby,api,keyword,Ruby,Api,Keyword,在使用关键字API获取流行关键字和短语之后，我还得到了很多带有太多额外单词的“肮脏”术语（“the”、“a”等） 我还想在搜索词中隔离名称 是否有一个Ruby库来清理关键字列表？这样的算法存在吗？你说的是“stopwords”，它是演讲稿，如“the”和“a”，加上经常遇到的毫无价值的单词 存在停止词列表；如果我没记错的话，它有一个，在Ruby或模块中可能有一个，但实际上它们很容易自己生成。而且，你可能需要这样做，因为垃圾词因特定主题而异 最简单的方法是使用几个示例文档运行一个初步过程，将文本拆
text = <<EOT
You have reached this web page by typing "example.com", "example.net","example.org"
or "example.edu" into your web browser.
These domain names are reserved for use in documentation and are not available
for registration. See RFC 2606, Section 3.
EOT
# do this against several documents to build a stopword list. Tweak as necessary to fine-tune the words.
stopwords = text.downcase.split(/\W+/).inject(Hash.new(0)) { |h,w| h[w] += 1; h }.select{ |n,v| n.length < 5 }
print "Stopwords => ", stopwords.keys.sort.join(', '), "\n"
# >> Stopwords => 2606, 3, and, are, by, com, edu, for, have, in, into, net, not, or, org, page, rfc, see, this, use, web, you, your
text=>Stopwords=>2606，3和，是，由，com，edu，for，have，in，into，net，not，or，org，page，rfc，see，this，use，web，you，your
text = <<EOT
You have reached this web page by typing "example.com", "example.net","example.org"
or "example.edu" into your web browser.
These domain names are reserved for use in documentation and are not available
for registration. See RFC 2606, Section 3.
EOT
stopwords = %w[2606 3 and are by com edu for have in into net not or org page rfc see this use web you your]
keywords = text.downcase.split(/\W+/).inject(Hash.new(0)) { |h,w| h[w] += 1; h }
stopwords.each { |s| keywords.delete(s) }
# output in order of most often seen to least often seen.
keywords.keys.sort{ |a,b| keywords[b] <=> keywords[a] }.each { |k| puts "#{k} => #{keywords[k]}"}
# >> example => 4
# >> names => 1
# >> reached => 1
# >> browser => 1
# >> these => 1
# >> domain => 1
# >> typing => 1
# >> reserved => 1
# >> documentation => 1
# >> available => 1
# >> registration => 1
# >> section => 1
text=>example=>4
#>>名称=>1
#>>达到=>1
#>>浏览器=>1
#>>这些=>1
#>>域=>1
#>>键入=>1
#>>保留=>1
#>>文档=>1
#>>可用=>1
#>>注册=>1
#>>部分=>1
text = <<EOT
You have reached this web page by typing "example.com", "example.net","example.org"
or "example.edu" into your web browser.
These domain names are reserved for use in documentation and are not available
for registration. See RFC 2606, Section 3.
EOT
# do this against several documents to build a stopword list. Tweak as necessary to fine-tune the words.
stopwords = text.downcase.split(/\W+/).inject(Hash.new(0)) { |h,w| h[w] += 1; h }.select{ |n,v| n.length < 5 }
print "Stopwords => ", stopwords.keys.sort.join(', '), "\n"
# >> Stopwords => 2606, 3, and, are, by, com, edu, for, have, in, into, net, not, or, org, page, rfc, see, this, use, web, you, your
text=>Stopwords=>2606，3和，是，由，com，edu，for，have，in，into，net，not，or，org，page，rfc，see，this，use，web，you，your
text = <<EOT
You have reached this web page by typing "example.com", "example.net","example.org"
or "example.edu" into your web browser.
These domain names are reserved for use in documentation and are not available
for registration. See RFC 2606, Section 3.
EOT
stopwords = %w[2606 3 and are by com edu for have in into net not or org page rfc see this use web you your]
keywords = text.downcase.split(/\W+/).inject(Hash.new(0)) { |h,w| h[w] += 1; h }
stopwords.each { |s| keywords.delete(s) }
# output in order of most often seen to least often seen.
keywords.keys.sort{ |a,b| keywords[b] <=> keywords[a] }.each { |k| puts "#{k} => #{keywords[k]}"}
# >> example => 4
# >> names => 1
# >> reached => 1
# >> browser => 1
# >> these => 1
# >> domain => 1
# >> typing => 1
# >> reserved => 1
# >> documentation => 1
# >> available => 1
# >> registration => 1
# >> section => 1
text=>example=>4
#>>名称=>1
#>>达到=>1
#>>浏览器=>1
#>>这些=>1
#>>域=>1
#>>键入=>1
#>>保留=>1
#>>文档=>1
#>>可用=>1
#>>注册=>1
#>>部分=>1
缩小单词列表后，您可以通过WordNet运行候选词，查找同义词、同音词、单词关系、带复数等。如果您要对大量文本执行此操作，您需要将停止词保存在数据库中，以便不断对其进行微调。同样的道理也适用于你的关键词，因为你可以从这些关键词开始判断语调和其他语义的优点。顺便说一句，我决定走这条路：
bad_words = ["the", "a", "for", "on"] #etc etc
# Strip non alpha chars, and split into a temp array, then cut out the bad words
tmp_str = str.gsub(/[^A-Za-z0-9\s]/, "").split - bad_words
str = tmp_str.join(" ")
顺便说一句，我决定走这条路线：
bad_words = ["the", "a", "for", "on"] #etc etc
# Strip non alpha chars, and split into a temp array, then cut out the bad words
tmp_str = str.gsub(/[^A-Za-z0-9\s]/, "").split - bad_words
str = tmp_str.join(" ")
我在你的标签列表中添加了
rubyruby