Python CountVectorizer失败,出现了一些不好的词
我使用的是pandas数据帧,我试图获取具有字符串的特定列的单词出现次数。代码运行良好,直到某行出现以下错误Python CountVectorizer失败,出现了一些不好的词,python,pandas,scikit-learn,countvectorizer,Python,Pandas,Scikit Learn,Countvectorizer,我使用的是pandas数据帧,我试图获取具有字符串的特定列的单词出现次数。代码运行良好,直到某行出现以下错误 --------------------------------------------------------------------------- ValueError Traceback (most recent call last) <ipython-input-36-af8291199984> in &l
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-36-af8291199984> in <module>
6
7 cv = CountVectorizer(stop_words=None)
----> 8 cv_fit=cv.fit_transform(texts)
9 word_list = cv.get_feature_names();
10 count_list = cv_fit.toarray().sum(axis=0)
~/anaconda3/envs/turiCreate/lib/python3.8/site-packages/sklearn/feature_extraction/text.py in fit_transform(self, raw_documents, y)
1196 max_features = self.max_features
1197
-> 1198 vocabulary, X = self._count_vocab(raw_documents,
1199 self.fixed_vocabulary_)
1200
~/anaconda3/envs/turiCreate/lib/python3.8/site-packages/sklearn/feature_extraction/text.py in _count_vocab(self, raw_documents, fixed_vocab)
1127 vocabulary = dict(vocabulary)
1128 if not vocabulary:
-> 1129 raise ValueError("empty vocabulary; perhaps the documents only"
1130 " contain stop words")
1131
ValueError: empty vocabulary; perhaps the documents only contain stop words
如何使CountVectorizer克服此问题?您遇到的问题是使用标记化模式,在
标记\u模式='(?u)\\b\\w\\w+\\b'
中指定。您可以根据您的任务对其进行调整:
import pandas as pd
import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
texts=["hello :)"]
cv = CountVectorizer(stop_words=None, token_pattern=r'(?u)\b\w\w+\b|[:)]+')
cv_fit=cv.fit_transform(texts)
word_list = cv.get_feature_names();
count_list = cv_fit.toarray().sum(axis=0)
print(word_list)
print(dict(zip(word_list,count_list)))
[':)', 'hello']
{':)': 1, 'hello': 1}
如果您关心emojis,那么spacy可以为您的目标提供一个更强大的解决方案(“工业”如他们所说):
导入空间
从spacymoji导入表情符号
从收款进口柜台
nlp=spacy.load('en\u core\u web\u sm')
表情符号=表情符号(nlp)
nlp.add_管道(表情符号,first=True)
tokens=[tok for tok in nlp.tokenizer(“嗨:)非常感谢。
import pandas as pd
import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
texts=["hello :)"]
cv = CountVectorizer(stop_words=None, token_pattern=r'(?u)\b\w\w+\b|[:)]+')
cv_fit=cv.fit_transform(texts)
word_list = cv.get_feature_names();
count_list = cv_fit.toarray().sum(axis=0)
print(word_list)
print(dict(zip(word_list,count_list)))
[':)', 'hello']
{':)': 1, 'hello': 1}