Python 3.x 使用TFIDF矢量器的字频
我正在尝试使用TF-IDF计算消息传递数据帧的词频。到目前为止,我有这个Python 3.x 使用TFIDF矢量器的字频,python-3.x,pandas,word-frequency,tfidfvectorizer,Python 3.x,Pandas,Word Frequency,Tfidfvectorizer,我正在尝试使用TF-IDF计算消息传递数据帧的词频。到目前为止,我有这个 import nltk from sklearn.feature_extraction.text import TfidfVectorizer new_group['tokenized_sents'] = new_group.apply(lambda row: nltk.word_tokenize(row['message']),axis=1).astype(str).lower() vectoriser=TfidfVe
import nltk
from sklearn.feature_extraction.text import TfidfVectorizer
new_group['tokenized_sents'] = new_group.apply(lambda row: nltk.word_tokenize(row['message']),axis=1).astype(str).lower()
vectoriser=TfidfVectorizer()
new_group['tokenized_vector'] = list(vectoriser.fit_transform(new_group['tokenized_sents']).toarray())
然而,通过上面的代码,我得到了一堆零,而不是单词频率。如何解决此问题以获得消息的正确数字频率。这是我的数据帧
user_id date message tokenized_sents tokenized_vector
X35WQ0U8S 2019-02-17 Need help ['need','help'] [0.0,0.0]
X36WDMT2J 2019-03-22 Thank you! ['thank','you','!'] [0.0,0.0,0.0]
首先,对于计数,您不希望使用TfidfVectorizer,因为它是标准化的。您想使用CountVectorizer。其次,您不需要对单词进行标记,因为sklearn有一个内置的标记器,它同时具有TfidfVectorizer和CountVectorizer
#add whatever settings you want
countVec =CountVectorizer()
#fit transform
cv = countVec.fit_transform(df['message'].str.lower())
#feature names
cv_feature_names = countVec.get_feature_names()
#feature counts
feature_count = cv.toarray().sum(axis = 0)
#feature name to count
dict(zip(cv_feature_names, feature_count))