Performance Python-将GridSearchCV与NLTK结合使用
我有点不确定如何将SKLearn的GridSearchCV应用到我与NLTK一起使用的随机林中。讨论了如何正常使用GridSearchCV,但是我的数据格式与标准的x和y分割不同。这是我的密码:Performance Python-将GridSearchCV与NLTK结合使用,performance,scikit-learn,nltk,random-forest,hyperparameters,Performance,Scikit Learn,Nltk,Random Forest,Hyperparameters,我有点不确定如何将SKLearn的GridSearchCV应用到我与NLTK一起使用的随机林中。讨论了如何正常使用GridSearchCV,但是我的数据格式与标准的x和y分割不同。这是我的密码: import nltk import numpy as np from nltk.classify.scikitlearn import SklearnClassifier from nltk.corpus.reader import CategorizedPlaintextCorpusReader f
import nltk
import numpy as np
from nltk.classify.scikitlearn import SklearnClassifier
from nltk.corpus.reader import CategorizedPlaintextCorpusReader
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
reader_train = CategorizedPlaintextCorpusReader('C:/Users/User/Documents/Sentiment/machine_learning/amazon/amazon/', r'.*\.txt', cat_pattern=r'(\w+)/*', encoding='latin1')
documents_train = [ (list(reader_train.words(fileid)), category)
for category in reader_train.categories()
for fileid in reader_train.fileids(category) ]
all_words = []
for w in reader_train.words():
all_words.append(w.lower())
all_words = nltk.FreqDist(all_words)
word_features = list(all_words.keys())[:3500]
def find_features(documents):
words = set(documents)
features = {}
for w in word_features:
features[w] = (w in words)
return features
featuresets_train = [(find_features(rev), category) for (rev, category) in documents_train]
np.random.shuffle(featuresets_train)
training_set = featuresets_train[:1600]
testing_set = featuresets_train[:400]
RandFor = SklearnClassifier(RandomForestClassifier())
RandFor.train(training_set)
print("RandFor accuracy:", (nltk.classify.accuracy(RandFor, testing_set)) *100)
此代码不是生成常规的x和y拆分,而是生成一个元组列表,其中每个元组的格式如下:
({'i': True, 'am': False, 'conflicted': False ... 'about': False}, neg)
有没有办法将GridSearchCV应用于此格式的数据?要将
培训集
转换为scikit可用表单,只需执行以下操作
from sklearn.feature_extraction import DictVectorizer
vectorizer = DictVectorizer()
X_train, y_train = list(zip(*training_set))
X_train = vectorizer.fit_transform(X_train)
X_test, y_test = list(zip(*testing_set))
X_test = vectorizer.transform(X_test)
之后,你可以很容易地打电话
clf = RandomForestClassifier()
clf.fit(X_train, y_train)
accuracy = clf.score(X_test, y_test)
print("RandFor accuracy:", (accuracy) * 100)
您正在将已经非常完美的scikit学习估计器(
RandomForestClassifier
)包装为与nltk兼容的估计器。你需要使用RandomForestClassifier
和GridSearchCV
吗?GridSearchCV不是必须的,我很乐意尝试其他优化算法。不过,这确实需要改进。谢谢你,伙计,非常感谢,而且似乎工作得很好!祝您今天过得愉快。