Keras 重塑输入标记化文本时出错,预测lstm rnn中的情绪

Keras 重塑输入标记化文本时出错,预测lstm rnn中的情绪,keras,lstm,sentiment-analysis,rnn,text-analysis,Keras,Lstm,Sentiment Analysis,Rnn,Text Analysis,我是神经网络新手,一直在学习它在文本分析领域的应用,所以我在python中使用了lstm rnn 在维度为20000*1的数据集上对模型进行训练后(2000是文本,1是文本的情感),我获得了99%的准确率,之后我验证了运行良好的模型(使用model.predict()函数) 现在,为了测试我的模型,我一直在尝试从数据帧或包含一些文本的变量中随机输入文本,但我总是遇到重塑数组的错误,其中要求rnn模型的输入为维度(1,30) 但是,当我将训练数据重新输入到模型中进行预测时,模型工作得非常好,为什么

我是神经网络新手,一直在学习它在文本分析领域的应用,所以我在python中使用了lstm rnn

在维度为20000*1的数据集上对模型进行训练后(2000是文本,1是文本的情感),我获得了99%的准确率,之后我验证了运行良好的模型(使用model.predict()函数)

现在,为了测试我的模型,我一直在尝试从数据帧或包含一些文本的变量中随机输入文本,但我总是遇到重塑数组的错误,其中要求rnn模型的输入为维度(1,30)

但是,当我将训练数据重新输入到模型中进行预测时,模型工作得非常好,为什么会发生这种情况

我只是停留在这里,任何类型的建议都将帮助我了解更多关于rnn的信息,我将随此请求附上错误和rnn模型代码

多谢各位

问候

图沙乌帕迪亚

    import numpy as np 
    import pandas as pd 
    import keras
    import sklearn

    from sklearn.feature_extraction.text import CountVectorizer
    from keras.preprocessing.text import Tokenizer
    from keras.preprocessing.sequence import pad_sequences
    from keras.models import Sequential
    from keras.layers import Dense, Embedding, LSTM
     from sklearn.model_selection import train_test_split
    from keras.utils.np_utils import to_categorical
    import re


    data=pd.read_csv('..../twitter_tushar_data.csv')
    max_fatures = 4000
    tokenizer = Tokenizer(num_words=max_fatures, split=' ')
    tokenizer.fit_on_texts(data['tweetText'].values)
    X = tokenizer.texts_to_sequences(data['tweetText'].values)
    X = pad_sequences(X)


    embed_dim = 128
    lstm_out = 196
    model = Sequential()
    keras.layers.core.SpatialDropout1D(0.2) #used to avoid overfitting
    model.add(Embedding(max_fatures, embed_dim,input_length = X.shape[1]))
    model.add(LSTM(196, recurrent_dropout=0.2, dropout=0.2))
   model.add(Dense(2,activation='softmax'))
   model.compile(loss = 'categorical_crossentropy', optimizer='adam',metrics 
   = ['accuracy'])
   print(model.summary())
   #splitting data in training and testing parts

   Y = pd.get_dummies(data['SA']).values
   X_train, X_test, Y_train, Y_test = train_test_split(X,Y, test_size = 
   0.30, random_state = 42)
   print(X_train.shape,Y_train.shape)
   print(X_test.shape,Y_test.shape)
   batch_size = 128
   model.fit(X_train, Y_train, epochs = 7, batch_size=batch_size, verbose = 
   2)


   validation_size = 3500

   X_validate = X_test[-validation_size:]
   Y_validate = Y_test[-validation_size:]
   X_test = X_test[:-validation_size]
   Y_test = Y_test[:-validation_size]
   score,acc = model.evaluate(X_test, Y_test, verbose = 2, batch_size = 128)
   print("score: %.2f" % (score))
   print("acc: %.2f" % (acc))


   pos_cnt, neg_cnt, pos_correct, neg_correct = 0, 0, 0, 0
   for x in range(len(X_validate)):
   result = 
 model.predict(X_validate[x].reshape(1,X_test.shape[1]),batch_size=1,verbose 
 = 2)[0]
 if np.argmax(result) == np.argmax(Y_validate[x]):
    if np.argmax(Y_validate[x]) == 0:
        neg_correct += 1
    else:
        pos_correct += 1

if np.argmax(Y_validate[x]) == 0:
    neg_cnt += 1
else:
    pos_cnt += 1
print("pos_acc", pos_correct/pos_cnt*100, "%")
print("neg_acc", neg_correct/neg_cnt*100, "%")

我得到了我的问题的解决方案,这只是正确标记输入的问题,谢谢!!以下代码用于预测不同的用户输入

 text=np.array(['you are a pathetic awful movie'])
 print(text.shape)
 tk=Tokenizer(num_words=4000,lower=True,split=" ")
 tk.fit_on_texts(text)

prediction=model.predict(sequence.pad_sequences(tk.texts_to_sequences(text),
maxlen=max_review_length))
print(prediction)
print(np.argmax(prediction))