Encoding 运行文本摘要代码时出现编码问题
你好 我正在测试网站上发布的文本摘要代码的功能: 问题是,当我调用文本文件上的函数时,“cp949”编解码器无法解码位置205处的字节0xe2:出现非法多字节序列错误。我从其他帖子中知道,这是一个与文件编码类型相关的错误。因此,我将test2.txt文件的编码类型更改为UTF-8(以纯文本格式保存该文件,然后选择文本编码上的UTF-8>其他编码),但仍然收到此错误消息 以下是我编写的代码:Encoding 运行文本摘要代码时出现编码问题,encoding,sequence,codec,multibyte,summarization,Encoding,Sequence,Codec,Multibyte,Summarization,你好 我正在测试网站上发布的文本摘要代码的功能: 问题是,当我调用文本文件上的函数时,“cp949”编解码器无法解码位置205处的字节0xe2:出现非法多字节序列错误。我从其他帖子中知道,这是一个与文件编码类型相关的错误。因此,我将test2.txt文件的编码类型更改为UTF-8(以纯文本格式保存该文件,然后选择文本编码上的UTF-8>其他编码),但仍然收到此错误消息 以下是我编写的代码: Import libraries from nltk.corpus import stopwords fr
Import libraries
from nltk.corpus import stopwords
from nltk.cluster.util import cosine_distance
import numpy as np
import networkx as nx
test_text_word = "test2.txt"
def read_article(test_text_word):
file = open(test_text_word, "r")
filedata = file.readlines()
article = filedata[0].split(". ")
sentences = []`
for sentence in article:
print(sentence)
sentences.append(sentence.replace("[^a-zA-Z]", " ").split(" "))
sentences.pop()
return sentences
def sentence_similarity(sent1, sent2, stopwords=None):
if stopwords is None:
stopwords = []
sent1 = [w.lower() for w in sent1]
sent2 = [w.lower() for w in sent2]
all_words = list(set(sent1 + sent2))
vector1 = [0] * len(all_words)
vector2 = [0] * len(all_words)
# build the vector for the first sentence
for w in sent1:
if w in stopwords:
continue
vector1[all_words.index(w)] += 1
# build the vector for the second sentence
for w in sent2:
if w in stopwords:
continue
vector2[all_words.index(w)] += 1
return 1 - cosine_distance(vector1, vector2)
def build_similarity_matrix(sentences, stop_words):
# Create an empty similarity matrix
similarity_matrix = np.zeros((len(sentences), len(sentences)))
for idx1 in range(len(sentences)):
for idx2 in range(len(sentences)):
if idx1 == idx2: #ignore if both are same sentences
continue
similarity_matrix[idx1][idx2] = sentence_similarity(sentences[idx1], sentences[idx2], stop_words)
return similarity_matrix
def generate_summary(test_text_word, top_n=5):
stop_words = stopwords.words('english')
summarize_text = []
# Step 1 - Read text anc split it
sentences = read_article(test_text_word)
# Step 2 - Generate Similary Martix across sentences
sentence_similarity_martix = build_similarity_matrix(sentences, stop_words)
# Step 3 - Rank sentences in similarity martix
sentence_similarity_graph = nx.from_numpy_array(sentence_similarity_martix)
scores = nx.pagerank(sentence_similarity_graph)
# Step 4 - Sort the rank and pick top sentences
ranked_sentence = sorted(((scores[i],s) for i,s in enumerate(sentences)), reverse=True)
print("Indexes of top ranked_sentence order are ", ranked_sentence)
for i in range(top_n):
summarize_text.append(" ".join(ranked_sentence[i][1]))
# Step 5 - Offcourse, output the summarize texr
print("Summarize Text: \n", ". ".join(summarize_text))
问题是,当我运行代码时,使用以下命令:
generate_summary("test2.txt", 2)
我收到此错误消息:“cp949”编解码器无法解码位置205中的字节0xe2:非法多字节序列
我应该更改代码中的某些内容吗?
谢谢你的支持