使用Python创建语料库

我是Python新手， 我已经使用R创建了一个术语文档矩阵，我想学习如何使用Python创建相同的矩阵

我正在从数据框RESU Desc列中可用的描述列中读取文本数据。但我不确定如何使用python中创建文档术语矩阵的功能。如果有任何有助于学习的文档，这将非常有用

下面是我在R中使用的代码

docs <- Corpus(VectorSource(Res_Desc_Train$Description))
docs <-tm_map(docs,content_transformer(tolower))

#remove potentially problematic symbols
toSpace <- content_transformer(function(x, pattern) { return (gsub(pattern, " ", x))})
removeSpecialChars <- function(x) gsub("[^a-zA-Z0-9 ]","",x)
docs <- tm_map(docs, toSpace, "/")
docs <- tm_map(docs, toSpace, "-")
docs <- tm_map(docs, toSpace, ":")
docs <- tm_map(docs, toSpace, ";")
docs <- tm_map(docs, toSpace, "@")
docs <- tm_map(docs, toSpace, "\\(" )
docs <- tm_map(docs, toSpace, ")")
docs <- tm_map(docs, toSpace, ",")
docs <- tm_map(docs, toSpace, "_")
docs <- tm_map(docs, content_transformer(removeSpecialChars))
docs <- tm_map(docs, content_transformer(tolower))
docs <- tm_map(docs, removeWords, stopwords("en"))
docs <- tm_map(docs, removePunctuation)
docs <- tm_map(docs, stripWhitespace)
docs <- tm_map(docs, removeNumbers)

#inspect(docs[440])
dataframe<-data.frame(text=unlist(sapply(docs, `[`, "content")), stringsAsFactors=F)

BigramTokenizer <-
  function(x)
    unlist(lapply(ngrams(words(x), 2), paste, collapse = " "), use.names = FALSE)

dtm <- DocumentTermMatrix(docs,control=list(stopwords=FALSE,wordLengths =c(2,Inf),tokenize = BigramTokenizer))

Weighteddtm <- weightTfIdf(dtm,normalize=TRUE)
mat.df <- as.data.frame(data.matrix(Weighteddtm), stringsAsfactors = FALSE)
mat.df <- cbind(mat.df, Res_Desc_Train$Group)
colnames(mat.df)[ncol(mat.df)] <- "Group"
Assignment.Distribution <- table(mat.df$Group)

Res_Desc_Train_Assign <- mat.df$Group

Assignment.Distribution <- table(mat.df$Group)

### Feature has different ranges, normalizing to bring ranges from 0 to 1
### Another way to standardize using z-scores

normalize <- function(x) {
  y <- min(x)
  z <- max(x)
  temp <- x - y
  temp1 <- (z - y)
  temp2 <- temp / temp1
  return(temp2)
}
#normalize(c(1,2,3,4,5))

num_col <- ncol(mat.df)-1
mat.df_normalize <- as.data.frame(lapply(mat.df[,1:num_col], normalize))
mat.df_normalize <- cbind(mat.df_normalize, Res_Desc_Train_Assign)
colnames(mat.df_normalize)[ncol(mat.df_normalize)] <- "Group"

---

docs通常，当您需要在python中处理文本时，最好的工具是。在您的特定案例中，有一个特定的python包创建术语文档矩阵。这个包叫做

此外，如果需要使用正则表达式，可以使用python的
re
包。否则，您可以从NLTK直接使用标记器
 感谢您分享NLTK和Textmining的文档，我将浏览并尝试准备所需的文本文档矩阵