R 使用字典在Quanteda中创建二元图

我试图从我的数据文本分析中删除打字错误。所以我使用quanteda包的字典功能。它适用于Unigram。但它为bigram提供了意想不到的输出。我不知道如何处理打字错误，这样他们就不会潜入我的大字和三字了

ZTestCorp1 <- c("The new law included a capital gains tax, and an inheritance tax.", 
                "New York City has raised a taxes: an income tax and a sales tax.")

ZcObj <- corpus(ZTestCorp1)

mydict <- dictionary(list("the"="the", "new"="new", "law"="law", 
                      "capital"="capital", "gains"="gains", "tax"="tax", 
                      "inheritance"="inheritance", "city"="city")) 

Zdfm1 <- dfm(ZcObj, ngrams=2, concatenator=" ", 
         what = "fastestword", 
         toLower=TRUE, removeNumbers=TRUE,
         removePunct=TRUE, removeSeparators=TRUE,
         removeTwitter=TRUE, stem=FALSE,
         ignoredFeatures=NULL,
         language="english", 
         dictionary=mydict, valuetype="fixed")

wordsFreq1 <- colSums(sort(Zdfm1))

在不使用字典的情况下，输出如下：

> wordsFreq
    tax and         the new         new law    law included      included a       a capital 
          2               1               1               1               1               1 
capital gains       gains tax          and an  an inheritance inheritance tax        new york 
          1               1               1               1               1               1 
  york city        city has      has raised        raised a         a taxes        taxes an 
          1               1               1               1               1               1 
  an income      income tax           and a         a sales       sales tax 
          1               1               1               1               1

预期的二元图

The new
new law
law capital
capital gains
gains tax
tax inheritance
inheritance city  

p、 我假设标记化是在字典匹配之后完成的。但从我看到的结果来看，情况并非如此

另一方面，我尝试将dictionary对象创建为

mydict <- dictionary(list(mydict=c("the", "new", "law", "capital", "gains", 
                      "tax", "inheritance", "city"))) 

---

mydict更新了2017年12月21日的quanteda新版本

很高兴看到您正在使用该软件包！我认为你正在努力解决两个问题。首先是如何在形成ngrams之前应用特征选择。第二个是如何定义特征选择（使用quanteda）

第一期：如何在形成ngrams之前应用特征选择。在这里，您定义了一个字典来实现这一点。（正如我将在下面展示的，这里没有必要这样做。）您希望删除选择列表中未包含的所有术语，然后形成bigrams。quanteda在默认情况下不会这样做，因为它不是“bigram”的标准形式，即单词不会根据严格由邻接定义的某个窗口进行并置。例如，在您预期的结果中，
法律大写字母
不是一对相邻的术语，这是bigram的常用定义

但是，我们可以通过更“手动”地构建文档特征矩阵来覆盖这种行为

首先，标记文本

# tokenize the original
toks <- tokens(ZcObj, removePunct = TRUE, removeNumbers = TRUE) %>%
  tokens_tolower()
toks
## tokens object from 2 documents.
## text1 :
##  [1] "the"         "new"         "law"         "included"    "a"           "capital"     "gains"       "tax"         "and"         "an"          "inheritance" "tax"        
## 
## text2 :
##  [1] "new"    "york"   "city"   "has"    "raised" "a"      "taxes"  "an"     "income" "tax"    "and"    "a"      "sales"  "tax"  

从这组选定的令牌中，我们现在可以形成BIGRAM（或者我们可以将
toksDict
直接馈送到
dfm（）
）：

（toks2感谢您慷慨而详细的解释。我发现了这个错误。有什么想法吗？`>（toksDict可能是因为
selectFeatures（）
的方法仅在最新版本中进行了扩展（GitHub）quanteda版本，并且您正在使用CRAN版本。请尝试从GitHub按照进行安装，截至今天的版本为0.9.1-7。（将于2016年1月更新CRAN版本。）谢谢@Ken。这太棒了！我将安装最新的cran软件包。事实上，我喜欢你提供的第二个解决方案，因为它考虑了停止词。这对我来说很重要，因为我正在进行一个单词预测项目。然而，我很好奇它是如何在约克成功的。我认为约克不是一个停止词。我在使用ngram时得到了这个s=2选项。谢谢！不知道你说的“约克”是什么意思。它不是dfm中的标记，因为您选择仅保留字典值中的单词。很高兴继续在私人消息或通过GitHub问题提供帮助。我设法使用Stafford dictionary创建了我的Unigram。但是，当我尝试创建Bigram时，它将花费永远的时间。我在20小时前启动它，它仍然运行ng.索引在不到10分钟的时间内完成。它显示674468个文档和5168973个功能类型。我正在使用Windows 10 12 GB ram的Dell inspiron 15笔记本电脑上运行。
> (myDfm1a <- dfm(ZcObj, verbose = FALSE, ngrams=2, 
+                keptFeatures = c("the", "new", "law", "capital", "gains",  "tax", "inheritance", "city")))
Document-feature matrix of: 2 documents, 14 features.
2 x 14 sparse Matrix of class "dfmSparse" features
docs    the_new new_law law_included a_capital capital_gains gains_tax   tax_and an_inheritance
text1       1       1            1         1             1         1       1               1
text2       0       0            0         0             0         0       1              0
   features
docs    inheritance_tax new_york york_city city_has income_tax sales_tax
text1               1        0         0        0          0         0
text2               0        1         1        1          1         1

# tokenize the original
toks <- tokens(ZcObj, removePunct = TRUE, removeNumbers = TRUE) %>%
  tokens_tolower()
toks
## tokens object from 2 documents.
## text1 :
##  [1] "the"         "new"         "law"         "included"    "a"           "capital"     "gains"       "tax"         "and"         "an"          "inheritance" "tax"        
## 
## text2 :
##  [1] "new"    "york"   "city"   "has"    "raised" "a"      "taxes"  "an"     "income" "tax"    "and"    "a"      "sales"  "tax"  

(toksDict <- tokens_select(toks, mydict, selection = "keep"))
## tokens object from 2 documents.
## text1 :
##  [1] "the"         "new"         "law"         "capital"     "gains"       "tax"         "inheritance" "tax"        
## 
## text2 :
##  [1] "new"  "city" "tax"  "tax" 

(toks2 <- tokens_ngrams(toksDict, n = 2, concatenator = " "))
## tokens object from 2 documents.
## text1 :
##  [1] "the new"         "new law"         "law capital"     "capital gains"   "gains tax"       "tax inheritance" "inheritance tax"
## 
## text2 :
##  [1] "new city" "city tax" "tax tax" 

# now create the dfm
(myDfm2 <- dfm(toks2))
## Document-feature matrix of: 2 documents, 10 features.
## 2 x 10 sparse Matrix of class "dfm"
##        features
## docs    the new new law law capital capital gains gains tax tax inheritance inheritance tax new city city tax tax tax
##   text1       1       1           1             1         1               1               1        0        0       0
##   text2       0       0           0             0         0               0               0        1        1       1
topfeatures(myDfm2)
#     the new         new law     law capital   capital gains       gains tax tax inheritance inheritance tax        new city        city tax         tax tax 
#           1               1               1               1               1               1               1               1               1               1 

(myDfm1 <- dfm(ZcObj, verbose = FALSE, 
               keptFeatures = c("the", "new", "law", "capital", "gains", "tax", "inheritance", "city")))
## Document-feature matrix of: 2 documents, 8 features.
## 2 x 8 sparse Matrix of class "dfm"
##        features
## docs    the new law capital gains tax inheritance city
##   text1   1   1   1       1     1   2           1    0
##   text2   0   1   0       0     0   2           0    1