R中的加速环问题

R中的加速环问题,r,performance,R,Performance,我有一个特别大的数据集,它由370万行和76个字符串列组成 我想比较一下上面的行和下面的行,看它们是否匹配,是否已经编写了这段代码。应注明上下行相同图案的数量 a <- c("a","a","a","a","a","a","a","a","a") b <- c("b","b","b","b","a","b","b","b","b") c <- c("c","c","c","c","a","a","a","b","b") d <- c("d","d"

我有一个特别大的数据集,它由370万行和76个字符串列组成

我想比较一下上面的行和下面的行,看它们是否匹配,是否已经编写了这段代码。应注明上下行相同图案的数量

   a <- c("a","a","a","a","a","a","a","a","a")
   b <- c("b","b","b","b","a","b","b","b","b")
   c <- c("c","c","c","c","a","a","a","b","b")
   d <- c("d","d","d","d","d","d","d","d","d")
   features_split   <- data.frame(a,b,c,d); features_split
   ncol = max(sapply(features_split,length))
   safe <- as.data.table(lapply(1:ncol,function(i)sapply(features_split,"[",i)))
   nrow(safe)
   df <- safe
   LIST  <-list() 
   LIST2 <-list() 
   for(i in 1:(nrow(df)-1)) 
   { 
   LIST[[i]] <-df[i+1,] %in% df[i,] 
   LIST2[[i]] <- length(LIST[[i]][LIST[[i]]==TRUE]) 
   } 
   safe2   <- unlist(LIST2)
   not_available <- rowSums(!is.na(safe))
a使用数据帧
使用
数据框进行概念验证

set.seed(4)
nr <- 1000
mydf <- data.frame(a=sample(letters[1:3], nr, repl=TRUE),
                   b=sample(letters[1:3], nr, repl=TRUE),
                   c=sample(letters[1:3], nr, repl=TRUE),
                   d=sample(letters[1:3], nr, repl=TRUE),
                   stringsAsFactors=FALSE)
matches <- vapply(seq.int(nrow(mydf)-1),
                  function(ii,zz) sum(mydf[ii,] == mydf[ii+1,]),
                  integer(1))
head(matches)
## [1] 0 3 4 2 1 0
sum(matches == 4) # total number of perfect row-matches
## 16
改用矩阵 如果您可以将其作为矩阵(因为您有一个“character”的同质数据类型)而不是data.frame,那么您将获得更好的性能:

nr <- 100000
nc <- 76
mymtx2 <- matrix(sample(letters[1:4], nr*nc, repl=TRUE), nc=nc)
dim(mymtx2)
## [1] 10000    76

system.time(
    matches2 <- vapply(seq.int(nrow(mymtx2)-1),
                       function(ii) sum(mymtx2[ii,] == mymtx2[ii+1,]),
                       integer(1))
    )
##     user  system elapsed 
##    0.81    0.00    0.81 
不幸的是,仍然没有比赛,但我认为37万人比赛36秒比10万人比赛1小时要好得多。(如果我的假设不正确,请纠正我。)


(参考:win7 x64,R-3.0.3-64位,英特尔i7-2640M 2.8GHz,8GB RAM)

首先:是
a、b、c、d
列还是行?如果列
data.frame(a,b,c,d)
可以,如果行则使用
data.frame(rbind(a,b,c,d))
。请清理你的帖子:)是的,请试着从输入和期望的输出两个方面说明你的例子……你可能想做一些更像
all.equal(df[i],df[i+1])
,但你需要提供可复制的输出和输入,您可能会假设此代码的输出是所需的输出…您是否有多个可用的内核?并行性在这里可能有所帮助。顺便说一句:最大匹配数为40(共76个)。毫不奇怪,
matches3
的直方图显示非常正态分布,平均值为19,标准偏差为3.78。如果这回答了您的问题,请将其标记为答案。否则,请随意详述我遗漏的任何部分。
nr <- 100000
nc <- 76
mymtx2 <- matrix(sample(letters[1:4], nr*nc, repl=TRUE), nc=nc)
dim(mymtx2)
## [1] 10000    76

system.time(
    matches2 <- vapply(seq.int(nrow(mymtx2)-1),
                       function(ii) sum(mymtx2[ii,] == mymtx2[ii+1,]),
                       integer(1))
    )
##     user  system elapsed 
##    0.81    0.00    0.81 
nr <- 3.7e6
nc <- 76
mymtx3 <- matrix(sample(letters[1:4], nr*nc, repl=TRUE), nc=nc)
dim(mymtx3)
## [1] 3700000      76
system.time(
    matches3 <- vapply(seq.int(nrow(mymtx3)-1),
                       function(ii) sum(mymtx3[ii,] == mymtx3[ii+1,]),
                       integer(1))
    )
##     user  system elapsed 
##   35.32    0.05   35.81 

length(matches3)
## [1] 3699999
sum(matches3 == nc)
## [1] 0