以R或SQL为间隔的每个组的最大n引用
我已经在下面描述了我的（非琐碎的）问题。这是我的第一篇文章，现在是修改版。任何投入或提出的解决方案都会有所帮助 这有几个方面：确定小规模问题的最优解决方案（下面已经有一些建议），时间（下面的数据表解决方案似乎选中了该框）和内存管理。问题是关于在一个表中枚举并由另一个表中的集群表示的标记（如果在同一条链上30 bp以内，则为同一集群） boils面临的挑战是确定将给定标记分配到适当间隔的有效过程。我们正在处理基因组数据，这意味着标签坐标由起始位置、结束位置（=起始位置+1）、染色体（完整数据集中的25个值）和链（位置在双链DNA的正链或负链上）决定。因此，簇在同一条线上是不重叠的，但如果它们的间隔在不同的线上，簇坐标可能会重叠，这会使事情变得复杂 这是我1月9日发布的文章的一个修改版本，它更好地概括了问题的内在困难。稍后将显示解决小规模问题的快速解决方案。 如果有人想处理完整的数据集，请告诉我。 非常感谢 问候, 尼克·克拉克 背景 该问题涉及间隔和每组最大n。我有两个包含聚集基因坐标（集群）和输入数据（标签）的表。clusters表包含来自tags表中相同链上每个覆盖的非重叠间隔的总计标记。完整集群表有160万行。标签表大约有4百万行，因此理想情况下解决方案应该是矢量化的。请参见下面的一些示例数据。该设置是关于人类转录起始位点（CAGE）的数据集 当我在R中工作时，我正在寻找基于R或SQL的解决方案。我以前曾通过R中的plyr和sqldf包进行过失败的尝试 挑战 我缺少的是集群表中的一行，该行从与最大标记贡献相关联的输入数据表中标识起始坐标 注意 1） 来自不同股的簇可以有重叠的坐标， 2） chr/chr_clst可以采用25个不同的值（示例中未显示）， 3） 解决方案需要同时考虑strand和chr/chr\u clst 我的理想解决方案： 矢量化的R代码或下面SQL语句的改进。下面解决方案的一个版本，可以解决内存问题。改进的sql语句可以有效地从clusters表中确定适当的行 目前的状态 这显然是迄今为止最好的解决方案帽子提示和酷点指向user1935457获取代码，指向mnel获取后续建议修改。这里的障碍是，由于对内存的过度需求，从玩具示例移动到填充比例表会导致R（和R Studio）崩溃以R或SQL为间隔的每个组的最大n引用,sql,r,memory-management,greatest-n-per-group,data.table,Sql,R,Memory Management,Greatest N Per Group,Data.table,我已经在下面描述了我的（非琐碎的）问题。这是我的第一篇文章，现在是修改版。任何投入或提出的解决方案都会有所帮助 这有几个方面：确定小规模问题的最优解决方案（下面已经有一些建议），时间（下面的数据表解决方案似乎选中了该框）和内存管理。问题是关于在一个表中枚举并由另一个表中的集群表示的标记（如果在同一条链上30 bp以内，则为同一集群） boils面临的挑战是确定将给定标记分配到适当间隔的有效过程。我们正在处理基因组数据，这意味着标签坐标由起始位置、结束位置（=起始位置+1）、染色体（完整数据集中的
# Convert sample data provided in question
clusters <- as.data.table(clusters)
tags <- as.data.table(tags)
# Rename chr and strand for easier joining
setnames(clusters, c("chr_clst", "strand_clst"), c("chr", "strand"))
# Set key on each table for next step
setkey(clusters, chr, strand)
setkey(tags, chr, strand)
# Merge on the keys
tmp <- merge(clusters, tags, by = c("chr", "strand"))
# Find index (in merged table, tmp) of largest tag_count in each
# group subject to start_clst <= end <= end_clst
idx <- tmp[between(end, start_clst, end_clst),
list(IDX=.I[which.max(tag_count)]),
by=list(chr, start_clst,end_clst,strand)]$IDX
# Get those rows from merged table
tmp[idx]
#转换有问题的样本数据
集群这里有一个建议使用应用：
transform(
clusters,
start = apply(clusters[c("chr_clst", "start_clst", "end_clst", "strand_clst")],
1, function(x) {
tmp <- tags[tags$start >= as.numeric(x[2]) &
tags$end <= as.numeric(x[3]) &
tags$chr == x[1] &
tags$strand == x[4], c("tag_count", "start")]
tmp$start[which.max(tmp$tag_count)]}))
下面是对数据的尝试。表包：
# Convert sample data provided in question
clusters <- as.data.table(clusters)
tags <- as.data.table(tags)
# Rename chr and strand for easier joining
setnames(clusters, c("chr_clst", "strand_clst"), c("chr", "strand"))
# Set key on each table for next step
setkey(clusters, chr, strand)
setkey(tags, chr, strand)
# Merge on the keys
tmp <- merge(clusters, tags, by = c("chr", "strand"))
# Find index (in merged table, tmp) of largest tag_count in each
# group subject to start_clst <= end <= end_clst
idx <- tmp[between(end, start_clst, end_clst),
list(IDX=.I[which.max(tag_count)]),
by=list(chr, start_clst,end_clst,strand)]$IDX
# Get those rows from merged table
tmp[idx]
编辑
基于下面评论中讨论的内存问题，下面是另一个尝试。我使用interval
包查找两个表之间的重叠间隔。您还可以探索并行化for
循环以提高速度
require(data.table)
require(intervals)
clusters <- data.table(clusters)
tags <- data.table(tags)
# Find all unique combinations of chr and strand...
setkey(clusters, chr_clst, strand_clst)
setkey(tags, chr, strand)
unique.keys <- unique(rbind(clusters[, key(clusters), with=FALSE],
tags[, key(tags), with=FALSE], use.names=FALSE))
# ... and then work on each pair individually to avoid creating
# enormous objects in memory
result.list <- vector("list", nrow(unique.keys))
for(i in seq_len(nrow(unique.keys))) {
tmp.clst <- clusters[unique.keys[i]]
tmp.tags <- tags[unique.keys[i]]
# Keep track of each row for later
tmp.clst[, row.id := seq_len(nrow(tmp.clst))]
tmp.tags[, row.id := seq_len(nrow(tmp.tags))]
# Use intervals package to find all overlapping [start, end]
# intervals between the two tables
clst.intervals <- Intervals(tmp.clst[, list(start_clst, end_clst)],
type = "Z")
tags.intervals <- Intervals(tmp.tags[, list(start, end)],
type = "Z")
rownames(tags.intervals) <- tmp.tags$row.id
# This goes to C++ code in intervals package;
# I didn't spend too much time looking over how it works
overlaps <- interval_overlap(tags.intervals,
clst.intervals,
check_valid = FALSE)
# Retrieve rows from clusters table with overlaps and add a column
# indicating which intervals in tags table they overlapped with
matches <- lapply(as.integer(names(overlaps)), function(n) {
ans <- tmp.clst[overlaps[[n]]]
ans[, match.in.tags := n]
})
# List back to one table...
matches <- rbindlist(matches)
# ... and join each match from tags to its relevant row from tags
setkey(matches, match.in.tags)
setkey(tmp.tags, row.id)
# add the rows for max of tag_count by start_clst and
# end_clst from this particular unique key to master list...
result.list[[i]] <- tmp.tags[matches][, .SD[which.max(tag_count)],
by = list(start_clst, end_clst)]
}
# and concatenate master list into none table,
# getting rid of the helper columns
rbindlist(result.list)[, c("row.id", "row.id.1") := NULL][]
在其他答案和评论的基础上，给出一些提示
如果X[Y]
（或merge（X，Y）
）返回大量的行，大于max（nrow（X），nrow（Y））
（例如nrow（X）*nrow（Y）
），那么X[Y][where]
（即X[Y]
后面跟一个子集）将不会有任何帮助。最终的结果要小得多，但它必须首先创建大的X[Y]
如果需要范围，则一种方法是w=X[Y，roll=TRUE，which=TRUE]
或w=X[Y，mult=“first”，which=TRUE]
或类似的方法，第一次和最后一次可能两次。获得每个范围的行位置（w
）后，可以在开始和结束之间选择seq
或vecseq
，然后选择结果。本标签中的其他S.O.问题中有一些示例。当然，将其构建到data.table中会很好，并且有一个特性请求建议联接列本身可以是2列列表列，包含每行每列范围查询的边界
或者，by-without-by可以使用。这就是当没有by
子句时，为i
的每一行计算j
。搜索？data.table
，查找by而不是by，并查看示例。这就是为什么你可以坚持笛卡尔式然后是子集式的思维，而不需要先创建整个笛卡尔式的结果。类似于：X[Y，.SD[start这可以通过使用data.table中的foverlaps（）
函数非常有效地完成。table
从v1.9.4
开始提供：
require(data.table) #v1.9.4+
setDT(clusters, key=c("chr_clst", "strand_clst", "start_clst", "end_clst"))
setDT(tags, key=c("chr", "strand", "start", "end"))
ans = foverlaps(clusters, tags)[, .SD[which.max(tag_count)], by=.(chr_clst, strand_clst, start_clst, end_clst)]
# chr_clst strand_clst start_clst end_clst start end tag_count tags_clst
# 1: chr1 - 569926 569952 569942 569943 4 25
# 2: chr1 - 713987 714049 714011 714012 4 46
# 3: chr1 + 568911 568941 568940 568941 8 37
# 4: chr1 + 569233 569256 569255 569256 2 4
# 5: chr1 + 569454 569484 569471 569472 2 6
# 6: chr1 + 569793 569803 569793 569794 2 3
# 7: chr1 + 569877 569926 569925 569926 5 80
# 8: chr1 + 569972 569973 569972 569973 1 1
# 9: chr1 + 570048 570095 570048 570049 1 4
# 10: chr1 + 570166 570167 570166 570167 1 1
foverlaps（）参数。请同时发布预期结果，最好是您尝试过的代码。请注意，sqldf默认为SQLite，而不是MySQL，尽管它也支持MySQL，并且如果您加载了RMySQL，它将自动支持MySQL。它还支持PostgreSQL，PostgreSQL具有分区依据
。请使用sqldf
drv=
argument或sqldf.driver
选项，或者在sqldf之前加载RPostgreSQL，它会注意到它。
chr_clst start_clst end_clst strand_clst tags_clst start
1 chr1 568911 568941 + 37 568940
2 chr1 569233 569256 + 4 569255
3 chr1 569454 569484 + 6 569471
4 chr1 569793 569803 + 3 569793
5 chr1 569877 569926 + 80 569925
6 chr1 569926 569952 - 25 569942
7 chr1 569972 569973 + 1 569972
8 chr1 570048 570095 + 4 570048
9 chr1 570166 570167 + 1 570166
10 chr1 713987 714049 - 46 714011
# Convert sample data provided in question
clusters <- as.data.table(clusters)
tags <- as.data.table(tags)
# Rename chr and strand for easier joining
setnames(clusters, c("chr_clst", "strand_clst"), c("chr", "strand"))
# Set key on each table for next step
setkey(clusters, chr, strand)
setkey(tags, chr, strand)
# Merge on the keys
tmp <- merge(clusters, tags, by = c("chr", "strand"))
# Find index (in merged table, tmp) of largest tag_count in each
# group subject to start_clst <= end <= end_clst
idx <- tmp[between(end, start_clst, end_clst),
list(IDX=.I[which.max(tag_count)]),
by=list(chr, start_clst,end_clst,strand)]$IDX
# Get those rows from merged table
tmp[idx]
chr strand start_clst end_clst tags_clst start end tag_count
1: chr1 - 569926 569952 25 569942 569943 4
2: chr1 - 713987 714049 46 714011 714012 4
3: chr1 + 568911 568941 37 568940 568941 8
4: chr1 + 569233 569256 4 569255 569256 2
5: chr1 + 569454 569484 6 569471 569472 2
6: chr1 + 569793 569803 3 569793 569794 2
7: chr1 + 569877 569926 80 569925 569926 5
8: chr1 + 569972 569973 1 569972 569973 1
9: chr1 + 570048 570095 4 570048 570049 1
10: chr1 + 570166 570167 1 570166 570167 1
require(data.table)
require(intervals)
clusters <- data.table(clusters)
tags <- data.table(tags)
# Find all unique combinations of chr and strand...
setkey(clusters, chr_clst, strand_clst)
setkey(tags, chr, strand)
unique.keys <- unique(rbind(clusters[, key(clusters), with=FALSE],
tags[, key(tags), with=FALSE], use.names=FALSE))
# ... and then work on each pair individually to avoid creating
# enormous objects in memory
result.list <- vector("list", nrow(unique.keys))
for(i in seq_len(nrow(unique.keys))) {
tmp.clst <- clusters[unique.keys[i]]
tmp.tags <- tags[unique.keys[i]]
# Keep track of each row for later
tmp.clst[, row.id := seq_len(nrow(tmp.clst))]
tmp.tags[, row.id := seq_len(nrow(tmp.tags))]
# Use intervals package to find all overlapping [start, end]
# intervals between the two tables
clst.intervals <- Intervals(tmp.clst[, list(start_clst, end_clst)],
type = "Z")
tags.intervals <- Intervals(tmp.tags[, list(start, end)],
type = "Z")
rownames(tags.intervals) <- tmp.tags$row.id
# This goes to C++ code in intervals package;
# I didn't spend too much time looking over how it works
overlaps <- interval_overlap(tags.intervals,
clst.intervals,
check_valid = FALSE)
# Retrieve rows from clusters table with overlaps and add a column
# indicating which intervals in tags table they overlapped with
matches <- lapply(as.integer(names(overlaps)), function(n) {
ans <- tmp.clst[overlaps[[n]]]
ans[, match.in.tags := n]
})
# List back to one table...
matches <- rbindlist(matches)
# ... and join each match from tags to its relevant row from tags
setkey(matches, match.in.tags)
setkey(tmp.tags, row.id)
# add the rows for max of tag_count by start_clst and
# end_clst from this particular unique key to master list...
result.list[[i]] <- tmp.tags[matches][, .SD[which.max(tag_count)],
by = list(start_clst, end_clst)]
}
# and concatenate master list into none table,
# getting rid of the helper columns
rbindlist(result.list)[, c("row.id", "row.id.1") := NULL][]
start_clst end_clst chr strand start end tag_count chr_clst strand_clst tags_clst
1: 569926 569952 chr1 - 569942 569943 4 chr1 - 25
2: 713987 714049 chr1 - 714011 714012 4 chr1 - 46
3: 568911 568941 chr1 + 568940 568941 8 chr1 + 37
4: 569233 569256 chr1 + 569255 569256 2 chr1 + 4
5: 569454 569484 chr1 + 569471 569472 2 chr1 + 6
6: 569793 569803 chr1 + 569793 569794 2 chr1 + 3
7: 569877 569926 chr1 + 569925 569926 5 chr1 + 80
8: 569972 569973 chr1 + 569972 569973 1 chr1 + 1
9: 570048 570095 chr1 + 570048 570049 1 chr1 + 4
10: 570166 570167 chr1 + 570166 570167 1 chr1 + 1
setkey(clusters,chr,strand,end_clst)
setkey(tags,chr,strand,end)
begin = tags[clusters[,list(chr,strand,start_clst)],roll=-Inf,mult="first",which=TRUE]
end = tags[clusters[,list(chr,strand,end_clst)],roll=+Inf,mult="last",which=TRUE]
idx = mapply(function(x,y){.i=seq.int(x,y); .i[ which.max(tags$tag_count[.i]) ]}, begin, end)
cbind(clusters, tags[idx])
chr start_clst end_clst strand tags_clst chr start end strand tag_count
1: chr1 569926 569952 - 25 chr1 569942 569943 - 4
2: chr1 713987 714049 - 46 chr1 714011 714012 - 4
3: chr1 568911 568941 + 37 chr1 568940 568941 + 8
4: chr1 569233 569256 + 4 chr1 569255 569256 + 2
5: chr1 569454 569484 + 6 chr1 569471 569472 + 2
6: chr1 569793 569803 + 3 chr1 569793 569794 + 2
7: chr1 569877 569926 + 80 chr1 569925 569926 + 5
8: chr1 569972 569973 + 1 chr1 569972 569973 + 1
9: chr1 570048 570095 + 4 chr1 570048 570049 + 1
10: chr1 570166 570167 + 1 chr1 570166 570167 + 1
require(data.table) #v1.9.4+
setDT(clusters, key=c("chr_clst", "strand_clst", "start_clst", "end_clst"))
setDT(tags, key=c("chr", "strand", "start", "end"))
ans = foverlaps(clusters, tags)[, .SD[which.max(tag_count)], by=.(chr_clst, strand_clst, start_clst, end_clst)]
# chr_clst strand_clst start_clst end_clst start end tag_count tags_clst
# 1: chr1 - 569926 569952 569942 569943 4 25
# 2: chr1 - 713987 714049 714011 714012 4 46
# 3: chr1 + 568911 568941 568940 568941 8 37
# 4: chr1 + 569233 569256 569255 569256 2 4
# 5: chr1 + 569454 569484 569471 569472 2 6
# 6: chr1 + 569793 569803 569793 569794 2 3
# 7: chr1 + 569877 569926 569925 569926 5 80
# 8: chr1 + 569972 569973 569972 569973 1 1
# 9: chr1 + 570048 570095 570048 570049 1 4
# 10: chr1 + 570166 570167 570166 570167 1 1