Clojure错误-超出GC开销限制
我试图随机抽取一个大的FASTQ文件,并将其写入standard out。我不断收到“超出GC开销限制”错误,我不确定我做错了什么。我曾尝试在莱宁根增加Xmx,但没有效果。这是我的密码:Clojure错误-超出GC开销限制,clojure,garbage-collection,large-files,fasta,Clojure,Garbage Collection,Large Files,Fasta,我试图随机抽取一个大的FASTQ文件,并将其写入standard out。我不断收到“超出GC开销限制”错误,我不确定我做错了什么。我曾尝试在莱宁根增加Xmx,但没有效果。这是我的密码: (ns fastq-sample.core (:gen-class) (:use clojure.java.io)) (def n-read-pair-lines 8) (defn sample? [sample-rate] (> sample-rate (rand))) ; ; Age
(ns fastq-sample.core
(:gen-class)
(:use clojure.java.io))
(def n-read-pair-lines 8)
(defn sample? [sample-rate]
(> sample-rate (rand)))
;
; Agent for writing the reads asynchronously
;
(def wtr (agent (writer *out*)))
(defn write-out [r]
(letfn [(write [out msg] (.write out msg) out)]
(send wtr write r)))
(defn write-close []
(send wtr #(.close %))
(await wtr))
;
; Main
;
(defn reads [file]
(->>
(input-stream file)
(java.util.zip.GZIPInputStream.)
(reader)
(line-seq)))
(defn -main [fastq-file sample-rate-str]
(let [sample-rate (Float. sample-rate-str)
in-reads (partition n-read-pair-lines (reads fastq-file))]
(doseq [x (filter (fn [_] (sample? sample-rate)) in-reads)]
(write-out (clojure.string/join "\n" x)))
(write-close)
(shutdown-agents)))
当我试图将一个无限序列合并成一个简单的数据结构(如地图或向量)时,我经常会遇到同样的症状。这通常意味着内存紧张,垃圾收集器无法满足对新对象的需求。wtr代理很可能太大,无法存储内存。也许您不希望通过更改来将打印结果存储在atom中
(write [out msg] (.write out msg) out)
到
我认为这只会在第一次将输出作为代理的状态写入时起作用,并将输出更改为write to out函数的结果。我是基于此编写代码的-
(write [out msg] (.write out msg))