使用生成器(python)解析fasta文件
我试图解析一个大的fasta文件,但遇到内存不足的错误。如能提供一些改进数据处理的建议,将不胜感激。目前,程序正确地打印出名称,但部分通过文件我得到了一个MemoryError 这是发电机使用生成器(python)解析fasta文件,python,file,parsing,fasta,Python,File,Parsing,Fasta,我试图解析一个大的fasta文件,但遇到内存不足的错误。如能提供一些改进数据处理的建议,将不胜感激。目前,程序正确地打印出名称,但部分通过文件我得到了一个MemoryError 这是发电机 def readFastaEntry( fp ): name = "" seq = "" for line in fp: if line.startswith( ">" ): tmp = [] tmp.append(
def readFastaEntry( fp ):
name = ""
seq = ""
for line in fp:
if line.startswith( ">" ):
tmp = []
tmp.append( name )
tmp.append( seq )
name = line
seq = ""
yield tmp
else:
seq = seq.join( line )
这是调用方存根,这部分工作完成后将添加更多
fp = open( sys.argv[1], 'r' )
for seq in readFastaEntry( fp ) :
print seq[0]
对于那些与fasta格式不相似的人,这里有一个例子
>1 (PB2)
AATATATTCAATATGGAGAGAATAAAAGAACTAAGAGATCTAATGTCACAGTCTCGCACTCGCGAGATAC
TCACCAAAACCACTGTGGACCACATGGCCATAATCAAAAAGTACACATCAGGAAGGCAAGAGAAGAACCC
TGCACTCAGGATGAAGTGGATGATG
>2 (PB1)
AACCATTTGAATGGATGTCAATCCGACTTTACTTTTCTTGAAAGTTCCAGCGCAAAATGCCATAAGCACC
ACATTTCCCTATACTGGAGACCCTCC
每个条目都以“>”开头,说明名称等,接下来的N行是数据。除了下一行开头有一个“>”之外,没有定义数据的结尾。您是否考虑过使用。他们有一个可以读取fasta文件的系统。如果你有兴趣自己编写一个,你可以看看
编辑:添加代码
def read_fasta(fp):
name, seq = None, []
for line in fp:
line = line.rstrip()
if line.startswith(">"):
if name: yield (name, ''.join(seq))
name, seq = line, []
else:
seq.append(line)
if name: yield (name, ''.join(seq))
with open('f.fasta') as fp:
for name, seq in read_fasta(fp):
print(name, seq)
如果不了解您正在做什么,我会这样编写代码:
def readFastaEntry( fp ):
name = ""
while True:
line = name or f.readline()
if not line:
break
seq = []
while True:
name = f.readline()
if not name or name.startswith(">"):
break
else:
seq.append(name)
yield (line, "".join(seq))
这将收集从一个起始行到下一个起始行的数据。使
seq
成为一个数组意味着您将最小化字符串连接,直到最后一个可能的时刻。产生一个元组比产生一个列表更有意义。这种格式的pyparsing解析器只有几行长。请参见以下代码中的注释:
data = """>1 (PB2)
AATATATTCAATATGGAGAGAATAAAAGAACTAAGAGATCTAATGTCACAGTCTCGCACTCGCGAGATAC
TCACCAAAACCACTGTGGACCACATGGCCATAATCAAAAAGTACACATCAGGAAGGCAAGAGAAGAACCC
TGCACTCAGGATGAAGTGGATGATG
>2 (PB1)
AACCATTTGAATGGATGTCAATCCGACTTTACTTTTCTTGAAAGTTCCAGCGCAAAATGCCATAAGCACC
ACATTTCCCTATACTGGAGACCCTCC"""
from pyparsing import Word, nums, QuotedString, Combine, OneOrMore
# define some basic forms
integer = Word(nums)
key = QuotedString("(", endQuoteChar=")")
# sequences are "words" made up of the characters A, G, C, and T
# we want to match one or more of them, and have the parser combine
# them into a single string (Combine by default requires all of its
# elements to be adjacent within the input string, but we want to allow
# for the intervening end of lines, so we add adjacent=False)
sequence = Combine(OneOrMore(Word("AGCT")), adjacent=False)
# define the overall pattern to scan for - attach results names
# to each matched element
seqEntry = ">" + integer("index") + key("key") + sequence("sequence")
for seq,s,e in seqEntry.scanString(data):
# just dump out the matched data
print seq.dump()
# could also access fields as seq.index, seq.key and seq.sequence
印刷品:
['>', '1', 'PB2', 'AATATATTCAATATGGAGAGAATAAAAGAACTAAGAGATCTAATGTCACAGTCTCGCACTCGCGAGATACTCACCAAAACCACTGTGGACCACATGGCCATAATCAAAAAGTACACATCAGGAAGGCAAGAGAAGAACCCTGCACTCAGGATGAAGTGGATGATG']
- index: 1
- key: PB2
- sequence: AATATATTCAATATGGAGAGAATAAAAGAACTAAGAGATCTAATGTCACAGTCTCGCACTCGCGAGATACTCACCAAAACCACTGTGGACCACATGGCCATAATCAAAAAGTACACATCAGGAAGGCAAGAGAAGAACCCTGCACTCAGGATGAAGTGGATGATG
['>', '2', 'PB1', 'AACCATTTGAATGGATGTCAATCCGACTTTACTTTTCTTGAAAGTTCCAGCGCAAAATGCCATAAGCACCACATTTCCCTATACTGGAGACCCTCC']
- index: 2
- key: PB1
- sequence: AACCATTTGAATGGATGTCAATCCGACTTTACTTTTCTTGAAAGTTCCAGCGCAAAATGCCATAAGCACCACATTTCCCTATACTGGAGACCCTCC
这个文件有多大?最大序列有多大?文件大小从几百兆到GB不等。最大序列大小不应超过1000-2000个碱基(字符,不确定的行数),内存错误在哪里?请发布完整的回溯。感谢您提供指向文件解析器实际代码的链接。我将能够修改,以工作刚刚好。你知道为什么当前的代码不起作用吗?我想从“下次不做什么”的角度来了解。很难说为什么你的代码会失败,但我会按照hughdbrown的建议重新编写。将每个块累积到列表中,仅在最后,将列表加入一个字符串并返回一个具有名称和序列的元组。另外,我不知道您使用的是哪个python版本,但是Python3对Python2.x使用的字符串使用了更多的内存。他们正在用Python 3.3解决这个问题。我已经用一个可能的解决方案编辑了我的答案。我真的无法理解这个概念。为什么“如果名称:”?没有发电机我也能做到,但发电机压在我头上了。有人请解释一下“如果名称”和屈服是如何工作的。如果你的名字?这个名字没有。这是怎么回事?!何时为数据分配名称?在询问bool(name)==True后,它会给出结果。我不明白什么时候开始data@jukhamil该文件将在
中用
打开。此外,文件是自己的迭代器。看看:
def read_fasta(filename):
name = None
with open(filename) as file:
for line in file:
if line[0] == ">":
if name:
yield (name, seq)
name = line[1:-1].split("|")[0]
seq = ""
else:
seq += line[:-1]
yield (name, seq)