Python 处理大文件的最快方法？
我有多个3 GB制表符分隔的文件。每个文件中有2000万行。所有行都必须独立处理，任何两行之间都没有关系。 我的问题是，什么会更快A.使用以下方法逐行阅读：Python 处理大文件的最快方法？,python,file,python-2.7,filereader,Python,File,Python 2.7,Filereader,我有多个3 GB制表符分隔的文件。每个文件中有2000万行。所有行都必须独立处理，任何两行之间都没有关系。 我的问题是，什么会更快A.使用以下方法逐行阅读： with open() as infile: for line in infile: 或者B.将文件以块的形式读入内存并进行处理，比如每次250 MB 处理不是很复杂，我只是获取列1到List1、列2到List2等中的值。可能需要同时添加一些列值 我在一个有30GB内存的linux机器上使用Python2.7。ASCII文本 有没
with open() as infile:
for line in infile:
List1List2CSVReader我不必用python来做，任何其他语言或数据库使用想法都是受欢迎的。听起来您的代码是I/O绑定的。这意味着，如果您花费90%的时间从磁盘读取数据，多处理将不会有任何帮助，让额外的7个进程等待下一次读取也不会有任何帮助 而且，虽然使用CSV读取模块（无论是stdlib的
CSVpassos.openos.read（1024*1024）由于您使用的是Python2.x，Python依赖于C stdio库来猜测一次要缓冲多少，因此强制它缓冲更多可能是值得的。最简单的方法是对一些较大的
bufsizereadlines（bufsize）bufsize = 65536
with open(path) as infile:
while True:
lines = infile.readlines(bufsize)
if not lines:
break
for line in lines:
process(line)
with open(path) as infile:
m = mmap.mmap(infile, 0, access=mmap.ACCESS_READ)
同时，假设您在一个64位系统上，您可能首先想尝试使用而不是读取文件。这当然不能保证会更好，但可能会更好，这取决于您的系统。例如：
bufsize = 65536
with open(path) as infile:
while True:
lines = infile.readlines(bufsize)
if not lines:
break
for line in lines:
process(line)
with open(path) as infile:
m = mmap.mmap(infile, 0, access=mmap.ACCESS_READ)
mmapstr文件readlinereadlinesre不幸的是，Python没有公开用于在C语言中进行优化的调用（例如，显式设置
MADV_SEQUENTIALctypeslibcdef chunkify_file(fname, size=1024*1024*1000, skiplines=-1):
"""
function to divide a large text file into chunks each having size ~= size so that the chunks are line aligned
Params :
fname : path to the file to be chunked
size : size of each chink is ~> this
skiplines : number of lines in the begining to skip, -1 means don't skip any lines
Returns :
start and end position of chunks in Bytes
"""
chunks = []
fileEnd = os.path.getsize(fname)
with open(fname, "rb") as f:
if(skiplines > 0):
for i in range(skiplines):
f.readline()
chunkEnd = f.tell()
count = 0
while True:
chunkStart = chunkEnd
f.seek(f.tell() + size, os.SEEK_SET)
f.readline() # make this chunk line aligned
chunkEnd = f.tell()
chunks.append((chunkStart, chunkEnd - chunkStart, fname))
count+=1
if chunkEnd > fileEnd:
break
return chunks
def parallel_apply_line_by_line_chunk(chunk_data):
"""
function to apply a function to each line in a chunk
Params :
chunk_data : the data for this chunk
Returns :
list of the non-None results for this chunk
"""
chunk_start, chunk_size, file_path, func_apply = chunk_data[:4]
func_args = chunk_data[4:]
t1 = time.time()
chunk_res = []
with open(file_path, "rb") as f:
f.seek(chunk_start)
cont = f.read(chunk_size).decode(encoding='utf-8')
lines = cont.splitlines()
for i,line in enumerate(lines):
ret = func_apply(line, *func_args)
if(ret != None):
chunk_res.append(ret)
return chunk_res
def parallel_apply_line_by_line(input_file_path, chunk_size_factor, num_procs, skiplines, func_apply, func_args, fout=None):
"""
function to apply a supplied function line by line in parallel
Params :
input_file_path : path to input file
chunk_size_factor : size of 1 chunk in MB
num_procs : number of parallel processes to spawn, max used is num of available cores - 1
skiplines : number of top lines to skip while processing
func_apply : a function which expects a line and outputs None for lines we don't want processed
func_args : arguments to function func_apply
fout : do we want to output the processed lines to a file
Returns :
list of the non-None results obtained be processing each line
"""
num_parallel = min(num_procs, psutil.cpu_count()) - 1
jobs = chunkify_file(input_file_path, 1024 * 1024 * chunk_size_factor, skiplines)
jobs = [list(x) + [func_apply] + func_args for x in jobs]
print("Starting the parallel pool for {} jobs ".format(len(jobs)))
lines_counter = 0
pool = mp.Pool(num_parallel, maxtasksperchild=1000) # maxtaskperchild - if not supplied some weird happend and memory blows as the processes keep on lingering
outputs = []
for i in range(0, len(jobs), num_parallel):
print("Chunk start = ", i)
t1 = time.time()
chunk_outputs = pool.map(parallel_apply_line_by_line_chunk, jobs[i : i + num_parallel])
for i, subl in enumerate(chunk_outputs):
for x in subl:
if(fout != None):
print(x, file=fout)
else:
outputs.append(x)
lines_counter += 1
del(chunk_outputs)
gc.collect()
print("All Done in time ", time.time() - t1)
print("Total lines we have = {}".format(lines_counter))
pool.close()
pool.terminate()
return outputs
def count_words_line(line):
return len(line.strip().split())
parallel_apply_line_by_line(input_file_path, 100, 8, 0, count_words_line, [], fout=None)
使用这种方法，我的速度比普通的逐行读取大小为20GB的示例文件的速度提高了约8倍。在这种文件中，我对每行执行一些中等复杂的处理。您的代码是受I/O限制的，还是受CPU限制的？换言之，处理比阅读花费更多的时间吗？如果是这样，您可能可以通过多处理来加速它；如果没有，您的后台进程将花费所有时间等待下一次读取，您将得不到任何好处。同时，
for line in infle:ioinfle.readlines（65536）：