如何在Python中将文本文件中的块数据解析为2D数组?
我正在尝试解析具有以下结构的文本文件:如何在Python中将文本文件中的块数据解析为2D数组?,python,arrays,parsing,text,block,Python,Arrays,Parsing,Text,Block,我正在尝试解析具有以下结构的文本文件: latitude 5.0000 number_of_data_values 9 0.1 0.2 0.3 0.4 1.1 1.2 1.3 1.4 8.1 latitude 4.3000 number_of_data_values 9 0.1 0.2 0.3
latitude 5.0000
number_of_data_values 9
0.1 0.2 0.3 0.4
1.1 1.2 1.3 1.4
8.1
latitude 4.3000
number_of_data_values 9
0.1 0.2 0.3 0.4
1.1 1.2 1.3 1.4
8.1
latitude 4.0000
number_of_data_values 9
0.1 0.2 0.3 0.4
1.1 1.2 1.3 1.4
8.1
...
每个不同的纬度
编号都是不同的数组线。
number\u of_data\u value
是列数(与文件一致)
对于本例,我希望读取文件并输出一个3×9的二维数组,如下所示:
array = [[0.1,0.2,0.3,0.4,1.1,1.2,1.3,1.4,8.1],
[0.1,0.2,0.3,0.4,1.1,1.2,1.3,1.4,8.1],
[0.1,0.2,0.3,0.4,1.1,1.2,1.3,1.4,8.1]]
我尝试过用循环遍历这一行,但我正在寻找一种更有效的方法,因为我可能会处理大量的输入文件。似乎很简单。解析数字的部分就是
line.split()
。根据输入数据格式的稳定性,rest或解析可以被硬化或软化
results = []
latitude = None
numbers_total = None
value_list = []
for line in text.splitlines():
if line.startswith('latitude '):
if latitude is not None:
assert len(value_list) == numbers_total
results.append((latitude, value_list))
value_list = []
latitude = line.split()[-1]
elif line.startswith('number_of_data_values '):
numbers_total = int(line.split()[-1])
else:
value_list.extend(line.split())
# Make sure the last block gets added to the results.
if latitude is not None:
assert len(value_list) == numbers_total
results.append((latitude, value_list))
value_list = []
for latitude, value_list in results:
print 'latitude %r: %r' % (latitude, value_list)
这将产生:
latitude '5.0000': ['0.1', '0.2', '0.3', '0.4', '1.1', '1.2', '1.3', '1.4', '8.1']
latitude '4.3000': ['0.1', '0.2', '0.3', '0.4', '1.1', '1.2', '1.3', '1.4', '8.1']
latitude '4.0000': ['0.1', '0.2', '0.3', '0.4', '1.1', '1.2', '1.3', '1.4', '8.1']
逐行实现非常容易理解。假设您的
latitude
总是从新行开始(这不是您的示例给出的,但可能是输入错误),您可以执行以下操作:
latitudes = []
counts = []
blocks = []
current_block = []
for line in test:
print line
if line.startswith("latitude"):
# New block: add the previous one to `blocks` and reset
blocks.append(current_block)
current_block = []
latitudes.append(float(line.split()[-1]))
elif line.startswith("number_of_data"):
# Just append the current count to the list
counts.append(int(line.split()[-1]))
else:
# Update the current block
current_block += [float(f) for f in line.strip().split()]
# Make sure to add the last block...
blocks.append(current_block)
# And to remove the first (empty) one
blocks.pop(0)
您可以检查所有块是否具有正确的大小:
all(len(b)==c for (c,b) in zip(counts,blocks))
替代解决方案
如果您关心循环,您可能需要考虑查询文件的内存映射版本。其思想是找到以
纬度开始的线的位置。一旦找到一行,找到下一行,就有了一块文本:将前两行(以纬度开始的一行和以数据数量开始的一行)合并并处理
import mmap
with open("crap.txt", "r+b") as f:
# Create the mapper
mapper = mmap.mmap(f.fileno(), 0)
# Initialize your output variables
latitudes = []
blocks = []
# Find the beginning of the first block
position = mapper.find("latitude")
# `position` will be -1 if we can't find it
while (position >= 0):
# Move to the beginning of the block
mapper.seek(position)
# Read the first line
lat_line = mapper.readline().strip()
latitudes.append(lat_line.split()[-1])
# Read the second one
zap = mapper.readline()
# Where are we ?
start = mapper.tell()
# Where's the next block ?
position = mapper.find("latitude")
# Read the lines and combine them into a large string
current_block = mapper.read(position-start).replace("\n", " ")
# Transform the string into a list of floats and update the block
blocks.append(list(float(i) for i in current_block.split() if i))