Python 最重要的是什么;蟒蛇的;以块的形式迭代列表的方法?
我有一个Python脚本,它将整数列表作为输入,我需要一次处理四个整数。不幸的是,我无法控制输入,否则我会将其作为四元素元组的列表传入。目前,我正在以这种方式对其进行迭代:Python 最重要的是什么;蟒蛇的;以块的形式迭代列表的方法?,python,list,loops,optimization,chunks,Python,List,Loops,Optimization,Chunks,我有一个Python脚本,它将整数列表作为输入,我需要一次处理四个整数。不幸的是,我无法控制输入,否则我会将其作为四元素元组的列表传入。目前,我正在以这种方式对其进行迭代: for i in range(0, len(ints), 4): # dummy op for example code foo += ints[i] * ints[i + 1] + ints[i + 2] * ints[i + 3] 不过,这看起来很像“C思维”,这让我怀疑有一种更像蟒蛇的方式来处理这种情
for i in range(0, len(ints), 4):
# dummy op for example code
foo += ints[i] * ints[i + 1] + ints[i + 2] * ints[i + 3]
不过,这看起来很像“C思维”,这让我怀疑有一种更像蟒蛇的方式来处理这种情况。该列表在迭代后被丢弃,因此不需要保留它。也许这样更好
while ints:
foo += ints[0] * ints[1] + ints[2] * ints[3]
ints[0:4] = []
不过,感觉还是不太对-/
相关问题:在您的第二种方法中,我将通过这样做进入下一组4人:
ints = ints[4:]
然而,我还没有做任何绩效评估,所以我不知道哪一个可能更有效
话虽如此,我通常会选择第一种方法。这并不漂亮,但这通常是与外部世界接触的结果
import itertools
def chunks(iterable,size):
it = iter(iterable)
chunk = tuple(itertools.islice(it,size))
while chunk:
yield chunk
chunk = tuple(itertools.islice(it,size))
# though this will throw ValueError if the length of ints
# isn't a multiple of four:
for x1,x2,x3,x4 in chunks(ints,4):
foo += x1 + x2 + x3 + x4
for chunk in chunks(ints,4):
foo += sum(chunk)
另一种方式:
import itertools
def chunks2(iterable,size,filler=None):
it = itertools.chain(iterable,itertools.repeat(filler,size-1))
chunk = tuple(itertools.islice(it,size))
while len(chunk) == size:
yield chunk
chunk = tuple(itertools.islice(it,size))
# x2, x3 and x4 could get the value 0 if the length is not
# a multiple of 4.
for x1,x2,x3,x4 in chunks2(ints,4,0):
foo += x1 + x2 + x3 + x4
chunk\u size=4
对于范围内的i(0,len(ints),块大小):
chunk=ints[i:i+chunk\u size]
#处理大小的块
使用任何顺序工作:
text = "I am a very, very helpful text"
for group in chunker(text, 7):
print(repr(group),)
# 'I am a ' 'very, v' 'ery hel' 'pful te' 'xt'
print '|'.join(chunker(text, 10))
# I am a ver|y, very he|lpful text
animals = ['cat', 'dog', 'rabbit', 'duck', 'bird', 'cow', 'gnu', 'fish']
for group in chunker(animals, 3):
print(group)
# ['cat', 'dog', 'rabbit']
# ['duck', 'bird', 'cow']
# ['gnu', 'fish']
如果列表较大,执行此操作的最佳方法是使用生成器:
def get_chunk(iterable, chunk_size):
result = []
for item in iterable:
result.append(item)
if len(result) == chunk_size:
yield tuple(result)
result = []
if len(result) > 0:
yield tuple(result)
for x in get_chunk([1,2,3,4,5,6,7,8,9,10], 3):
print x
(1, 2, 3)
(4, 5, 6)
(7, 8, 9)
(10,)
修改自Python文档的一节:
示例
grouper('ABCDEFG', 3, 'x') # --> 'ABC' 'DEF' 'Gxx'
注意:在Python 2上使用izip_longest
而不是zip_longest
,因为还没有人提到它,这里有一个zip()
解决方案:
>>> def chunker(iterable, chunksize):
... return zip(*[iter(iterable)]*chunksize)
只有当序列的长度总是可以被块大小整除,或者如果后面的块不可以整除,则不关心它时,它才起作用
例如:
>>> s = '1234567890'
>>> chunker(s, 3)
[('1', '2', '3'), ('4', '5', '6'), ('7', '8', '9')]
>>> chunker(s, 4)
[('1', '2', '3', '4'), ('5', '6', '7', '8')]
>>> chunker(s, 5)
[('1', '2', '3', '4', '5'), ('6', '7', '8', '9', '0')]
>>> s = '1234567890'
>>> chunker(s, 3)
[('1', '2', '3'), ('4', '5', '6'), ('7', '8', '9'), ('0', None, None)]
>>> chunker(s, 4)
[('1', '2', '3', '4'), ('5', '6', '7', '8'), ('9', '0', None, None)]
>>> chunker(s, 5)
[('1', '2', '3', '4', '5'), ('6', '7', '8', '9', '0')]
或使用返回迭代器而不是列表:
>>> from itertools import izip
>>> def chunker(iterable, chunksize):
... return izip(*[iter(iterable)]*chunksize)
可以使用以下方法固定填充:
使用map()而不是zip()修复了J.F.Sebastian回答中的填充问题:
>>> def chunker(iterable, chunksize):
... return map(None,*[iter(iterable)]*chunksize)
例如:
>>> s = '1234567890'
>>> chunker(s, 3)
[('1', '2', '3'), ('4', '5', '6'), ('7', '8', '9')]
>>> chunker(s, 4)
[('1', '2', '3', '4'), ('5', '6', '7', '8')]
>>> chunker(s, 5)
[('1', '2', '3', '4', '5'), ('6', '7', '8', '9', '0')]
>>> s = '1234567890'
>>> chunker(s, 3)
[('1', '2', '3'), ('4', '5', '6'), ('7', '8', '9'), ('0', None, None)]
>>> chunker(s, 4)
[('1', '2', '3', '4'), ('5', '6', '7', '8'), ('9', '0', None, None)]
>>> chunker(s, 5)
[('1', '2', '3', '4', '5'), ('6', '7', '8', '9', '0')]
这个问题的理想解决方案是使用迭代器(而不仅仅是序列)。它也应该很快
这是itertools文档提供的解决方案:
def grouper(n, iterable, fillvalue=None):
#"grouper(3, 'ABCDEFG', 'x') --> ABC DEF Gxx"
args = [iter(iterable)] * n
return itertools.izip_longest(fillvalue=fillvalue, *args)
在我的mac book air上使用ipython的%timeit
,我每个循环得到47.5 us
然而,这对我来说真的不起作用,因为结果被填充成了均匀大小的组。没有填充的解决方案稍微复杂一些。最简单的解决方案可能是:
def grouper(size, iterable):
i = iter(iterable)
while True:
out = []
try:
for _ in range(size):
out.append(i.next())
except StopIteration:
yield out
break
yield out
简单,但速度相当慢:每个循环693 us
我能想到的最佳解决方案是使用islice
进行内部循环:
def grouper(size, iterable):
it = iter(iterable)
while True:
group = tuple(itertools.islice(it, None, size))
if not group:
break
yield group
使用相同的数据集,我每个循环得到305个us
无法更快地获得纯解决方案,我提供了以下解决方案,并提出了一个重要的警告:如果您的输入数据中包含filldata
的实例,您可能会得到错误的答案
def grouper(n, iterable, fillvalue=None):
#"grouper(3, 'ABCDEFG', 'x') --> ABC DEF Gxx"
args = [iter(iterable)] * n
for i in itertools.izip_longest(fillvalue=fillvalue, *args):
if tuple(i)[-1] == fillvalue:
yield tuple(v for v in i if v != fillvalue)
else:
yield i
我真的不喜欢这个答案,但它要快得多。每环路124 us另一个答案,其优点是:
1) 易于理解
2) 适用于任何iterable,而不仅仅是序列(上面的一些答案会阻塞文件句柄)
3) 不会一次将块加载到内存中
4) 不会在内存中生成对同一迭代器的引用的块长列表
5) 列表末尾没有填充填充值
也就是说,我没有对它进行计时,因此它可能比一些更聪明的方法慢,并且考虑到用例,一些优势可能是不相关的
def chunkiter(iterable, size):
def inneriter(first, iterator, size):
yield first
for _ in xrange(size - 1):
yield iterator.next()
it = iter(iterable)
while True:
yield inneriter(it.next(), it, size)
In [2]: i = chunkiter('abcdefgh', 3)
In [3]: for ii in i:
for c in ii:
print c,
print ''
...:
a b c
d e f
g h
更新:
由于内部循环和外部循环从同一迭代器中提取值,因此存在两个缺点:
1) continue在外部循环中没有按预期工作-它只是继续到下一个项目,而不是跳过一个块。然而,这似乎不是一个问题,因为在外循环中没有什么可测试的。
2) break在内部循环中无法按预期工作-控件将在迭代器中的下一项再次进入内部循环。要跳过整个块,可以将内部迭代器(上面的ii)包装在一个元组中,例如元组(ii)中的c的,,或者设置一个标志并耗尽迭代器。
类似于其他建议,但不完全相同,我喜欢这样做,因为它简单易读:
it = iter([1, 2, 3, 4, 5, 6, 7, 8, 9])
for chunk in zip(it, it, it, it):
print chunk
>>> (1, 2, 3, 4)
>>> (5, 6, 7, 8)
这样就不会得到最后一个部分块。如果你想把(9,None,None,None)
作为最后一块,只需使用itertools
中的izip\u longest
,使用一些小功能和东西对我来说并不吸引;我更喜欢使用切片:
data = [...]
chunk_size = 10000 # or whatever
chunks = [data[i:i+chunk_size] for i in xrange(0,len(data),chunk_size)]
for chunk in chunks:
...
我需要一个解决方案,也将与集和发电机。我想不出任何简短而漂亮的东西,但至少它是可读的
def chunker(seq, size):
res = []
for el in seq:
res.append(el)
if len(res) == size:
yield res
res = []
if res:
yield res
名单:
设置:
发电机:
>>> list(chunker((i for i in range(10)), 3))
[[0, 1, 2], [3, 4, 5], [6, 7, 8], [9]]
另一种方法是使用iter的双参数形式:
from itertools import islice
def group(it, size):
it = iter(it)
return iter(lambda: tuple(islice(it, size)), ())
这可以很容易地调整为使用填充(这类似于的答案):
这些甚至可以组合用于可选填充:
_no_pad = object()
def group(it, size, pad=_no_pad):
if pad == _no_pad:
it = iter(it)
sentinel = ()
else:
it = chain(iter(it), repeat(pad))
sentinel = (pad,) * size
return iter(lambda: tuple(islice(it, size)), sentinel)
您可以使用或使用库中的函数:
这些函数还具有迭代器版本ipartition
和ichunks
,在这种情况下效率更高
您还可以查看。一行程序、临时解决方案,以按大小4
-
for a, b, c, d in zip(x[0::4], x[1::4], x[2::4], x[3::4]):
... do something with a, b, c and d ...
要避免所有到列表的转换导入itertools
,请执行以下操作:
>>> for k, g in itertools.groupby(xrange(35), lambda x: x/10):
... list(g)
产生:
...
0 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
1 [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
2 [20, 21, 22, 23, 24, 25, 26, 27, 28, 29]
3 [30, 31, 32, 33, 34]
>>>
我选中了groupby
,但它不会转换为list或使用len
,因此我(认为)这将延迟每个值的解析,直到实际使用为止。可悲的是,所有可用的答案(此时)似乎都没有提供这种变化
显然,如果您需要依次处理每个项,请在g上嵌套for循环:
for k,g in itertools.groupby(xrange(35), lambda x: x/10):
for i in g:
# do what you need to do with individual items
# now do what you need to do with the whole group
我对此特别感兴趣的是,需要使用生成器将最多1000次的更改批量提交到gmail API:
messages = a_generator_which_would_not_be_smart_as_a_list
for idx, batch in groupby(messages, lambda x: x/1000):
batch_request = BatchHttpRequest()
for message in batch:
batch_request.add(self.service.users().messages().modify(userId='me', id=message['id'], body=msg_labels))
http = httplib2.Http()
self.credentials.authorize(http)
batch_request.execute(http=http)
使用NumPy很简单:
ints = array([1, 2, 3, 4, 5, 6, 7, 8])
for int1, int2 in ints.reshape(-1, 2):
print(int1, int2)
输出:
1 2
3 4
5 6
7 8
关于J.F.Sebastian给出的解决方案:
它很聪明,但有一个缺点——总是返回元组。如何得到s
from funcy import partition
for a, b, c, d in partition(4, ints):
foo += a * b * c * d
for a, b, c, d in zip(x[0::4], x[1::4], x[2::4], x[3::4]):
... do something with a, b, c and d ...
>>> for k, g in itertools.groupby(xrange(35), lambda x: x/10):
... list(g)
...
0 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
1 [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
2 [20, 21, 22, 23, 24, 25, 26, 27, 28, 29]
3 [30, 31, 32, 33, 34]
>>>
for k,g in itertools.groupby(xrange(35), lambda x: x/10):
for i in g:
# do what you need to do with individual items
# now do what you need to do with the whole group
messages = a_generator_which_would_not_be_smart_as_a_list
for idx, batch in groupby(messages, lambda x: x/1000):
batch_request = BatchHttpRequest()
for message in batch:
batch_request.add(self.service.users().messages().modify(userId='me', id=message['id'], body=msg_labels))
http = httplib2.Http()
self.credentials.authorize(http)
batch_request.execute(http=http)
ints = array([1, 2, 3, 4, 5, 6, 7, 8])
for int1, int2 in ints.reshape(-1, 2):
print(int1, int2)
1 2
3 4
5 6
7 8
def chunker(iterable, chunksize):
return zip(*[iter(iterable)]*chunksize)
class IteratorExhausted(Exception):
pass
def translate_StopIteration(iterable, to=IteratorExhausted):
for i in iterable:
yield i
raise to # StopIteration would get ignored because this is generator,
# but custom exception can leave the generator.
def custom_zip(*iterables, reductor=tuple):
iterators = tuple(map(translate_StopIteration, iterables))
while True:
try:
yield reductor(next(i) for i in iterators)
except IteratorExhausted: # when any of iterators get exhausted.
break
def chunker(data, size, reductor=tuple):
return custom_zip(*[iter(data)]*size, reductor=reductor)
>>> for i in chunker('12345', 2):
... print(repr(i))
...
('1', '2')
('3', '4')
>>> for i in chunker('12345', 2, ''.join):
... print(repr(i))
...
'12'
'34'
def chunks(seq, size):
it = iter(seq)
while True:
ret = tuple(next(it) for _ in range(size))
if len(ret) == size:
yield ret
else:
raise StopIteration()
>>> def foo():
... i = 0
... while True:
... i += 1
... yield i
...
>>> c = chunks(foo(), 3)
>>> c.next()
(1, 2, 3)
>>> c.next()
(4, 5, 6)
>>> list(chunks('abcdefg', 2))
[('a', 'b'), ('c', 'd'), ('e', 'f')]
def chunker(iterable, n):
"""Yield iterable in chunk sizes.
>>> chunks = chunker('ABCDEF', n=4)
>>> chunks.next()
['A', 'B', 'C', 'D']
>>> chunks.next()
['E', 'F']
"""
it = iter(iterable)
while True:
chunk = []
for i in range(n):
try:
chunk.append(next(it))
except StopIteration:
yield chunk
raise StopIteration
yield chunk
if __name__ == '__main__':
import doctest
doctest.testmod()
def chunk_iter(iterable, chunk_size):
it = iter(iterable)
while True:
chunk = tuple(next(it) for _ in range(chunk_size))
if not chunk:
break
yield chunk
import itertools
def split_groups(iter_in, group_size):
return ((x for _, x in item) for _, item in itertools.groupby(enumerate(iter_in), key=lambda x: x[0] // group_size))
for x, y, z, w in split_groups(range(16), 4):
foo += x * y + z * w
#!/usr/bin/env python3
from itertools import zip_longest
_UNDEFINED = object()
def chunker(iterable, chunksize, fillvalue=_UNDEFINED):
"""
Collect data into chunks and optionally pad it.
Performance worsens as `chunksize` approaches 1.
Inspired by:
https://docs.python.org/3/library/itertools.html#itertools-recipes
"""
args = [iter(iterable)] * chunksize
chunks = zip_longest(*args, fillvalue=fillvalue)
yield from (
filter(lambda val: val is not _UNDEFINED, chunk)
if chunk[-1] is _UNDEFINED
else chunk
for chunk in chunks
) if fillvalue is _UNDEFINED else chunks
from iteration_utilities import grouper
seq = list(range(20))
for group in grouper(seq, 4):
print(group)
(0, 1, 2, 3)
(4, 5, 6, 7)
(8, 9, 10, 11)
(12, 13, 14, 15)
(16, 17, 18, 19)
from iteration_utilities import grouper
seq = list(range(17))
for group in grouper(seq, 4):
print(group)
# (0, 1, 2, 3)
# (4, 5, 6, 7)
# (8, 9, 10, 11)
# (12, 13, 14, 15)
# (16,)
for group in grouper(seq, 4, fillvalue=None):
print(group)
# (0, 1, 2, 3)
# (4, 5, 6, 7)
# (8, 9, 10, 11)
# (12, 13, 14, 15)
# (16, None, None, None)
for group in grouper(seq, 4, truncate=True):
print(group)
# (0, 1, 2, 3)
# (4, 5, 6, 7)
# (8, 9, 10, 11)
# (12, 13, 14, 15)
import iteration_utilities
import itertools
from itertools import zip_longest
def consume_all(it):
return iteration_utilities.consume(it, None)
import simple_benchmark
b = simple_benchmark.BenchmarkBuilder()
@b.add_function()
def grouper(l, n):
return consume_all(iteration_utilities.grouper(l, n))
def Craz_inner(iterable, n, fillvalue=None):
args = [iter(iterable)] * n
return zip_longest(*args, fillvalue=fillvalue)
@b.add_function()
def Craz(iterable, n, fillvalue=None):
return consume_all(Craz_inner(iterable, n, fillvalue))
def nosklo_inner(seq, size):
return (seq[pos:pos + size] for pos in range(0, len(seq), size))
@b.add_function()
def nosklo(seq, size):
return consume_all(nosklo_inner(seq, size))
def SLott_inner(ints, chunk_size):
for i in range(0, len(ints), chunk_size):
yield ints[i:i+chunk_size]
@b.add_function()
def SLott(ints, chunk_size):
return consume_all(SLott_inner(ints, chunk_size))
def MarkusJarderot1_inner(iterable,size):
it = iter(iterable)
chunk = tuple(itertools.islice(it,size))
while chunk:
yield chunk
chunk = tuple(itertools.islice(it,size))
@b.add_function()
def MarkusJarderot1(iterable,size):
return consume_all(MarkusJarderot1_inner(iterable,size))
def MarkusJarderot2_inner(iterable,size,filler=None):
it = itertools.chain(iterable,itertools.repeat(filler,size-1))
chunk = tuple(itertools.islice(it,size))
while len(chunk) == size:
yield chunk
chunk = tuple(itertools.islice(it,size))
@b.add_function()
def MarkusJarderot2(iterable,size):
return consume_all(MarkusJarderot2_inner(iterable,size))
@b.add_arguments()
def argument_provider():
for exp in range(2, 20):
size = 2**exp
yield size, simple_benchmark.MultiArgument([[0] * size, 10])
r = b.run()
def chunks(it, n, m):
"""Make an iterator over m first chunks of size n.
"""
it = iter(it)
# Chunks are presented as tuples.
return (tuple(next(it) for _ in range(n)) for _ in range(m))
In [2]: chunker(list_, 10)
[10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
[20, 21, 22, 23, 24, 25, 26, 27, 28, 29]
[30, 31, 32, 33, 34, 35, 36, 37, 38, 39]
[40, 41, 42, 43, 44, 45, 46, 47, 48, 49]
[50, 51, 52, 53, 54, 55, 56, 57, 58, 59]
[60, 61, 62, 63, 64, 65, 66, 67, 68, 69]
[70, 71, 72, 73, 74, 75, 76, 77, 78, 79]
[80, 81, 82, 83, 84, 85, 86, 87, 88, 89]
[90, 91, 92, 93, 94, 95, 96, 97, 98, 99]
import more_itertools
for s in more_itertools.chunked(range(9), 4):
print(s)
[0, 1, 2, 3]
[4, 5, 6, 7]
[8]