python中带循环的多线程
我正在尝试用python中的多线程技术解决euler项目中的问题 在1000位数字中找出五个连续数字的最大乘积。号码可以找到 我的方法是从原始列表的5块中生成产品,并重复这个过程5次,每次都将起始索引向右移动一个 这是我的线程类python中带循环的多线程,python,multithreading,Python,Multithreading,我正在尝试用python中的多线程技术解决euler项目中的问题 在1000位数字中找出五个连续数字的最大乘积。号码可以找到 我的方法是从原始列表的5块中生成产品,并重复这个过程5次,每次都将起始索引向右移动一个 这是我的线程类 class pThread(threading.Thread): def __init__(self, l): threading.Thread.__init__(self) self.l = l self.p =
class pThread(threading.Thread):
def __init__(self, l):
threading.Thread.__init__(self)
self.l = l
self.p = 0
def run(self):
def greatest_product(l):
"""
Divide the list into chunks of 5 and find the greatest product
"""
def product(seq):
return reduce(lambda x,y : x*y, seq)
def chunk_product(l, n=5):
for i in range(0, len(l), n):
yield product(l[i:i+n])
result = 0
for p in chunk_product(num):
result = result > p and result or p
return result
self.p = greatest_product(self.l)
当我尝试创建5个线程来覆盖原始列表中的所有5位数区块时,下面的手动方法给出了正确答案,num
是我从文本中解析的一位数列表:
thread1 = pThread(num)
del num[0]
thread2 = pThread(num)
del num[0]
thread3 = pThread(num)
del num[0]
thread4 = pThread(num)
del num[0]
thread5 = pThread(num)
thread1.start()
thread2.start()
thread3.start()
thread4.start()
thread5.start()
thread1.join()
thread2.join()
thread3.join()
thread4.join()
thread5.join()
def max(*args):
result = 0
for i in args:
result = i > result and i or result
return result
print max(thread1.p, thread2.p, thread3.p, thread4.p, thread5.p)
但这并不能给出正确的结果:
threads = []
for i in range(0, 4):
tmp = num[:]
del tmp[0:i+1]
thread = pThread(tmp)
thread.start()
threads.append(thread)
for i in range(0, 4):
threads[i].join()
我做错了什么?我对多线程非常陌生,所以请温和一点。我尝试这一点主要是为了练习多处理,并学习如何使用argparse 这需要大约4-5千兆内存,以防您的机器没有太多内存
python euler.py -l 50000000 -n 100 -p 8
Took 5.836833333969116 minutes
The largest product of 100 consecutive numbers is: a very large number
如果在命令行中键入python euler.py-h,则会得到:
usage: euler.py [-h] -l L [L ...] -n N [-p P]
Calculates the product of consecutive numbers and return the largest product.
optional arguments:
-h, --help show this help message and exit
-l L [L ...] A single number or list of numbers, where each # is seperated
by a space
-n N A number that specifies how many consecutive numbers should be
multiplied together.
-p P Number of processes to create. Optional, defaults to the # of
cores on the pc.
以及守则:
"""A multiprocess iplementation for calculation the maximum product of N consecutive
numbers in a given range (list of numbers)."""
import multiprocessing
import math
import time
import operator
from functools import reduce
import argparse
def euler8(alist,lenNums):
"""Returns the largest product of N consecutive numbers in a given range"""
return max(reduce(operator.mul, alist[i:i+lenNums]) for i in range(len(alist)))
def split_list_multi(listOfNumbers,numLength,threads):
"""Split a list into N parts where N is the # of processes."""
fullLength = len(listOfNumbers)
single = math.floor(fullLength/threads)
results = {}
counter = 0
while counter < threads:
if counter == (threads-1):
temp = listOfNumbers[single*counter::]
if counter == 0:
results[str(counter)] = listOfNumbers[single*counter::]
else:
prevListIndex = results[str(counter-1)][-int('{}'.format(numLength-1))::]
newlist = prevListIndex + temp
results[str(counter)] = newlist
else:
temp = listOfNumbers[single*counter:single*(counter+1)]
if counter == 0:
newlist = temp
else:
prevListIndex = results[str(counter-1)][-int('{}'.format(numLength-1))::]
newlist = prevListIndex + temp
results[str(counter)] = newlist
counter += 1
return results,threads
def worker(listNumbers,number,output):
"""A worker. Used to run seperate processes and put the results in the queue"""
result = euler8(listNumbers,number)
output.put(result)
def main(listOfNums,lengthNumbers,numCores=multiprocessing.cpu_count()):
"""Runs the module.
listOfNums must be a list of ints, or single int
lengthNumbers is N (an int) where N is the # of consecutive numbers to multiply together
numCores (an int) defaults to however many the cpu has, can specify a number if you choose."""
if isinstance(listOfNums,list):
if len(listOfNums) == 1:
valuesToSplit = [i for i in range(int(listOfNums[0]))]
else:
valuesToSplit = [int(i) for i in listOfNums]
elif isinstance(listOfNums,int):
valuesToSplit = [i for i in range(listOfNums)]
else:
print('First arg must be a number or a list of numbers')
split = split_list_multi(valuesToSplit,lengthNumbers,numCores)
done_queue = multiprocessing.Queue()
jobs = []
startTime = time.time()
for num in range(split[1]):
numChunks = split[0][str(num)]
thread = multiprocessing.Process(target=worker, args=(numChunks,lengthNumbers,done_queue))
jobs.append(thread)
thread.start()
resultlist = []
for i in range(split[1]):
resultlist.append(done_queue.get())
for j in jobs:
j.join()
resultlist = max(resultlist)
endTime = time.time()
totalTime = (endTime-startTime)/60
print("Took {} minutes".format(totalTime))
return print("The largest product of {} consecutive numbers is: {}".format(lengthNumbers, resultlist))
if __name__ == '__main__':
#To call the module from the commandline with arguments
parser = argparse.ArgumentParser(description="""Calculates the product of consecutive numbers \
and return the largest product.""")
parser.add_argument('-l', nargs='+', required=True,
help='A single number or list of numbers, where each # is seperated by a space')
parser.add_argument('-n', required=True, type=int,
help = 'A number that specifies how many consecutive numbers should be \
multiplied together.')
parser.add_argument('-p', default=multiprocessing.cpu_count(), type=int,
help='Number of processes to create. Optional, defaults to the # of cores on the pc.')
args = parser.parse_args()
main(args.l, args.n, args.p)
threads = []
for i in range(5):
tmp = num[:]
del tmp[0:i]
thread = pThread(tmp)
thread.start()
threads.append(thread)
for i in range(5):
threads[i].join()
print len(threads), map(lambda th: th.p, threads)
print max(map(lambda th: th.p, threads))
用于计算N个连续变量的最大乘积的多进程i实现
给定范围内的数字(数字列表)。“”“
导入多处理
输入数学
导入时间
进口经营者
从functools导入reduce
导入argparse
def euler8(列表、列侬):
“”“返回给定范围内N个连续数字的最大乘积”“”
对于范围内的i(len(alist)),返回最大值(reduce(operator.mul,alist[i:i+lenNums]))
def split_list_multi(列表编号、长度、线程):
“”“将列表拆分为N个部分,其中N是进程的#。”“”
fullLength=len(列表编号)
单=数学地板(全长/螺纹)
结果={}
计数器=0
而计数器<线程:
如果计数器==(线程-1):
temp=列表编号[单个*计数器::]
如果计数器==0:
结果[str(计数器)]=列表编号[单个*计数器::]
其他:
prevListIndex=results[str(counter-1)][-int(“{}”。格式(numLength-1)):]
newlist=prevListIndex+temp
结果[str(计数器)]=新列表
其他:
temp=列表编号[单个*计数器:单个*(计数器+1)]
如果计数器==0:
newlist=temp
其他:
prevListIndex=results[str(counter-1)][-int(“{}”。格式(numLength-1)):]
newlist=prevListIndex+temp
结果[str(计数器)]=新列表
计数器+=1
返回结果、线程
def工作者(列表编号、编号、输出):
“”“工作进程。用于运行单独的进程并将结果放入队列”“”
结果=euler8(列表编号,编号)
输出输出(结果)
def main(列表编号、长度编号、numCores=multiprocessing.cpu\u count()):
“”运行该模块。
listOfNums必须是整数列表或单个整数
lengthNumbers是N(整数),其中N是要相乘的连续数字的#
numCores(int)默认为cpu的数量,如果您选择,可以指定一个数字。”“”
如果isinstance(列表项,列表项):
如果len(listOfNums)==1:
valuesToSplit=[i代表范围内的i(int(listOfNums[0])]
其他:
valuesToSplit=[listOfNums中i的int(i)]
elif isinstance(列表编号,int):
valuesToSplit=[i代表范围内的i(列表编号)]
其他:
打印('第一个参数必须是数字或数字列表')
拆分=拆分\列表\多个(值拆分、长度编号、numCores)
已完成\u队列=多处理。队列()
工作=[]
startTime=time.time()
对于范围内的num(拆分[1]):
numChunks=split[0][str(num)]
线程=多处理。进程(目标=工作者,参数=(numChunks,lengthNumbers,done_队列))
jobs.append(线程)
thread.start()
结果列表=[]
对于范围内的i(拆分[1]):
resultlist.append(done\u queue.get())
对于工作中的j:
j、 加入
结果列表=最大值(结果列表)
endTime=time.time()
总时间=(结束时间开始时间)/60
打印(“用了{}分钟”。格式(总时间))
返回打印(“{}个连续数字的最大乘积为:{}”。格式(lengthNumbers,resultlist))
如果uuuu name uuuuuu='\uuuuuuu main\uuuuuuu':
#使用参数从命令行调用模块
parser=argparse.ArgumentParser(description=“”计算连续数字的乘积\
并返回最大的产品。”“”)
add_参数('-l',nargs='+',required=True,
help='一个数字或数字列表,其中每个#用空格分隔')
add_参数('-n',required=True,type=int,
help='一个数字,指定应输入多少个连续数字\
相乘。”)
parser.add_参数('-p',default=multiprocessing.cpu_count(),type=int,
help=”要创建的进程数。可选,默认为“pc上的内核数”。)
args=parser.parse_args()
main(args.l、args.n、args.p)
有三个问题:
import operator as op
print max(reduce(op.mul, num[i:i+5]) for i in range(1000))
for k in range(5):
print max(reduce(op.mul, num[i:i+5]) for i in range(k, 1000, 5))
“手动”方法的一个问题是线程共享num
变量,每个线程都有相同的列表。因此,当您执行delnum[0]
时,所有threadX.l
都会受到影响。事实上,你总是得到相同的答案是由于第二个问题for p in chunk_product(num):
应该是:
for p in chunk_product(l):
因为您希望使用函数最大乘积(l)
的参数,而不是全局变量num
[0、1、2、3]
,因此只生成4个线程。另外,您希望删除值tmp[0:i]
,而不是tmp[0:i+1]
。这是c