Python 限制并行工作的线程数

我正在制作一个函数，将文件从本地计算机复制到远程计算机，并创建线程并行执行sftp

def copyToServer():
    //does  copy file given host name and credentials

for i in hostsList:
    hostname = i
    username = defaultLogin
    password = defaultPassword
    thread = threading.Thread(target=copyToServer, args=(hostname, username, password, destPath, localPath))
    threadsArray.append(thread)
    thread.start()

---

这将创建线程并开始并行复制，但我想将其限制为每次处理50个线程，因为服务器总数可能太多

您需要调整代码以共享和跟踪公共值

这可以用一个简单的方法来完成。对象持有一个内部计数器，每个线程都试图获取它。如果计数器大于您定义的最大值，线程将无法获取一个计数器，并将被阻塞，直到有一个计数器释放

一个简短的示例显示，对于最多5个并行线程，其中一半线程立即执行，其他线程被阻塞并等待：

import threading
import time

maxthreads = 5
sema = threading.Semaphore(value=maxthreads)
threads = list()

def task(i):
    sema.acquire()
    print "start %s" % (i,)
    time.sleep(2)
    sema.release()

for i in range(10):
    thread = threading.Thread(target=task,args=(str(i)))
    threads.append(thread)
    thread.start()

输出

start 0
start 1
start 2
start 3
start 4

几秒钟后，第一个线程完成，下一个线程执行

start 5
start 6
start 7
start 8
start 9

---

这应该会给你一些

---

对于那些希望通过“快速修复”解决方案来限制python3中“线程化”模块中的线程数量的人 -基本逻辑是将主函数包装到包装器中，并调用包含停止/执行逻辑的包装器

下面这篇文章重用了Andpei提出的解决方案，但是他的文章中的逐字代码不起作用，下面是我的修改

蟒蛇3：

import threading
import time

maxthreads = 3
smphr = threading.Semaphore(value=maxthreads)
threads = list()

SomeInputCollection=("SomeInput1","SomeInput2","SomeInput3","SomeInput4","SomeInput5","SomeInput6")

def yourmainfunction(SomeInput):
    #main function
    print ("Your input was: "+ SomeInput)

def task(SomeInput):
    #yourmainfunction wrapped in a task
    print(threading.currentThread().getName(), 'Starting')
    smphr.acquire()
    yourmainfunction(SomeInput)
    time.sleep(2)
    print(threading.currentThread().getName(), 'Exiting')
    smphr.release()


def main():
    threads = [threading.Thread(name="worker/task", target=task, args=(SomeInput,)) for SomeInput in SomeInputCollection]
    for thread in threads:
        thread.start()
    for thread in threads:
        thread.join()
if __name__== "__main__":
  main()

输出：

worker/task Starting
Your input was: SomeInput1
worker/task Starting
Your input was: SomeInput2
worker/task Starting
Your input was: SomeInput3
worker/task Starting
worker/task Starting
worker/task Starting
worker/task Exiting
Your input was: SomeInput4
worker/task Exiting
worker/task Exiting
Your input was: SomeInput6
Your input was: SomeInput5
worker/task Exiting
worker/task Exiting
worker/task Exiting

---

这应该是公认的答案。效果很好。顺便问一下，你有什么方法可以使用这个解决方案为每个任务增加优先级吗？就像我有一些任务应该能够获得比其他任务更高的优先级。有线程是没有意义的。附加（线程）。此外，此解决方案不适用于

守护进程

线程

import threading
import time

maxthreads = 3
smphr = threading.Semaphore(value=maxthreads)
threads = list()

SomeInputCollection=("SomeInput1","SomeInput2","SomeInput3","SomeInput4","SomeInput5","SomeInput6")

def yourmainfunction(SomeInput):
    #main function
    print ("Your input was: "+ SomeInput)

def task(SomeInput):
    #yourmainfunction wrapped in a task
    print(threading.currentThread().getName(), 'Starting')
    smphr.acquire()
    yourmainfunction(SomeInput)
    time.sleep(2)
    print(threading.currentThread().getName(), 'Exiting')
    smphr.release()


def main():
    threads = [threading.Thread(name="worker/task", target=task, args=(SomeInput,)) for SomeInput in SomeInputCollection]
    for thread in threads:
        thread.start()
    for thread in threads:
        thread.join()
if __name__== "__main__":
  main()

worker/task Starting
Your input was: SomeInput1
worker/task Starting
Your input was: SomeInput2
worker/task Starting
Your input was: SomeInput3
worker/task Starting
worker/task Starting
worker/task Starting
worker/task Exiting
Your input was: SomeInput4
worker/task Exiting
worker/task Exiting
Your input was: SomeInput6
Your input was: SomeInput5
worker/task Exiting
worker/task Exiting
worker/task Exiting