Python-多处理器，给每个处理器一封来自文本文件的电子邮件

所以我一直在玩弄多重处理，我想提高我的知识水平，在这里我可以从文本文件中读到进程1的第一句话，然后是进程2的第二句话，等等

txt file:

helloworld@world.com
helloworld2@world.com
helloworld3@world.com
helloworld4@world.com
helloworld5@world.com

这就是代码的外观：

def info(thread):
    global prod
    prod = int(thread) + 1
    runit()


def runit():

    log("Profile-" + str(prod) + Fore.GREEN + ' - ' + email)
    #From here I can then use the email for each worker basically. Or thats the plan atleast. Theplan is that every worker will have its own email that can be used in here.
    sys.exit()

def main():
    user_input = 0
    while True:
        try:
            user_input = int(input(Fore.WHITE + 'How many tasks do you wanna run? [NUMBERS] \n' + Fore.RESET))
        except ValueError:
            print(Fore.RED + "Stop being stupid" + Fore.RESET)
            continue
        else:
            with open('email.txt') as f:
                content = f.readlines()
            content = [x.strip('\n') for x in content]

            try:
                for i, email in enumerate(content):
                    print(email)

            except ValueError as e:
                print(e)

            HowManyThread = user_input
            i = 0
            jobs = []
            for i in range(HowManyThread):
                p = multiprocessing.Process(target=info, args=(str(i),))
                jobs.append(p)
                time.sleep(.5)
                p.start()

            for p in jobs:
                p.join()

            sys.exit()

日志基本上只是一条日志消息，没有什么特别的

---

Fore.COLOR以下方法将多处理委托给一个工作池，每个工作池接收一个索引块，并一次处理一行这些索引这里poolsize=8和chunksize=5的选择是任意的，可以根据您的要求进行调整

然后将所有工作人员的结果收集到最终列表中。请注意，imap_unordered仅适用于不关心行处理顺序的情况，即结果_列表不保持内容的原始顺序

---

您需要的是一个工作进程池——即使对于您的用例，我真的想知道线程或multiprocessing.dummy是否足够

池启动所需数量的工作进程，您可以将异步任务提交给池，该池将由第一个空闲工作进程处理

示例的精简版本—无需花哨的打印，无需对列表中的顺序文件进行不必要的读取—可能是：

import multiprocessing
import time

def runit(prod, email):

    print("Profile-" + str(prod) + ' - ' + email)
    #From here I can then use the email for each worker basically. Or thats the plan atleast. Theplan is that every worker will have its own email that can be used in here.
    # sys.exit() # NEVER CALL EXPLICITELY sys.exit() in a worker process
    time.sleep(1) # to add a delay inside each task

def main():
    while True:
        try:
            HowManyThread = int(input(
                'How many tasks do you wanna run? [NUMBERS] \n'))
        except ValueError:
            print("Stop being stupid")
            continue

        if HowManyThread == 0: break

        pool = multiprocessing.Pool(HowManyThread)
        with open('email.txt') as f:
            for i, email in enumerate(f):
                email = email.strip()
                # runit will be runned by a worker process
                pool.apply_async(runit, (i, email))

        pool.close()         # no more task to add
        pool.join()          # wait for last worker to end

if __name__ == "__main__":
    main()

---

您可以简单地向每个进程传递一行，如args=content[i]，然后按照文档的建议运行一个池。大多数情况下，跨进程内存访问是不必要的。你说的运行池是什么意思？这是嗯的第一段。我很不明白，我的意思是，与我的相比，有什么大的区别？哦，这对我来说似乎太新了，所有的事情，这对我来说都很难理解，因为它对我来说是全新的。。。但是，我的意思是，如果我们从您的代码开始，并且正如您在我的代码中看到的，我的代码中有p=multiprocessing.Processtarget=main，args=stri，其中target=main意味着它将与每个工作进程一起在主进程内运行，对吗。在这种情况下，我将如何执行main？在这里，我假设您将委托每个worker运行main函数，而您在代码中没有提供这些函数。上面只是假设一个从您的内容派生的行列表，以及您定义的一个函数worker_函数，该函数对一行内容进行操作，并执行您需要的任何操作。您可以将主函数的主体调整到此处。你最初的工作是做什么的？我已经升级了我的脚本，现在可能更容易看到了。我补充说。基本上，如果你检查runit方法中的代码，我希望每个员工都有自己的电子邮件，我可以在那里使用它@Twolffpiggottool，感谢您的更新。所以我称之为worker_函数基本上就是你的runit函数。它将从全球行列表中的电子邮件中一次接收一封电子邮件。然后结果列表包含所有工作人员发送的每封电子邮件的处理结果。你介意更新你的代码以便我知道我在寻找什么吗？因为我现在很困惑，我会说哈哈！

from multiprocessing import pool, Process, Queue
from tqdm import tqdm

with open('email.txt') as f:
    content = f.readlines()

global email_list
email_list = [x.strip('\n') for x in content]


def info(thread):
    global prod
    prod = int(thread) + 1
    runit()


def runit(email_index):
    email = email_list[email_index]

    log("Profile-" + str(prod) + Fore.GREEN + ' - ' + email)
    sys.exit()



def main():
    wipe()
    text()
    Infotext = "First name : Last name : Email: : Random char + Street"
    with open('data.json', 'w') as f:
        json.dump(Infotext, f)
        f.write("\n")

    with Pool(8) as pool:
        result_list = list(tqdm(pool.imap_unordered(, range(len(email_list)), chunksize=5), total=len(email_list))))


if __name__ == '__main__':
    try:
        main()

    except Exception as e:
        print(e)
        print(traceback.print_exc())
        print(traceback)

from multiprocessing import Pool
# progress bar to track your multiproc
from tqdm import tqdm

with open('email.txt') as f:
    content = f.readlines()

# this list will be accessed by each worker
global email_list
email_list = [x.strip('\n') for x in content]

# define function that worker will apply to each email
# it gets sent an index for the list of emails
# it accesses the email at that index, performs its function and returns
def runit(email_index):
    email = email_list[email_index]
    # do the stuff you're interested in for a single email

# run the multiprocessing to get your results
# this sends the indexes for the emails out to the workers
# and collects the results of runit into result list
with Pool(8) as pool:                                                
  result_list = list(tqdm(pool.imap_unordered(runit,    
                          range(len(email_list)), chunksize=5),                 
                          total=len(email_list)))

import multiprocessing
import time

def runit(prod, email):

    print("Profile-" + str(prod) + ' - ' + email)
    #From here I can then use the email for each worker basically. Or thats the plan atleast. Theplan is that every worker will have its own email that can be used in here.
    # sys.exit() # NEVER CALL EXPLICITELY sys.exit() in a worker process
    time.sleep(1) # to add a delay inside each task

def main():
    while True:
        try:
            HowManyThread = int(input(
                'How many tasks do you wanna run? [NUMBERS] \n'))
        except ValueError:
            print("Stop being stupid")
            continue

        if HowManyThread == 0: break

        pool = multiprocessing.Pool(HowManyThread)
        with open('email.txt') as f:
            for i, email in enumerate(f):
                email = email.strip()
                # runit will be runned by a worker process
                pool.apply_async(runit, (i, email))

        pool.close()         # no more task to add
        pool.join()          # wait for last worker to end

if __name__ == "__main__":
    main()