Fatal编程技术网
  • multithreading/
  • Multithreading 批处理中用于并行和顺序工作的for和start命令

Multithreading 批处理中用于并行和顺序工作的for和start命令

multithreading batch-file

Multithreading 批处理中用于并行和顺序工作的for和start命令,multithreading,batch-file,Multithreading,Batch File,我有一个8GB内存的8核CPU，我正在创建一个批处理文件，以自动执行7-zip CLI，耗尽大部分参数和变量，压缩同一组文件，最终目标是找到参数和变量的最佳组合，从而使归档大小尽可能小 这在本质上非常耗时，尤其是当要处理的文件集以GB为单位时。我不仅需要一种自动化的方法，还需要一种加速整个过程的方法 7-zip使用不同的压缩算法，有些是单线程的，有些是多线程的，有些不需要太多内存，有些需要大量内存，甚至可能超过8GB的限制。我已经成功地创建了一个自动批处理，它按顺序工作，排除了需要超过8GB内存

我有一个8GB内存的8核CPU，我正在创建一个批处理文件，以自动执行7-zip CLI，耗尽大部分参数和变量，压缩同一组文件，最终目标是找到参数和变量的最佳组合，从而使归档大小尽可能小

这在本质上非常耗时，尤其是当要处理的文件集以GB为单位时。我不仅需要一种自动化的方法，还需要一种加速整个过程的方法

7-zip使用不同的压缩算法，有些是单线程的，有些是多线程的，有些不需要太多内存，有些需要大量内存，甚至可能超过8GB的限制。我已经成功地创建了一个自动批处理，它按顺序工作，排除了需要超过8GB内存的组合

我将不同的压缩算法分为几个批次，以简化整个过程。例如，PPMd中作为7z归档文件的压缩使用1个线程，最高可达1024MB。这是我的当前批次：

@echo off
echo mem=1m 2m 3m 4m 6m 8m 12m 16m 24m 32m 48m 64m 96m 128m 192m 256m 384m 512m 768m 1024m
echo o=2 3 4 5 6 7 8 10 12 14 16 20 24 28 32
echo s=off 1m 2m 4m 8m 16m 32m 64m 128m 256m 512m 1g 2g 4g 8g 16g 32g 64g on
echo x=1 3 5 7 9

for %%x IN (9) DO for %%d IN (1024m 768m 512m 384m 256m 192m 128m 96m 64m 48m 32m 24m 16m 12m 8m 6m 4m 3m 2m 1m) DO for %%w IN (32 28 24 20 16 14 12 10 8 7 6 5 4 3 2) DO for %%s IN (on) DO 7z.exe a teste.resultado\%%xx.ppmd.%%dd.%%ww.%%ss.7z .\teste.original\* -mx=%%x -m0=PPMd:mem=%%d:o=%%w -ms=%%s
exit

x
、
s
、
o
和
mem
是参数，它们后面的是7z.exe将使用的变量
x
和
s
在这种情况下不需要考虑，它们表示存档的压缩强度和实体块大小

该批处理可以正常工作，但一次只能运行一个7z.exe实例，现在我正在寻找一种方法，使其并行运行更多7z.exe实例，但一次不超过8GB的RAM或8个线程，以先到者为准，然后再继续执行序列中的下一个

我该如何改进这一点？我有一些想法，但我不知道如何使它们成批工作。我正在考虑另外两个变量，它们不会与7z进程交互，但会控制下一个7z实例何时启动。一个变量将跟踪当前正在使用的线程数量，另一个变量将跟踪正在使用的内存数量。那行吗

编辑： 很抱歉，我需要添加详细信息，我对这种发布方式不熟悉。根据这个答案--我提到创建了8个批次，其中7z.PPMd批次就是其中之一。也许列出所有批次以及7z如何处理参数可以更好地了解整个问题。我将从简单的问题开始：

  • 7z.PPMd-1根据每个实例的32m-1055m内存使用情况，充分利用线程和字典
  • 7z.BZip2-8个充分利用的线程和固定的每个实例109m内存使用率
  • zip.Bzip2-8个部分使用的线程和固定的每个实例336m内存使用率
  • Deflate-8个部分使用的线程和固定的每个实例的260m内存使用率
  • zip.PPMd-8部分使用线程和字典，每个实例的内存使用量取决于280m-2320m
对于部分使用的线程，我的意思是，虽然我为每个7.exe实例分配了8个线程，但该算法可以以随机的方式使用不同的CPU，超出我的控制，不可预测，但限制设置在那里-不超过8个线程。对于8个充分利用的线程，这意味着在我的8核CPU上，每个实例都在利用100%的CPU

最复杂的一个-7z.LZMA，7z.LZMA2，zip.LZMA-将需要详细解释，但我现在时间不够。只要我有更多的空闲时间，我就会回来编辑LZMA部分

再次感谢

编辑：添加LZMA零件

  • 7z.LZMA-每个实例都是n线程的，范围从1到2：

    • 1个充分利用的线程，依赖于字典，64k到512m：
      • 64k字典使用32m内存
      • 512m字典使用5407m内存
      • 排除范围：768m到1024m（超过可用内存8192m的限制）
    • 2个部分使用的线程，依赖于字典，64k到512m：
      • 64k字典使用38m内存
      • 512m字典使用5413m内存
      • 排除范围：768m到1024m（超过可用内存8192m的限制）
  • 7z.LZMA2-每个实例都是n线程的，范围从1到8：

    • 1个充分利用的线程，依赖于字典，64k到512m：
      • 64k字典使用32m内存
      • 512m字典使用5407m内存
      • 排除范围：768m到1024m（超过可用内存8192m的限制）
    • 2或3个部分使用的线程，取决于字典，64k到512m：
      • 64k字典使用38m内存
      • 512m字典使用5413m内存
      • 排除范围：768m到1024m（超过可用内存8192m的限制）
    • 4或5个部分使用的线程，取决于字典，64k到256m：
      • 64k字典使用51m内存
      • 256m字典使用5677m内存
      • 排除范围：384m到1024m（超过可用内存8192m的限制）
    • 6或7个部分使用的线程，依赖于字典，64k到192m：
      • 64k字典使用62m内存
      • 192m字典使用6965m内存
      • 排除范围：256m到1024m（超过可用内存8192m的限制）
    • 8个部分使用的线程，依赖于字典，64k到128m：
      • 64k字典使用72m内存
      • 128m字典使用6717m内存
    • 排除范围：192m到1024m（超过可用内存8192m的限制）
  • zip.LZMA-每个实例都是n线程的，范围从1到8：

    • 1个充分利用的线程，依赖于字典，64k到512m：
      • 64k字典使用3m内存
      • 512m字典使用5378m内存
      • 排除范围：768m到1024m（超过可用内存8192m的限制）
    • 2或3部分uti 
          2>nul del %lock%!nextProc!
    %= Redirect the lock handle to the lock file. The CMD process will     =%
    %= maintain an exclusive lock on the lock file until the process ends. =%
    start /b "" cmd /c %lockHandle%^>"%lock%!nextProc!" 2^>^&1 !cpu%%N! !cmd!
  )
  set "launch="

          ) 9>>"%lock%%%N"
  ) 2>nul
  if %endCount% lss %startCount% (
    1>nul 2>nul ping /n 2 ::1
    goto :wait
  )

2>nul del %lock%*

      start /b /low cmd /c !cmd!>"%lock%!nextProc!"

      echo 8 threads - maxproc=1
for %%x IN (9) DO for %%t IN (8) DO for %%d IN (900k) DO for %%s IN (on) DO 7z.exe a teste.resultado\%%xx.bzip2.%%tt.%%dd.%%ss.7z .\teste.original\* -mx=%%x -ms=%%s -m0=BZip2:d=%%d:mt=%%t
for %%x IN (9) DO for %%t IN (8) DO for %%d IN (900k) DO 7z.exe a teste.resultado\%%xx.bzip2.%%tt.%%dd.zip .\teste.original\* -mx=%%x -mm=BZip2:d=%%d -mmt=%%t
for %%x IN (9) DO for %%t IN (8) DO for %%w IN (257 256 192 128 96 64 48 32 24 16 12 8) DO 7z.exe a teste.resultado\%%xx.deflate64.%%tt.%%ww.zip .\teste.original\* -mx=%%x -mm=deflate64:fb=%%w -mmt=%%t
for %%x IN (9) DO for %%t IN (8) DO for %%w IN (258 256 192 128 96 64 48 32 24 16 12 8) DO 7z.exe a teste.resultado\%%xx.deflate.%%tt.%%ww.zip .\teste.original\* -mx=%%x -mm=deflate:fb=%%w -mmt=%%t
for %%x IN (9) DO for %%t IN (8) DO for %%d IN (256m 128m 64m 32m 16m 8m 4m 2m 1m) DO for %%w IN (16 15 14 13 12 11 10 9 8 7 6 5 4 3 2) DO 7z.exe a teste.resultado\%%xx.ppmd.%%tt.%%dd.%%ww.zip .\teste.original\* -mx=%%x -mm=PPMd:mem=%%d:o=%%w -mmt=%%t

echo 4 threads - maxproc=2
for %%x IN (9) DO for %%t IN (4) DO for %%d IN (256m) DO for %%w IN (273 256 192 128 96 64 48 32 24 16 12 8) DO for %%s IN (on) DO 7z.exe a teste.resultado\%%xx.lzma2.%%tt.%%dd.%%ww.%%ss.7z .\teste.original\* -mx=%%x -ms=%%s -m0=lzma2:d=%%d:fb=%%w -mmt=%%t

echo 2 threads - maxproc=4
for %%x IN (9) DO for %%t IN (2) DO for %%d IN (512m) DO for %%w IN (273 256 192 128 96 64 48 32 24 16 12 8) DO for %%s IN (on) DO 7z.exe a teste.resultado\%%xx.lzma.%%tt.%%dd.%%ww.%%ss.7z .\teste.original\* -mx=%%x -ms=%%s -m0=LZMA:d=%%d:fb=%%w -mmt=%%t
for %%x IN (9) DO for %%t IN (2) DO for %%d IN (512m) DO for %%w IN (273 256 192 128 96 64 48 32 24 16 12 8) DO for %%s IN (on) DO 7z.exe a teste.resultado\%%xx.lzma2.%%tt.%%dd.%%ww.%%ss.7z .\teste.original\* -mx=%%x -ms=%%s -m0=lzma2:d=%%d:fb=%%w -mmt=%%t
for %%x IN (9) DO for %%t IN (2) DO for %%d IN (512m) DO for %%w IN (273 256 192 128 96 64 48 32 24 16 12 8) DO 7z.exe a teste.resultado\%%xx.lzma.%%tt.%%dd.%%ww.zip .\teste.original\* -mx=%%x -mm=lzma:d=%%d:fb=%%w -mmt=%%t

echo 1 threads - maxproc=8
for %%x IN (9) DO for %%t IN (1) DO for %%d IN (512m) DO for %%w IN (273 256 192 128 96 64 48 32 24 16 12 8) DO for %%s IN (on) DO 7z.exe a teste.resultado\%%xx.lzma.%%tt.%%dd.%%ww.%%ss.7z .\teste.original\* -mx=%%x -ms=%%s -m0=LZMA:d=%%d:fb=%%w -mmt=%%t
for %%x IN (9) DO for %%t IN (1) DO for %%d IN (512m) DO for %%w IN (273 256 192 128 96 64 48 32 24 16 12 8) DO for %%s IN (on) DO 7z.exe a teste.resultado\%%xx.lzma2.%%tt.%%dd.%%ww.%%ss.7z .\teste.original\* -mx=%%x -ms=%%s -m0=lzma2:d=%%d:fb=%%w -mmt=%%t
for %%x IN (9) DO for %%d IN (1024m 768m 512m 384m 256m 192m 128m 96m 64m 48m 32m 24m 16m 12m 8m 6m 4m 3m 2m 1m) DO for %%w IN (32 28 24 20 16 14 12 10 8 7 6 5 4 3 2) DO for %%s IN (on) DO 7z.exe a teste.resultado\%%xx.ppmd.%%dd.%%ww.%%ss.7z .\teste.original\* -mx=%%x -m0=PPMd:mem=%%d:o=%%w -ms=%%s
for %%x IN (9) DO for %%t IN (1) DO for %%d IN (512m) DO for %%w IN (273 256 192 128 96 64 48 32 24 16 12 8) DO 7z.exe a teste.resultado\%%xx.lzma.%%tt.%%dd.%%ww.zip .\teste.original\* -mx=%%x -mm=lzma:d=%%d:fb=%%w -mmt=%%t

      @echo off
setlocal enableDelayedExpansion

set "maxMem=8192"
set "maxThreads=8"

:cycle1
set "cycleCount=4"
set "cycleThreads=1"
set "maxProc="
set /a "maxProc=maxThreads/cycleThreads"
set "cycleFor1=for %%x IN (9) DO for %%t IN (1) DO for %%d IN (512m) DO for %%w IN (273 256 192 128 96 64 48 32 24 16 12 8) DO for %%s IN (on) DO ("
set "cycleFor2=for %%x IN (9) DO for %%t IN (1) DO for %%d IN (512m) DO for %%w IN (273 256 192 128 96 64 48 32 24 16 12 8) DO for %%s IN (on) DO ("
set "cycleFor3=for %%x IN (9) DO for %%d IN (1024m 768m 512m 384m 256m 192m 128m 96m 64m 48m 32m 24m 16m 12m 8m 6m 4m 3m 2m 1m) DO for %%w IN (32 28 24 20 16 14 12 10 8 7 6 5 4 3 2) DO for %%s IN (on) DO ("
set "cycleFor4=for %%x IN (9) DO for %%t IN (1) DO for %%d IN (512m) DO for %%w IN (273 256 192 128 96 64 48 32 24 16 12 8) DO ("
set "cycleCmd1=7z.exe a teste.resultado\%%xx.lzma.%%tt.%%dd.%%ww.%%ss.7z .\teste.original\* -mx=%%x -ms=%%s -m0=LZMA:d=%%d:fb=%%w -mmt=%%t"
set "cycleCmd2=7z.exe a teste.resultado\%%xx.lzma2.%%tt.%%dd.%%ww.%%ss.7z .\teste.original\* -mx=%%x -ms=%%s -m0=lzma2:d=%%d:fb=%%w -mmt=%%t"
set "cycleCmd3=7z.exe a teste.resultado\%%xx.ppmd.%%dd.%%ww.%%ss.7z .\teste.original\* -mx=%%x -m0=PPMd:mem=%%d:o=%%w -ms=%%s"
set "cycleCmd4=7z.exe a teste.resultado\%%xx.lzma.%%tt.%%dd.%%ww.zip .\teste.original\* -mx=%%x -mm=lzma:d=%%d:fb=%%w -mmt=%%t"
set "tempMem1=5407"
set "tempMem2=5407"
set "tempMem3=1055"
set "tempMem4=5378"
rem set "tempMem1=5407"
rem set "tempMem2=5407"
rem set "tempMem3=1055 799 543 415 287 223 159 127 95 79 63 55 47 43 39 37 35 34 33 32"
rem set "tempMem4=5378"
set "memSum=0"
if not defined memRem set "memRem=!maxMem!"

for /l %%N in (1 1 %cycleCount%) DO (set "tempProc%%N=")
for /l %%N in (1 1 %cycleCount%) DO (
  set memRem
  set /a "tempProc%%N=%memRem%/tempMem%%N"
  set /a "memSum+=tempMem%%N"
  set /a "memRem-=tempMem%%N"
  set /a "maxProc=!tempProc%%N!"
  call :executeCycle
  set /a "memRem+=tempMem%%N"
  set /a "memSum-=tempMem%%N"
  set /a "maxProc-=!tempProc%%!
)
goto :fim

:executeCycle
set "lock=lock_%random%_"
set /a "startCount=0, endCount=0"
for /l %%N in (1 1 %maxProc%) DO set "endProc%%N="

    set launch=1
    for %%x IN (9) DO for %%t IN (1) DO for %%d IN (512m) DO for %%w IN (273 256 192 128 96 64 48 32 24 16 12 8) DO for %%s IN (on) DO (
      set "cmd=7z.exe a teste.resultado\%%xx.lzma.%%tt.%%dd.%%ww.%%ss.7z .\teste.original\* -mx=%%x -ms=%%s -m0=LZMA:d=%%d:fb=%%w -mmt=%%t"
      if !startCount! lss %maxProc%  (
        set /a "startCount+=1, nextProc=startCount"
      ) else (
        call :wait
      )
      set cmd!nextProc!=!cmd!
      echo !time! - proc!nextProc!: starting !cmd!
      2>nul del %lock%!nextProc!
      start /b /low cmd /c !cmd!>"%lock%!nextProc!"
    )
    set "launch="
:wait
        for /l %%N in (1 1 %startCount%) do (
          if not defined endProc%%N if exist "%lock%%%N" (
            echo !time! - proc%%N: finished !cmd%%N!
            if defined launch (
              set nextProc=%%N
              exit /b
            )
            set /a "endCount+=1, endProc%%N=1"
          ) 9>>"%lock%%%N"
        ) 2>nul
        if %endCount% lss %startCount% (
        1>nul 2>nul ping /n 2 ::1
        goto :wait
      )

2>nul del %lock%*
echo ===
echo Thats all folks!
exit /b
:fim
pause

      for %%d IN (1024m 768m 512m 384m 256m 192m 128m 96m 64m 48m 32m 24m 16m 12m 8m 6m 4m 3m 2m 1m) DO
set "tempMem3=1055 799 543 415 287 223 159 127 95 79 63 55 47 43 39 37 35 34 33 32"

      @Echo OFF & Setlocal EnableDelayedExpansion

Set /A "pCount=0" & REm Process count

For
...
) DO (
     Set /A "pCount+=1"
     If !pCount! LEQ 8 (
       Start /B 7z.exe a teste.resultado\%%xx.ppmd.%%dd.%%ww.%%ss.7z .\teste.original\* -mx=%%x -m0=PPMd:mem=%%d:o=%%w -ms=%%s
     )
)
...

      CMD /C "Start /w 7z.exe a teste.resultado\%%xx.ppmd.%%dd.%%ww.%%ss.7z .\teste.original\* -mx=%%x -m0=PPMd:mem=%%d:o=%%w -ms=%%s"

      @echo off
setlocal enableDelayedExpansion

:: Display the output of each process if the /O option is used
:: else ignore the output of each process
if /i "%~1" equ "/O" (
  set "lockHandle=1"
  set "showOutput=1"
) else (
  set "lockHandle=1^>nul 9"
  set "showOutput="
)

:: Define the maximum number of parallel processes to run.
:: Each process number can optionally be assigned to a particular server
:: and/or cpu via psexec specs (untested).
set "maxProc=8"

:: Optional - Define CPU targets in terms of PSEXEC specs
::           (everything but the command)
::
:: If a cpu is not defined for a proc, then it will be run on the local machine.
:: I haven't tested this feature, but it seems like it should work.
::
:: set cpu1=psexec \\server1 ...
:: set cpu2=psexec \\server1 ...
:: set cpu3=psexec \\server2 ...
:: etc.

:: For this demo force all cpu specs to undefined (local machine)
for /l %%N in (1 1 %maxProc%) do set "cpu%%N="

:: Get a unique base lock name for this particular instantiation.
:: Incorporate a timestamp from WMIC if possible, but don't fail if
:: WMIC not available. Also incorporate a random number.
  set "lock="
  for /f "skip=1 delims=-+ " %%T in ('2^>nul wmic os get localdatetime') do (
    set "lock=%%T"
    goto :break
  )
  :break
  set "lock=%temp%\lock%lock%_%random%_"

:: Initialize the counters
  set /a "startCount=0, endCount=0"

:: Clear any existing end flags
  for /l %%N in (1 1 %maxProc%) do set "endProc%%N="

:: Launch the commands in a loop
  set launch=1
  echo mem=1m 2m 3m 4m 6m 8m 12m 16m 24m 32m 48m 64m 96m 128m 192m 256m 384m 512m 768m 1024m
  echo o=2 3 4 5 6 7 8 10 12 14 16 20 24 28 32
  echo s=off 1m 2m 4m 8m 16m 32m 64m 128m 256m 512m 1g 2g 4g 8g 16g 32g 64g on
  echo x=1 3 5 7 9
  for %%x IN (9) DO for %%d IN (1024m 768m 512m 384m 256m 192m 128m 96m 64m 48m 32m 24m 16m 12m 8m 6m 4m 3m 2m 1m) DO (
    set "cmd=7z.exe a teste.resultado\%%xx.ppmd.%%dd.%%ww.%%ss.7z .\teste.original\* -mx=%%x -m0=PPMd:mem=%%d:o=%%w -ms=%%s"
    if !startCount! lss %maxProc% (
      set /a "startCount+=1, nextProc=startCount"
    ) else (
      call :wait
    )
    set cmd!nextProc!=!cmd!
    if defined showOutput echo -------------------------------------------------------------------------------
    echo !time! - proc!nextProc!: starting !cmd!
    2>nul del %lock%!nextProc!
    %= Redirect the lock handle to the lock file. The CMD process will     =%
    %= maintain an exclusive lock on the lock file until the process ends. =%
    start /b "" cmd /c %lockHandle%^>"%lock%!nextProc!" 2^>^&1 !cpu%%N! !cmd!
  )
  set "launch="

:wait
:: Wait for procs to finish in a loop
:: If still launching then return as soon as a proc ends
:: else wait for all procs to finish
  :: redirect stderr to null to suppress any error message if redirection
  :: within the loop fails.
  for /l %%N in (1 1 %startCount%) do (
    %= Redirect an unused file handle to the lock file. If the process is    =%
    %= still running then redirection will fail and the IF body will not run =%
    if not defined endProc%%N if exist "%lock%%%N" (
      %= Made it inside the IF body so the process must have finished =%
      if defined showOutput echo ===============================================================================
      echo !time! - proc%%N: finished !cmd%%N!
      if defined showOutput type "%lock%%%N"
      if defined launch (
        set nextProc=%%N
        exit /b
      )
      set /a "endCount+=1, endProc%%N=1"
    ) 9>>"%lock%%%N"
  ) 2>nul
  if %endCount% lss %startCount% (
    1>nul 2>nul ping /n 2 ::1
    goto :wait
  )

2>nul del %lock%*
if defined showOutput echo ===============================================================================
echo Thats all folks!