限制bash脚本中并发进程的数量

我正在尝试使用bash脚本并行执行多个任务。作业是内存密集型的，所以我需要控制每次启动的数量。下面是我的内容，它基本上可以工作，但有时延迟循环不知道刚刚启动的作业，因此会启动几个额外的作业，导致系统内存不足

在延迟循环中的while语句之前添加sleep可以减少这个问题，但不能完全消除它。任何人都知道一种治疗方法。如果相关的话，我正在Solaris上运行

#!/bin/bash
delay(){
while [ 8 -le $(ps -ef |grep  myjob |wc -l) ]
do
sleep 1
done
}

./myjob -params1 &
delay
./myjob -params2 &
delay
./myjob -params3 &
delay
./myjob -params4 &
delay
.
.
.

---

使用

xargs

执行此操作。将其传递给

-n1

以指示每个作业的一个参数，并使用

--max jobs

参数来指定并发进程的数量。

GNU

parallel

实用程序可能是正确的工具，因为可以说它比xargs

根据

makefile

制定脚本，并让

make-jN

把它分类。

首先，我将给你一个简单的例子，说明我在几个linux脚本中所做的事情。这个应该可以在solaris上运行，但我目前没有任何系统可供测试。我修改了一些使用/proc的东西，所以如果有任何东西不起作用，请告诉我

#!/bin/bash

# set the max # of threads
max_threads=4
# set the max system load
max_load=4

print_jobs(){
# flush finished jobs messages
  jobs > /dev/null
  for x in $(jobs -p) ; do
   # print all jobs
    echo "$x"
  done
}

job_count(){
  cnt=$(print_jobs $1)
  if [ -n "$cnt" ]; then
    wc -l <<< "$cnt"
  else
    echo 0
  fi
}

cur_load(){
  # get the 1 minute load average integer
  uptime |sed 's/.*load average[s]*:[[:space:]]*\([^.]*\)\..*/\1/g'
}


main_function(){
 # get current job count and load
  jcnow=$(job_count)
  loadnow=$(cur_load)

 # first, enter a loop waiting for load/threads to be below thresholds
  while [ $loadnow -ge $max_load ] || [ $jcnow -ge $max_threads ]; do
    if ! [ $firstout ]; then
      echo "entering sleep loop. load: $loadnow, threads: $jcnow"
      st=$(date +%s)
      local firstout=true
    else
      now=$(date +%s)
     # if it's been 5 minutes, echo again:
      if [ $(($now - $st)) -ge 300 ]; then
        echo "still sleeping. load: $loadnow, threads: $jcnow"
        st=$(date +%s)
      fi
    fi
    sleep 5s

   # refresh these variables for loop
    loadnow=$(cur_load)
    jcnow=$(job_count)
  unset firstout
  done

  ( ./myjob $@ ) &
}

# do some actual work
for jobparams in "params1" "params2" "params3" "params4" "params5" "params6" "params7" ; do
   main_function $jobparams
done

wait

可能重复的

#!/bin/bash

# set the max # of threads
max_threads=4

print_jobs(){
# flush finished jobs messages
  jobs > /dev/null
  for x in $(jobs -p) ; do
   # print all jobs
    echo "$x"
  done
}

job_count(){
  cnt=$(print_jobs $1)
  if [ -n "$cnt" ]; then
    wc -l <<< "$cnt"
  else
    echo 0
  fi
}

main_function(){
 # get current job count
  jcnow=$(job_count)

 # first, enter a loop waiting for threads to be below thresholds
  while [ $jcnow -ge $max_threads ]; do
    if ! [ $firstout ]; then
      echo "entering sleep loop. threads: $jcnow"
      st=$(date +%s)
      local firstout=true
    else
      now=$(date +%s)
     # if it's been 5 minutes, echo again:
      if [ $(($now - $st)) -ge 300 ]; then
        echo "still sleeping. threads: $jcnow"
        st=$(date +%s)
      fi
    fi
    sleep 5s

   # refresh these variables for loop
    jcnow=$(job_count)
  unset firstout
  done


  ( ./myjob $@ ) &
}

# do some actual work
for jobparams in "params1" "params2" "params3" "params4" "params5" "params6" "params7" ; do
   main_function $jobparams
done

wait