Java:ExecutorService，在达到一定队列大小后阻止提交

我正在尝试编写一个解决方案，其中单个线程生成可以并行执行的I/O密集型任务。每个任务都有重要的内存数据。因此，我希望能够限制当前挂起的任务数量

如果我像这样创建ThreadPoolExecutor：

    ThreadPoolExecutor executor = new ThreadPoolExecutor(numWorkerThreads, numWorkerThreads,
                                  0L, TimeUnit.MILLISECONDS,
                                  new LinkedBlockingQueue<Runnable>(maxQueue));

它在某种程度上达到了我想要达到的效果，但方式不雅观（基本上被拒绝的线程在调用线程中运行，因此这会阻止调用线程提交更多）

编辑：（提问5年后）

---

对于阅读此问题及其答案的任何人，请不要将已接受的答案视为一个正确答案。请通读所有答案和评论。

我认为这就像使用

ArrayBlockingQueue

而不是

LinkedBlockingQueue

一样简单

别理我。。。那是完全错误的

ThreadPoolExecutor

calls

Queue#offer

not

put

，这将产生您需要的效果

您可以扩展

ThreadPoolExecutor

，并提供

execute（Runnable）

的实现，该实现调用

put

代替

offer

---

恐怕这不是一个完全令人满意的答案。

我也做过同样的事情。诀窍是创建一个BlockingQueue，其中offer（）方法实际上是put（）。（您可以使用所需的任何基本BlockingQueue impl）

公共类LimitedQueue扩展LinkedBlockingQueue
{
public LimitedQueue（int-maxSize）
{
超级（最大尺寸）；
}
@凌驾
公共布尔报价（E）
{
//将offer（）和add（）转换为阻塞调用（除非中断）
试一试{
付诸表决（e）；
返回true；
}捕获（中断异常ie）{
Thread.currentThread（）.interrupt（）；
}
返回false；
}
}

---

请注意，这仅适用于

corePoolSize==maxPoolSize

的线程池，因此请注意（请参见注释）。

我也有类似的问题，我使用

ThreadPoolExecutor

中的

beforeExecute/afterExecute

挂钩实现了这一点：

import java.util.concurrent.BlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.Condition;
import java.util.concurrent.locks.ReentrantLock;

/**
 * Blocks current task execution if there is not enough resources for it.
 * Maximum task count usage controlled by maxTaskCount property.
 */
public class BlockingThreadPoolExecutor extends ThreadPoolExecutor {

    private final ReentrantLock taskLock = new ReentrantLock();
    private final Condition unpaused = taskLock.newCondition();
    private final int maxTaskCount;

    private volatile int currentTaskCount;

    public BlockingThreadPoolExecutor(int corePoolSize, int maximumPoolSize,
            long keepAliveTime, TimeUnit unit,
            BlockingQueue<Runnable> workQueue, int maxTaskCount) {
        super(corePoolSize, maximumPoolSize, keepAliveTime, unit, workQueue);
        this.maxTaskCount = maxTaskCount;
    }

    /**
     * Executes task if there is enough system resources for it. Otherwise
     * waits.
     */
    @Override
    protected void beforeExecute(Thread t, Runnable r) {
        super.beforeExecute(t, r);
        taskLock.lock();
        try {
            // Spin while we will not have enough capacity for this job
            while (maxTaskCount < currentTaskCount) {
                try {
                    unpaused.await();
                } catch (InterruptedException e) {
                    t.interrupt();
                }
            }
            currentTaskCount++;
        } finally {
            taskLock.unlock();
        }
    }

    /**
     * Signalling that one more task is welcome
     */
    @Override
    protected void afterExecute(Runnable r, Throwable t) {
        super.afterExecute(r, t);
        taskLock.lock();
        try {
            currentTaskCount--;
            unpaused.signalAll();
        } finally {
            taskLock.unlock();
        }
    }
}

import java.util.concurrent.BlockingQueue；
导入java.util.concurrent.ThreadPoolExecutor；
导入java.util.concurrent.TimeUnit；
导入java.util.concurrent.locks.Condition；
导入java.util.concurrent.locks.ReentrantLock；
/**
*如果没有足够的资源执行当前任务，则阻止该任务的执行。
*maxTaskCount属性控制的最大任务计数使用率。
*/
公共类BlockingThreadPoolExecutor扩展ThreadPoolExecutor{
private final ReentrantLock taskLock=new ReentrantLock（）；
private final Condition unpaused=taskLock.newCondition（）；
私有最终整数maxTaskCount；
私有volatile int currentTaskCount；
public BlockingThreadPoolExecutor（int-corePoolSize，int-maximumPoolSize，
长keepAliveTime，时间单位，
BlockingQueue（工作队列，int maxTaskCount）{
super（corePoolSize、maximumPoolSize、keepAliveTime、unit、workQueue）；
this.maxTaskCount=maxTaskCount；
}
/**
*如果有足够的系统资源，则执行任务。否则
*等待。
*/
@凌驾
执行前受保护的void（线程t，可运行r）{
超级执行前（t，r）；
taskLock.lock（）；
试一试{
//旋转，而我们将没有足够的能力完成此工作
同时（maxTaskCount

这对你来说应该足够好了。顺便说一句，最初的实现是基于任务大小的，因为一个任务可能比另一个任务大100倍，提交两个大任务会让整个系统崩溃，但是运行一个大任务和大量的小任务是可以的。如果I/O密集型任务的大小大致相同，则可以使用该类，否则请告诉我，我将发布基于大小的实现

---

另外，您需要检查

ThreadPoolExecutor

javadoc。Doug Lea提供的关于如何轻松定制的用户指南非常好。

我知道这是一个老问题，但有一个类似的问题，即创建新任务非常快，如果有太多OutOfMemory错误发生，因为现有任务完成得不够快

在我的例子中，

Callables

已提交，我需要结果，因此我需要存储由

executor.submit（）返回的所有
Futures
。我的解决方案是将
期货
放入一个最大大小的
阻塞队列
。一旦队列已满，在完成某些任务（从队列中删除元素）之前，不会生成更多的任务。在伪代码中：

final ExecutorService executor = Executors.newFixedThreadPool(numWorkerThreads);
final LinkedBlockingQueue<Future> futures = new LinkedBlockingQueue<>(maxQueueSize);
try {   
    Thread taskGenerator = new Thread() {
        @Override
        public void run() {
            while (reader.hasNext) {
                Callable task = generateTask(reader.next());
                Future future = executor.submit(task);
                try {
                    // if queue is full blocks until a task
                    // is completed and hence no future tasks are submitted.
                    futures.put(future);
                } catch (InterruptedException ex) {
                    Thread.currentThread().interrupt();         
                }
            }
        executor.shutdown();
        }
    }
    taskGenerator.start();
    
    // read from queue as long as task are being generated
    // or while Queue has elements in it
    while (taskGenerator.isAlive()
                    || !futures.isEmpty()) {
        Future future = futures.take();
        // do something
    }
} catch (InterruptedException ex) {
    Thread.currentThread().interrupt();     
} catch (ExecutionException ex) {
    throw new MyException(ex);
} finally {
    executor.shutdownNow();
}

final executor服务executor=Executors.newFixedThreadPool（numWorkerThreads）；
最终LinkedBlockingQueue期货=新LinkedBlockingQueue（maxQueueSize）；
试试{
线程任务生成器=新线程（）{
@凌驾
公开募捐{
while（reader.hasNext）{
Callable task=generateTask（reader.next（））；
未来=执行者提交（任务）；
试一试{
//如果队列已满，则阻塞直到任务完成
//已完成，因此不会提交未来的任务。
期货。看跌期权（fu）
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.Condition;
import java.util.concurrent.locks.ReentrantLock;

/**
 * Blocks current task execution if there is not enough resources for it.
 * Maximum task count usage controlled by maxTaskCount property.
 */
public class BlockingThreadPoolExecutor extends ThreadPoolExecutor {

    private final ReentrantLock taskLock = new ReentrantLock();
    private final Condition unpaused = taskLock.newCondition();
    private final int maxTaskCount;

    private volatile int currentTaskCount;

    public BlockingThreadPoolExecutor(int corePoolSize, int maximumPoolSize,
            long keepAliveTime, TimeUnit unit,
            BlockingQueue<Runnable> workQueue, int maxTaskCount) {
        super(corePoolSize, maximumPoolSize, keepAliveTime, unit, workQueue);
        this.maxTaskCount = maxTaskCount;
    }

    /**
     * Executes task if there is enough system resources for it. Otherwise
     * waits.
     */
    @Override
    protected void beforeExecute(Thread t, Runnable r) {
        super.beforeExecute(t, r);
        taskLock.lock();
        try {
            // Spin while we will not have enough capacity for this job
            while (maxTaskCount < currentTaskCount) {
                try {
                    unpaused.await();
                } catch (InterruptedException e) {
                    t.interrupt();
                }
            }
            currentTaskCount++;
        } finally {
            taskLock.unlock();
        }
    }

    /**
     * Signalling that one more task is welcome
     */
    @Override
    protected void afterExecute(Runnable r, Throwable t) {
        super.afterExecute(r, t);
        taskLock.lock();
        try {
            currentTaskCount--;
            unpaused.signalAll();
        } finally {
            taskLock.unlock();
        }
    }
}

final ExecutorService executor = Executors.newFixedThreadPool(numWorkerThreads);
final LinkedBlockingQueue<Future> futures = new LinkedBlockingQueue<>(maxQueueSize);
try {   
    Thread taskGenerator = new Thread() {
        @Override
        public void run() {
            while (reader.hasNext) {
                Callable task = generateTask(reader.next());
                Future future = executor.submit(task);
                try {
                    // if queue is full blocks until a task
                    // is completed and hence no future tasks are submitted.
                    futures.put(future);
                } catch (InterruptedException ex) {
                    Thread.currentThread().interrupt();         
                }
            }
        executor.shutdown();
        }
    }
    taskGenerator.start();
    
    // read from queue as long as task are being generated
    // or while Queue has elements in it
    while (taskGenerator.isAlive()
                    || !futures.isEmpty()) {
        Future future = futures.take();
        // do something
    }
} catch (InterruptedException ex) {
    Thread.currentThread().interrupt();     
} catch (ExecutionException ex) {
    throw new MyException(ex);
} finally {
    executor.shutdownNow();
}

public class BoundedExecutor extends ThreadPoolExecutor{

    private final Semaphore semaphore;

    public BoundedExecutor(int bound) {
        super(bound, Integer.MAX_VALUE, 60L, TimeUnit.SECONDS, new SynchronousQueue<Runnable>());
        semaphore = new Semaphore(bound);
    }

    /**Submits task to execution pool, but blocks while number of running threads 
     * has reached the bound limit
     */
    public <T> Future<T> submitButBlockIfFull(final Callable<T> task) throws InterruptedException{

        semaphore.acquire();            
        return submit(task);                    
    }


    @Override
    protected void afterExecute(Runnable r, Throwable t) {
        super.afterExecute(r, t);

        semaphore.release();
    }
}

    if (isRunning(c) && workQueue.offer(command)) {
        int recheck = ctl.get();
        if (! isRunning(recheck) && remove(command))
            reject(command);
        else if (workerCountOf(recheck) == 0)
            addWorker(null, false);
    }
    else if (!addWorker(command, false))
        reject(command);

public void rejectedExecution(Runnable r, ThreadPoolExecutor executor) {
    try {
        if (!executor.isShutdown()) {
            executor.getQueue().put(r);
        }
    } catch (InterruptedException e) {
        Thread.currentThread().interrupt();
        throw new RejectedExecutionException("Executor was interrupted while the task was waiting to put on work queue", e);
    }
}

executor.setRejectedExecutionHandler(new CallerBlocksPolicy());