Java completionservice：如何杀死所有线程并在5秒内返回结果？_Java_Multithreading_Concurrency

Java completionservice：如何杀死所有线程并在5秒内返回结果？

java multithreading concurrency

Java completionservice：如何杀死所有线程并在5秒内返回结果？,java,multithreading,concurrency,Java,Multithreading,Concurrency,我对CompletionService有些问题。我的任务：要并行解析大约300个html页面，我只需等待所有结果5秒钟，然后将结果返回主代码。我已经决定为此使用CompletionService+Callable。问题是如何停止CompletionService导致的所有线程，并从成功解析的页面返回结果？在这段代码中，删除了打印行，但我可以说5秒就足够了（有很好的结果，但程序等待所有线程完成）。我的代码执行了大约2分钟我的呼叫代码： Collection<Callable<H

我对CompletionService有些问题。我的任务：要并行解析大约300个html页面，我只需等待所有结果5秒钟，然后将结果返回主代码。我已经决定为此使用CompletionService+Callable。 问题是如何停止CompletionService导致的所有线程，并从成功解析的页面返回结果？在这段代码中，删除了打印行，但我可以说5秒就足够了（有很好的结果，但程序等待所有线程完成）。我的代码执行了大约2分钟

我的呼叫代码：

Collection<Callable<HCard>> solvers = new ArrayList<Callable<HCard>>();
for (final String currentUrl : allUrls) {
    solvers.add(new Callable<HCard>() {
        public HCard call() throws ParserException {
            HCard hCard = HCardParser.parseOne(currentUrl);                      
            if (hCard != null) {
                return hCard;
            } else {
                return null;
            }
        }
    });
}
ExecutorService execService = Executors.newCachedThreadPool();
Helper helper = new Helper();
List<HCard> result = helper.solve(execService, solvers);
//then i do smth with result list

public class Helper {
List<HCard> solve(Executor e, Collection<Callable<HCard>> solvers) throws InterruptedException {
    CompletionService<HCard> cs = new ExecutorCompletionService<HCard>(e);
    int n = solvers.size();

    Future<HCard> future = null;
    HCard hCard = null;
    ArrayList<HCard> result = new ArrayList<HCard>();

    for (Callable<HCard> s : solvers) {
        cs.submit(s);
    }
    for (int i = 0; i < n; ++i) {
        try {
            future = cs.take();
            hCard = future.get();
            if (hCard != null) {
                result.add(hCard);
            }
        } catch (ExecutionException e1) {
            future.cancel(true);
        }
    }
    return result;
}

Collection solvers=new ArrayList（）；
for（最终字符串currentUrl:allUrls）{
add（新的可调用（）{
public HCard call（）引发ParserException{
HCard-HCard=HCardParser.parseOne（当前URL）；
如果（hCard！=null）{
返回hCard；
}否则{
返回null；
}
}
});
}
ExecutorService execService=Executors.newCachedThreadPool（）；
Helper=newhelper（）；
列表结果=helper.solve（execService，solver）；
//然后我用结果表做smth

我的呼叫代码：

Collection<Callable<HCard>> solvers = new ArrayList<Callable<HCard>>();
for (final String currentUrl : allUrls) {
    solvers.add(new Callable<HCard>() {
        public HCard call() throws ParserException {
            HCard hCard = HCardParser.parseOne(currentUrl);                      
            if (hCard != null) {
                return hCard;
            } else {
                return null;
            }
        }
    });
}
ExecutorService execService = Executors.newCachedThreadPool();
Helper helper = new Helper();
List<HCard> result = helper.solve(execService, solvers);
//then i do smth with result list

public class Helper {
List<HCard> solve(Executor e, Collection<Callable<HCard>> solvers) throws InterruptedException {
    CompletionService<HCard> cs = new ExecutorCompletionService<HCard>(e);
    int n = solvers.size();

    Future<HCard> future = null;
    HCard hCard = null;
    ArrayList<HCard> result = new ArrayList<HCard>();

    for (Callable<HCard> s : solvers) {
        cs.submit(s);
    }
    for (int i = 0; i < n; ++i) {
        try {
            future = cs.take();
            hCard = future.get();
            if (hCard != null) {
                result.add(hCard);
            }
        } catch (ExecutionException e1) {
            future.cancel(true);
        }
    }
    return result;
}

公共类助手{
列表求解（执行器e、集合求解器）抛出InterruptedException{
CompletionService cs=新的ExecutionCompletionService（e）；
int n=solvers.size（）；
Future=null；
HCard HCard=null；
ArrayList结果=新建ArrayList（）；
for（可调用的s:解算器）{
政务司司长提交；
}
对于（int i=0；i


我试图使用：

等待终止（5000，时间单位毫秒）
future.cancel（真）
execService.shutdownNow（）
get（5000，时间单位毫秒）
TimeOutException:我无法获取TimeOutException

请帮助我了解代码的上下文。

提前感谢！
您需要确保您提交的任务正确响应中断，即检查Thread.isInterrupted（）或以其他方式被视为“可中断”
我不确定您是否需要完成此项服务
ExecutorService service = ...

// Submit all your tasks
for (Task t : tasks) {
    service.submit(t);
}

service.shutdown();

// Wait for termination
boolean success = service.awaitTermination(5, TimeUnit.SECONDS);
if (!success) {
    // awaitTermination timed out, interrupt everyone
    service.shutdownNow();
}

此时，如果您的任务对象不响应中断，您将无能为力。您需要确保您提交的任务正确响应中断，即检查Thread.isInterrupted（）或以其他方式被视为“可中断”
我不确定您是否需要完成此项服务
ExecutorService service = ...

// Submit all your tasks
for (Task t : tasks) {
    service.submit(t);
}

service.shutdown();

// Wait for termination
boolean success = service.awaitTermination(5, TimeUnit.SECONDS);
if (!success) {
    // awaitTermination timed out, interrupt everyone
    service.shutdownNow();
}

此时，如果您的任务对象不响应中断，您将无能为力
我从未使用过CompletionService，但我确信有一个轮询（timeunit，unit）调用来执行有限量的等待。然后检查null。测量等待的时间，并在5秒后停止等待。大约：
public class Helper {
List<HCard> solve(Executor e, Collection<Callable<HCard>> solvers) 
throws InterruptedException {
CompletionService<HCard> cs = new ExecutorCompletionService<HCard>(e);
int n = solvers.size();

Future<HCard> future = null;
HCard hCard = null;
ArrayList<HCard> result = new ArrayList<HCard>();

for (Callable<HCard> s : solvers) {
    cs.submit(s);
}
long timeleft = 5000;
for (int i = 0; i < n; ++i) {
    if (timeleft <= 0) {
        break;
    }
    try {
        long t = System.currentTimeMillis();
        future = cs.poll(timeleft, TimeUnit.MILLISECONDS);
        timeleft -= System.currentTimeMillis() - t;
        if (future != null) {
            hCard = future.get();
            if (hCard != null) {
                result.add(hCard);
            }
        } else {
           break;
        }
    } catch (ExecutionException e1) {
        future.cancel(true);
    }
}
return result;
}

公共类助手{
列表求解（执行器e、集合求解器）
抛出中断异常{
CompletionService cs=新的ExecutionCompletionService（e）；
int n=solvers.size（）；
Future=null；
HCard HCard=null；
ArrayList结果=新建ArrayList（）；
for（可调用的s:解算器）{
政务司司长提交；
}
长时间有效期=5000；
对于（int i=0；i如果（timeleft我从未使用过CompletionService，但我确定有一个轮询（timeunit，unit）调用来执行有限量的等待。然后检查null。测量等待的时间，并在5秒后停止等待。大约：
public class Helper {
List<HCard> solve(Executor e, Collection<Callable<HCard>> solvers) 
throws InterruptedException {
CompletionService<HCard> cs = new ExecutorCompletionService<HCard>(e);
int n = solvers.size();

Future<HCard> future = null;
HCard hCard = null;
ArrayList<HCard> result = new ArrayList<HCard>();

for (Callable<HCard> s : solvers) {
    cs.submit(s);
}
long timeleft = 5000;
for (int i = 0; i < n; ++i) {
    if (timeleft <= 0) {
        break;
    }
    try {
        long t = System.currentTimeMillis();
        future = cs.poll(timeleft, TimeUnit.MILLISECONDS);
        timeleft -= System.currentTimeMillis() - t;
        if (future != null) {
            hCard = future.get();
            if (hCard != null) {
                result.add(hCard);
            }
        } else {
           break;
        }
    } catch (ExecutionException e1) {
        future.cancel(true);
    }
}
return result;
}

公共类助手{
列表求解（执行器e、集合求解器）
抛出中断异常{
CompletionService cs=新的ExecutionCompletionService（e）；
int n=solvers.size（）；
Future=null；
HCard HCard=null；
ArrayList结果=新建ArrayList（）；
for（可调用的s:解算器）{
政务司司长提交；
}
长时间有效期=5000；
对于（int i=0；i如果（timeleft问题是你总是得到每一个结果，所以代码总是运行到完成。我会按照下面的代码使用倒计时锁存器来完成
另外，不要使用Executors.newCachedThreadPool-这很可能会产生大量线程（如果任务花费任何时间，则最多会产生300个线程，因为executor不会让空闲线程的数量降至零）
类都是内联的，以使其更容易-将整个代码块粘贴到名为LotsOfTasks的类中并运行它
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.List;
import java.util.Random;
import java.util.concurrent.Callable;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;

public class LotsOfTasks {
    private static final int SIZE = 300;

    public static void main(String[] args) throws InterruptedException {

        String[] allUrls = generateUrls(SIZE);

        Collection<Callable<HCard>> solvers = new ArrayList<Callable<HCard>>();
        for (final String currentUrl : allUrls) {
            solvers.add(new Callable<HCard>() {
                public HCard call() {
                    HCard hCard = HCardParser.parseOne(currentUrl);
                    if (hCard != null) {
                        return hCard;
                    } else {
                        return null;
                    }
                }
            });
        }
        ExecutorService execService = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors()); // One thread per cpu, ideal for compute-bound
        Helper helper = new Helper();

        System.out.println("Starting..");
        long start = System.nanoTime();
        List<HCard> result = helper.solve(execService, solvers, 5);
        long stop = System.nanoTime();
        for (HCard hCard : result) {
            System.out.println("hCard = " + hCard);
        }

        System.out.println("Took: " + TimeUnit.SECONDS.convert((stop - start), TimeUnit.NANOSECONDS) + " seconds");
    }

    private static String[] generateUrls(final int size) {
        String[] urls = new String[size];
        for (int i = 0; i < size; i++) {
            urls[i] = "" + i;
        }
        return urls;
    }

    private static class HCardParser {
        private static final Random random = new Random();

        public static HCard parseOne(String currentUrl) {
            try {
                Thread.sleep(random.nextInt(1000)); // Wait for a random time up to 1 seconds per task (simulate some activity)
            } catch (InterruptedException e) {
                // ignore
            }
            return new HCard(currentUrl);
        }
    }

    private static class HCard {
        private final String currentUrl;

        public HCard(String currentUrl) {
            this.currentUrl = currentUrl;
        }

        @Override
        public String toString() {
            return "HCard[" + currentUrl + "]";
        }
    }

    private static class Helper {
        List<HCard> solve(ExecutorService e, Collection<Callable<HCard>> solvers, int timeoutSeconds) throws InterruptedException {

            final CountDownLatch latch = new CountDownLatch(solvers.size());

            final ConcurrentLinkedQueue<HCard> executionResults = new ConcurrentLinkedQueue<HCard>();

            for (final Callable<HCard> s : solvers) {
                e.submit(new Callable<HCard>() {
                    public HCard call() throws Exception {
                        try {
                            executionResults.add(s.call());
                        } finally {
                            latch.countDown();
                        }
                        return null;
                    }
                });
            }

            latch.await(timeoutSeconds, TimeUnit.SECONDS);

            final List<Runnable> unfinishedTasks = e.shutdownNow();
            System.out.println("There were " + unfinishedTasks.size() + " urls not processed");

            return Arrays.asList(executionResults.toArray(new HCard[executionResults.size()]));
        }
    }
}

问题是你总是得到每一个结果，所以代码总是运行到完成。我会按照下面的代码使用倒计时锁存器
另外，不要使用Executors.newCachedThreadPool-这很可能会产生大量线程（如果任务花费任何时间，则最多会产生300个线程，因为executor不会让空闲线程的数量降至零）
类都是内联的，以使其更容易-将整个代码块粘贴到名为LotsOfTasks的类中并运行它
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.List;
import java.util.Random;
import java.util.concurrent.Callable;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;

public class LotsOfTasks {
    private static final int SIZE = 300;

    public static void main(String[] args) throws InterruptedException {

        String[] allUrls = generateUrls(SIZE);

        Collection<Callable<HCard>> solvers = new ArrayList<Callable<HCard>>();
        for (final String currentUrl : allUrls) {
            solvers.add(new Callable<HCard>() {
                public HCard call() {
                    HCard hCard = HCardParser.parseOne(currentUrl);
                    if (hCard != null) {
                        return hCard;
                    } else {
                        return null;
                    }
                }
            });
        }
        ExecutorService execService = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors()); // One thread per cpu, ideal for compute-bound
        Helper helper = new Helper();

        System.out.println("Starting..");
        long start = System.nanoTime();
        List<HCard> result = helper.solve(execService, solvers, 5);
        long stop = System.nanoTime();
        for (HCard hCard : result) {
            System.out.println("hCard = " + hCard);
        }

        System.out.println("Took: " + TimeUnit.SECONDS.convert((stop - start), TimeUnit.NANOSECONDS) + " seconds");
    }

    private static String[] generateUrls(final int size) {
        String[] urls = new String[size];
        for (int i = 0; i < size; i++) {
            urls[i] = "" + i;
        }
        return urls;
    }

    private static class HCardParser {
        private static final Random random = new Random();

        public static HCard parseOne(String currentUrl) {
            try {
                Thread.sleep(random.nextInt(1000)); // Wait for a random time up to 1 seconds per task (simulate some activity)
            } catch (InterruptedException e) {
                // ignore
            }
            return new HCard(currentUrl);
        }
    }

    private static class HCard {
        private final String currentUrl;

        public HCard(String currentUrl) {
            this.currentUrl = currentUrl;
        }

        @Override
        public String toString() {
            return "HCard[" + currentUrl + "]";
        }
    }

    private static class Helper {
        List<HCard> solve(ExecutorService e, Collection<Callable<HCard>> solvers, int timeoutSeconds) throws InterruptedException {

            final CountDownLatch latch = new CountDownLatch(solvers.size());

            final ConcurrentLinkedQueue<HCard> executionResults = new ConcurrentLinkedQueue<HCard>();

            for (final Callable<HCard> s : solvers) {
                e.submit(new Callable<HCard>() {
                    public HCard call() throws Exception {
                        try {
                            executionResults.add(s.call());
                        } finally {
                            latch.countDown();
                        }
                        return null;
                    }
                });
            }

            latch.await(timeoutSeconds, TimeUnit.SECONDS);

            final List<Runnable> unfinishedTasks = e.shutdownNow();
            System.out.println("There were " + unfinishedTasks.size() + " urls not processed");

            return Arrays.asList(executionResults.toArray(new HCard[executionResults.size()]));
        }
    }
}

您使用缓存线程池有什么原因吗？在最坏的情况下它将启动300个线程。考虑使用FixedThreadPool吗？考虑过。我通过探查器看到-我的代码没有区别。您使用缓存线程池有什么原因吗？在最坏的情况下它将启动300个线程。考虑使用FixedThreadPool吗？考虑过。我通过探查器看到-没有o我的代码的区别。+1提到可中断性：这一点让我遇到了类似的问题（见我的个人资料）.不确定HCardParser.parseOne在做什么，但在我的例子中，我调用的函数不受中断的影响..使用HTTP_连接超时，或者不管你在做什么，你可能会更幸运地使你的可调用项停止。+1提到可中断性：那一位