Java 使用CallableTask/Futures和ObjectMapper时多线程代码的性能问题
我正在从事一个基于REST服务的项目,其中我有两个组件,如下所述-Java 使用CallableTask/Futures和ObjectMapper时多线程代码的性能问题,java,multithreading,jackson,future,Java,Multithreading,Jackson,Future,我正在从事一个基于REST服务的项目,其中我有两个组件,如下所述- 客户端,它将为服务组件生成必要的URL 然后服务(REST服务)组件将使用这些URL的从数据库中获取数据 一般来说,URL将如下所示- http://host.qa.ebay.com:8080/deservice/DEService/get/USERID=9012/PROFILE.ACCOUNT,个人资料。广告,个人资料。人口统计,个人资料。财务 上面URL的意思是-对于USERID-9012请从数据库中为这些列提供数据- [P
URL
URL的
从数据库中获取数据http://host.qa.ebay.com:8080/deservice/DEService/get/USERID=9012/PROFILE.ACCOUNT,个人资料。广告,个人资料。人口统计,个人资料。财务
上面URL的意思是-对于USERID-9012
请从数据库中为这些列提供数据-
[PROFILE.ACCOUNT、PROFILE.advisting、PROFILE.DEMOGRAPHIC、PROFILE.FINANCIAL]
目前,我正在客户机组件端进行基准测试。我发现下面的方法需要花费大量的时间(95个百分点)
大约~15ms
下面的方法将接受两个参数-
列出密钥-密钥中的示例数据的USERID=9012
列出reqAttrNames-reqAttrNames的示例数据将为-
[PROFILE.ACCOUNT、PROFILE.advisting、PROFILE.DEMOGRAPHIC、PROFILE.FINANCIAL]
下面是代码-
public DEResponse getDEAttributes(List<DEKey> keys, List<String> reqAttrNames) {
DEResponse response = null;
try {
String url = buildGetUrl(keys,reqAttrNames);
if(url!=null){
List<CallableTask<DEResponse>> tasks = new ArrayList<CallableTask<DEResponse>>();
CallableTask<DEResponse> task = new DEResponseTask(url);
tasks.add(task);
// STEP 2: Execute worker threads for all the generated urls
List<LoggingFuture<DEResponse>> futures = null;
try {
long waitTimeout = getWaitTimeout(keys);
futures = executor.executeAll(tasks, null, waitTimeout, TimeUnit.MILLISECONDS);
// STEP 3: Consolidate results of the executed worker threads
if(futures!=null && futures.size()>0){
LoggingFuture<DEResponse> future = futures.get(0);
response = future.get();
}
} catch (InterruptedException e1) {
logger.log(LogLevel.ERROR,"Transport:getDEAttributes Request timed-out :",e1);
}
}else{
//
}
} catch(Throwable th) {
}
return response;
}
这个多线程代码的编写方式有问题吗?如果是,我如何才能使其高效
executeAll的签名executor的方法
与我的公司一样,他们有自己的executor,它将实现Sun executor类-
/**
* Executes the given tasks, returning a list of futures holding their
* status and results when all complete or the timeout expires, whichever
* happens first. <tt>Future.isDone()</tt> is <tt>true</tt> for each
* element of the returned list. Upon return, tasks that have not completed
* are cancelled. Note that a <i>completed</i> task could have terminated
* either normally or by throwing an exception. The results of this method
* are undefined if the given collection is modified while this operation is
* in progress. This is entirely analogous to
* <tt>ExecutorService.invokeAll()</tt> except for a couple of important
* differences. First, it cancels but does not <b>interrupt</b> any
* unfinished tasks, unlike <tt>ExecutorService.invokeAll()</tt> which
* cancels and interrupts unfinished tasks. This results in a better
* adherence to the specified timeout value, as interrupting threads may
* have unexpected delays depending on the nature of the tasks. Also, all
* eBay-specific features apply when the tasks are submitted with this
* method.
*
* @param tasks the collection of tasks
* @param timeout the maximum time to wait
* @param unit the time unit of the timeout argument
* @return a list of futures representing the tasks, in the same sequential
* order as produced by the iterator for the given task list. If the
* operation did not time out, each task will have completed. If it did
* time out, some of these tasks will not have completed.
* @throws InterruptedException if interrupted while waiting, in which case
* unfinished tasks are cancelled
*/
public <V> List<LoggingFuture<V>> executeAll(Collection<? extends CallableTask<V>> tasks,
Options options,
long timeout, TimeUnit unit)
throws InterruptedException {
return executeAll(tasks, options, timeout, unit, false);
}
除了使用复杂的非标准执行器之外,我看不到任何会导致性能下降的因素。我知道在使用哪种执行器的问题上您没有任何选择,但出于好奇,我会尝试用ThreadPoolExecutor
替换它,看看这是否有什么不同,如果你注意到一个重大的改进,那么就用你工作中的能力来解释它——在我的工作中,我们发现由另一个部门编写的加密库完全是垃圾(我们80-90%的CPU时间都花在了代码上),并成功地游说他们重写它
编辑:
公共类聚合器实现可运行{
私有静态ConcurrentLinkedQueue=新ConcurrentLinkedQueue();
私有静态ArrayList聚合=新建ArrayList();
公开静态无效要约(未来){
报价(未来);
}
公共静态ArrayList getAggregation(){
收益聚合;
}
公开募捐{
while(!queue.isEmpty()){//请确保在此循环开始之前添加了所有的未来;更好的是,如果您知道有多少工作线程,那么请记录聚合器中有多少个未来,并在aggregator.size()=[预期的未来数]时退出此循环
add(queue.poll().get());
}
}
}
public void getdeAttribute(列表键、列表名称){
试一试{
如果(url!=null){
试一试{
futures=executor.executeAll(任务,null,等待超时,时间单位为毫秒);
if(futures!=null&&futures.size()>0){
聚合器.offer(futures.get(0));
}
}
}
}
}
如果我正确阅读了您的代码,那么有一个明显的性能问题。这:
public class DEResponseTask extends BaseNamedTask implements CallableTask<DEResponse> {
private final ObjectMapper m_mapper = new ObjectMapper();
公共类DEResponseTask扩展BaseNamedTask实现CallableTask{
私有最终ObjectMapper m_mapper=新ObjectMapper();
每个任务调用一次,创建ObjectMapper
实例的成本非常高
有很多方法可以解决此问题,但您可能希望:
使m_映射器引用为静态(仅创建一次)--映射器在配置后可以安全共享,或者
传入共享ObjectMapper
(共享是安全的)
这样做会大大提高JSON的处理效率。如果任务超过一定的运行时间,执行者是否是唯一可以终止任务的点?@Perception是的,我相信是的。这就是为什么我们在那里有超时的原因,这样我们就可以在占用大量时间后立即超时。谢谢Zim Zam的建议。是的,我会试试这个当然。我也更新了我的问题,没有更多的细节。也许你可以得到更多的想法。谢谢。future.get()
正在阻塞;假设这个方法在它自己的线程中,你可以用while(!future.isDone()){thread.sleep(500);}future.get()替换它
这样你就不用再等待了。谢谢Zim Zam。是的,它会在它自己的线程中。我用你的建议更新了我的问题。你能看一下吗,我在正确的地方做了正确的更改吗?谢谢。我又错过了一个要添加的单词。我再次添加了。再看一看,让我知道它是否好看。很抱歉。但是通过这种方式,什么我将获得的好处?看起来是正确的。好处是在未来之前。get()可能是自旋等待,这意味着它将继续消耗CPU资源,同时等待未来的完成-如果有很多线程这样做,那么它可能会对性能产生负面影响。这样,您只需在半秒的时间间隔内使用CPU,并且您可以让任意多的线程睡眠等待,而无需阻塞是的,我怎么能忘记呢。我只需要添加静态对象对吗?类似这样的东西?private static final ObjectMapper m\u mapper=new ObjectMapper();
right?是的,这是最简单的方法。
if(futures!=null && futures.size()>0){
LoggingFuture<DEResponse> future = futures.get(0);
//response = future.get();//replace this with below code-
while(!future.isDone()) {
Thread.sleep(500);
}
response = future.get();
}
public class Aggregator implements Runnable {
private static ConcurrentLinkedQueue<Future<DEResponse>> queue = new ConcurrentLinkedQueue<>();
private static ArrayList<DEResponse> aggregation = new ArrayList<>();
public static void offer(Future<DEResponse> future) {
queue.offer(future);
}
public static ArrayList<DEResponse> getAggregation() {
return aggregation;
}
public void run() {
while(!queue.isEmpty()) { // make sure that all of the futures are added before this loop starts; better still, if you know how many worker threads there are then keep a count of how many futures are in your aggregator and quit this loop when aggregator.size() == [expected number of futures]
aggregation.add(queue.poll().get());
}
}
}
public void getDEAttributes(List<DEKey> keys, List<String> reqAttrNames) {
try {
if(url!=null){
try {
futures = executor.executeAll(tasks, null, waitTimeout, TimeUnit.MILLISECONDS);
if(futures!=null && futures.size()>0){
Aggregator.offer(futures.get(0));
}
}
}
}
}
public class DEResponseTask extends BaseNamedTask implements CallableTask<DEResponse> {
private final ObjectMapper m_mapper = new ObjectMapper();