Multithreading 在scala崩溃的情况下使用框架2.7

我有一个play scala应用程序运行在play 2.7上。它被用作我们前端的中间件，它有rest端点

现在，我在云上运行两个不同的实例，并使用nginx绑定这两个服务器，并使用round-robin进行负载平衡

现在我遇到了一个问题，服务器经常停机，即一天3次，有趣的是两台服务器同时停机。当我看到它时，它说两台服务器都出现了内存不足异常。我试图打印javaheapdump，因为内存不足，但没有得到转储。我仍在分析线程转储，以找出服务器宕机的实际原因，但让我感到困惑的是，这两台服务器同时宕机的原因

在线程转储之外，我看到有7707个线程处于休眠状态。给你

"Connection evictor" #146 daemon prio=5 os_prio=0 cpu=2.33ms elapsed=1822.02s tid=0x00007f8a840c4800 nid=0x194 waiting on condition  [0x00007f8a58a5e000]
  java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(java.base@11/Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(java.base@11/Thread.java:834)

这就是我在服务器停机时看到的

[35966.967s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
Uncaught error from thread [application-akka.actor.default-dispatcher-1398Uncaught error from thread [application-akka.actor.default-dispatcher-1395]: ]: unable to create native thread: possibly out of memory or process/resource limits reachedunable to create native thread: possibly out of memory or process/resource limits reached, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[ ActorSystem[applicationapplication]
]
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
  at java.base/java.lang.Thread.start0(Native Method)
  at java.base/java.lang.Thread.start(Thread.java:803)
  at org.apache.http.impl.client.IdleConnectionEvictor.start(IdleConnectionEvictor.java:96)
  at org.apache.http.impl.client.HttpClientBuilder.build(HttpClientBuilder.java:1219)
  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:287)
  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:298)
  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:236)
  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:223)
  at org.apache.solr.client.solrj.impl.HttpSolrClient.<init>(HttpSolrClient.java:198)
  at org.apache.solr.client.solrj.impl.HttpSolrClient$Builder.build(HttpSolrClient.java:934)
  at com.github.takezoe.solr.scala.SolrClient$.$anonfun$$lessinit$greater$default$2$1(SolrClient.scala:11)
  at com.github.takezoe.solr.scala.SolrClient.<init>(SolrClient.scala:14)
  at service.tvt.solr.SolrPolygonService.getSuburbBoundary(SolrPolygonService.scala:212)
  at service.tvt.search.OrbigoSearchService.mapfeeder(OrbigoSearchService.scala:54)
  at service.bto.business_categories.MeedssCountService.$anonfun$suburbMeedssCount$2(MeedssCountService.scala:81)
  at scala.collection.immutable.List.map(List.scala:287)
  at service.bto.business_categories.MeedssCountService.suburbMeedssCount(MeedssCountService.scala:80)
  at controllers.bto.industry_categories.meedss.MeedssController.$anonfun$suburbMeedssCount$1(MeedssController.scala:38)
  at play.api.mvc.ActionBuilder.$anonfun$apply$11(Action.scala:368)
  at scala.Function1.$anonfun$andThen$1(Function1.scala:52)
  at play.api.mvc.ActionBuilderImpl.invokeBlock(Action.scala:489)
  at play.api.mvc.ActionBuilderImpl.invokeBlock(Action.scala:487)
  at play.api.mvc.ActionBuilder$$anon$9.invokeBlock(Action.scala:336)
  at play.api.mvc.ActionBuilder$$anon$9.invokeBlock(Action.scala:331)
  at play.api.mvc.ActionBuilder$$anon$10.apply(Action.scala:426)
  at play.api.mvc.Action.$anonfun$apply$2(Action.scala:98)
  at play.api.libs.streams.StrictAccumulator.$anonfun$mapFuture$4(Accumulator.scala:184)
  at scala.util.Try$.apply(Try.scala:209)
  at play.api.libs.streams.StrictAccumulator.$anonfun$mapFuture$3(Accumulator.scala:184)
  at akka.stream.impl.Transform.apply(TraversalBuilder.scala:159)
  at akka.stream.impl.PhasedFusingActorMaterializer.materialize(PhasedFusingActorMaterializer.scala:515)
  at akka.stream.impl.PhasedFusingActorMaterializer.materialize(PhasedFusingActorMaterializer.scala:450)
  at akka.stream.impl.PhasedFusingActorMaterializer.materialize(PhasedFusingActorMaterializer.scala:443)
  at akka.stream.scaladsl.RunnableGraph.run(Flow.scala:629)
  at play.api.libs.streams.Accumulator$.$anonfun$futureToSink$2(Accumulator.scala:262)
  at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:303)
  at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:37)
  at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
  at play.api.libs.streams.Execution$trampoline$.execute(Execution.scala:72)
  at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
  at scala.concurrent.impl.Promise$DefaultPromise.dispatchOrAddCallback(Promise.scala:312)
  at scala.concurrent.impl.Promise$DefaultPromise.onComplete(Promise.scala:303)
  at scala.concurrent.impl.Promise.transformWith(Promise.scala:36)
  at scala.concurrent.impl.Promise.transformWith$(Promise.scala:34)
  at scala.concurrent.impl.Promise$DefaultPromise.transformWith(Promise.scala:183)
  at scala.concurrent.Future.flatMap(Future.scala:302)
  at scala.concurrent.Future.flatMap$(Future.scala:302)
  at scala.concurrent.impl.Promise$DefaultPromise.flatMap(Promise.scala:183)
  at play.api.libs.streams.Accumulator$.$anonfun$futureToSink$1(Accumulator.scala:261)
  at akka.stream.impl.Transform.apply(TraversalBuilder.scala:159)
  at akka.stream.impl.PhasedFusingActorMaterializer.materialize(PhasedFusingActorMaterializer.scala:515)
  at akka.stream.impl.PhasedFusingActorMaterializer.materialize(PhasedFusingActorMaterializer.scala:450)
  at akka.stream.impl.PhasedFusingActorMaterializer.materialize(PhasedFusingActorMaterializer.scala:443)
  at akka.stream.scaladsl.RunnableGraph.run(Flow.scala:629)
  at play.api.libs.streams.SinkAccumulator.run(Accumulator.scala:144)
  at play.api.libs.streams.SinkAccumulator.run(Accumulator.scala:148)
  at play.core.server.AkkaHttpServer.$anonfun$runAction$4(AkkaHttpServer.scala:441)
  at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
  at akka.http.scaladsl.util.FastFuture$.$anonfun$transformWith$3(FastFuture.scala:51)
  at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
  at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
  at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:92)
  at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
  at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
  at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:92)
  at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
  at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49)
  at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
  at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
  at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
  at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

[35966.967s][warning][os，thread]无法启动线程-pthread_创建失败（EAGAIN）属性：stacksize:1024k，guardsize:0k，已分离。
来自线程的未捕获错误[application-akka.actor.default-dispatcher-1398来自线程的未捕获错误[application-akka.actor.default-dispatcher-1395]：]：无法创建本机线程：可能内存不足或达到进程/资源限制无法创建本机线程：可能内存不足或达到进程/资源限制，由于为启用了“akka.JVM在致命错误时退出”而关闭JVM，由于为ActorSystem[ActorSystem[applicationapplication]启用了“akka.JVM在致命错误时退出”而关闭JVM
]
java.lang.OutOfMemoryError:无法创建本机线程：可能内存不足或已达到进程/资源限制
位于java.base/java.lang.Thread.start0（本机方法）
位于java.base/java.lang.Thread.start（Thread.java:803）
位于org.apache.http.impl.client.idleConnectionVictor.start（idleConnectionVictor.java:96）
位于org.apache.http.impl.client.HttpClientBuilder.build（HttpClientBuilder.java:1219）
位于org.apache.solr.client.solrj.impl.HttpClientUtil.createClient（HttpClientUtil.java:287）
位于org.apache.solr.client.solrj.impl.HttpClientUtil.createClient（HttpClientUtil.java:298）
位于org.apache.solr.client.solrj.impl.HttpClientUtil.createClient（HttpClientUtil.java:236）
位于org.apache.solr.client.solrj.impl.HttpClientUtil.createClient（HttpClientUtil.java:223）
位于org.apache.solr.client.solrj.impl.HttpSolrClient.（HttpSolrClient.java:198）
位于org.apache.solr.client.solrj.impl.HttpSolrClient$Builder.build（HttpSolrClient.java:934）
在com.github.takezoe.solr.scala.SolrClient$.$anonfun$$lessinit$morear$default$2$1（SolrClient.scala:11）
在com.github.takezoe.solr.scala.SolrClient.（SolrClient.scala:14）
at service.tvt.solr.SolrPolygonService.getSuburbBoundary（SolrPolygonService.scala:212）
at service.tvt.search.OrbigoSearchService.mapfeeder（OrbigoSearchService.scala:54）
在service.bto.business_categories.MeedssCountService.anonfun$suburbanmeedsscount$2（MeedssCountService.scala:81）
位于scala.collection.immutable.List.map（List.scala:287）
at service.bto.business_categories.MeedssCountService.suburbanmeedsscount（MeedssCountService.scala:80）
在controllers.bto.industry_categories.meedss.MeedssController.$anonfun$suburbanmeedsscount$1（MeedssController.scala:38）
at play.api.mvc.ActionBuilder.$anonfun$apply$11（Action.scala:368）
在scala.Function1.$anonfun$和$1（Function1.scala:52）
at play.api.mvc.ActionBuilderImpl.invokeBlock（Action.scala:489）
at play.api.mvc.ActionBuilderImpl.invokeBlock（Action.scala:487）
at play.api.mvc.ActionBuilder$$anon$9.invokeBlock（Action.scala:336）
at play.api.mvc.ActionBuilder$$anon$9.invokeBlock（Action.scala:331）
at play.api.mvc.ActionBuilder$$anon$10.apply（Action.scala:426）
at play.api.mvc.Action.$anonfun$apply$2（Action.scala:98）
atplay.api.libs.streams.StrictAccumulator.$anonfun$mapFuture$4（Accumulator.scala:184）
在scala.util.Try$.apply处（Try.scala:209）
atplay.api.libs.streams.StrictAccumulator.$anonfun$mapFuture$3（Accumulator.scala:184）
在akka.stream.impl.Transform.apply（TraversalBuilder.scala:159）
在akka.stream.impl.PhasedFusingActorMaterializer.materialize（PhasedFusingActorMaterializer.scala:515）
在akka.stream.impl.PhasedFusingActorMaterializer.materialize（PhasedFusingActorMaterializer.scala:450）
在akka.stream.impl.PhasedFusingActorMaterializer.materialize（PhasedFusingActorMaterializer.scala:443）
在akka.stream.scaladsl.RunnableGraph.run（Flow.scala:629）
at play.api.libs.streams.acculator$.$anonfun$futureToSink$2（acculator.scala:262）
在scala.concurrent.Future.$anonfun$flatMap$1（Future.scala:303）
在scala.concurrent.impl.Promise.$anonfun$transformWith$1（Promise.scala:37）
在scala.concurrent.impl.CallbackRunnable.run（Promise.scala:60）
at play.api.libs.streams.Execution$trampoline$.execute（Execution.scala:72）
位于scala.concurrent.impl.CallbackRunnable.executeWithValue（Promise.scala:68）
在scala.concurrent.impl.Promise$DefaultPromise.dispatcherAddCallback（Promise.scala:312）中
在scala.concurrent.impl.Promise$DefaultPromise.onComplete（Promise.scala:303）
位于scala.concurrent.impl.Promise.transformWith（Promise.scala:36）
在scala.concurrent.impl.Promise.transformWith$（Promise.scala:34）
位于scala.concurrent.impl.Promise$DefaultPromise.transformWith（Promise.scala:183）
位于scala.concurrent.Future.flatMap（Future.scala:302）
位于scala.concurrent.Future.flatMap$（Future.scala:302）
位于scala.concurrent.impl.Promise$DefaultPromise.flatMap（Promise.scala:183）
at play.api.libs.streams.acculator$.$anonfun$futureToSink$1（acculator.scala:261）
在akka.stream.impl.Transform.apply（TraversalBuilder.scala:159）
在akka.stream.impl.PhasedFusingActorMaterializer.materialize（PhasedFusingActorMaterializer.scala:515）
在akka.stream.impl.PhasedFusingActorMaterializer.materialize（PhasedFusingActorMaterializer.scala:450）
在akka.stream.impl.PhasedFusingActorMaterializer.materialize（PhasedFusingActorMaterializer.scala:443）
在akka.stream.scaladsl.Runn