scalaj HTTP发送1000'；Spark Streaming中的帖子数量-错误

我在一个流媒体应用程序中调用了一个简单的函数，它可以像这样发布数据：

def jsonPost(json: String) = {
  val result = Http("http://" + Address + "/api/v1/write").postData(json)
    .header("Content-Type", "application/json")
    .header("Charset", "UTF-8").asString
  result
}

如果我发送10条记录，发送数据就没有问题。当我想发送1000条时，我的Spark流代码中出现以下错误：

java.net.SocketException: Connection reset
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1890)
    at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1885)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1884)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1457)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
    at scalaj.http.HttpRequest.exec(Http.scala:351)
    at scalaj.http.HttpRequest.execute(Http.scala:322)
    at scalaj.http.HttpRequest.asString(Http.scala:537)
    at com.capitalone.streamingExample.streaming$.com$capitalone$streamingExample$streaming$$elasticPost$1(streaming.scala:83)
    at com.capitalone.streamingExample.streaming$$anonfun$createStream$1$$anonfun$apply$1.apply(streaming.scala:62)
    at com.capitalone.streamingExample.streaming$$anonfun$createStream$1$$anonfun$apply$1.apply(streaming.scala:59)
    at scala.collection.Iterator$class.foreach(Iterator.scala:727)
    at org.apache.spark.util.NextIterator.foreach(NextIterator.scala:21)
    at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$32.apply(RDD.scala:912)
    at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$32.apply(RDD.scala:912)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:89)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:209)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704)
    at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:789)
    at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1536)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
    at scalaj.http.HttpRequest.exec(Http.scala:349)
    ... 17 more

---

有没有什么解决方案可以让我扩展到发送API服务器1000条帖子而不产生错误？是否有一个更可扩展的解决方案，我应该看看

你确定瓶颈在哪里了吗？在发送端还是接收端？是的，因为当我从10-20个消息开始缩放时，我开始得到这个错误。我想知道我是否应该从scalaj http切换到另一个可以保持连接的库。你确定瓶颈在哪里了吗？在发送端还是接收端？是的，因为当我从10-20个消息开始缩放时，我开始得到这个错误。我想知道我是否应该从scalaj http切换到另一个可以保持连接的库。