Networking 如何在Clojure（/Java）中可靠地发出大量并发HTTPS请求

我有一个输入流，在将结果传递到程序的另一部分之前，我想为每个输入流发出2个

HTTPS

网络请求。典型的吞吐量是每秒50次

for each input:
    HTTP request A
    HTTP request B
    pass event on with (A.body and B.body)

我使用的是客户端，默认情况下是异步的。它返回一个承诺，也可以接受回调。Http工具包使用JavaNIO（请参阅和）

传入请求的速度，加上发出请求的时间，足够高，需要异步完成

我尝试了3种方法：

当一个事件进来时，把它放在一个频道上。许多退出频道的

go

例程。每个请求都通过从HTTP请求中删除承诺来“阻止”goblock。这不起作用，因为我认为promise不能很好地处理线程

当一个事件发生时，立即启动一个

future

，它将“阻止”异步进程。这会导致非常高的CPU使用率。加上网络资源的匮乏

当一个事件出现时，立即触发请求A的

http kit

请求，传入一个发出请求B的回调，传入一个传递事件的回调。这会导致几小时后出现内存不足错误

这些都可以工作并处理一段时间的容量。他们最终都崩溃了。最近一次坠机发生在大约12小时后：

Mar 10, 2016 2:05:59 AM com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector run
WARNING: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1bc8a7f5 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending
 tasks!
Mar 10, 2016 3:38:38 AM com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector run
WARNING: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1bc8a7f5 -- APPARENT DEADLOCK!!! Complete Status:
        Managed Threads: 3
        Active Threads: 1
        Active Tasks:
                com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@65d8b232 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
        Pending Tasks:
                com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@359acb0d
Pool thread stack traces:
        Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
                com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:560)
        Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
                java.lang.Object.wait(Native Method)
                com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
        Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
                java.lang.Object.wait(Native Method)
                com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)


Thu Mar 10 04:38:34 UTC 2016 [client-loop] ERROR - select exception, should not happen
java.lang.OutOfMemoryError: Java heap space
        at java.io.ByteArrayOutputStream.<init>(ByteArrayOutputStream.java:77)
        at sun.security.ssl.OutputRecord.<init>(OutputRecord.java:76)
        at sun.security.ssl.EngineOutputRecord.<init>(EngineOutputRecord.java:65)
        at sun.security.ssl.HandshakeOutStream.<init>(HandshakeOutStream.java:63)
        at sun.security.ssl.Handshaker.activate(Handshaker.java:514)
        at sun.security.ssl.SSLEngineImpl.kickstartHandshake(SSLEngineImpl.java:717)
        at sun.security.ssl.SSLEngineImpl.beginHandshake(SSLEngineImpl.java:743)
        at org.httpkit.client.HttpClient.finishConnect(HttpClient.java:310)
        at org.httpkit.client.HttpClient.run(HttpClient.java:375)
        at java.lang.Thread.run(Thread.java:745)
Mar 10, 2016 4:56:34 AM baleen.events invoke
SEVERE: Thread error: Java heap space
java.lang.OutOfMemoryError: Java heap space
Mar 10, 2016 5:00:43 AM baleen.events invoke
SEVERE: Thread error: Java heap space
java.lang.OutOfMemoryError: Java heap space
Mar 10, 2016 4:58:25 AM baleen.events invoke
SEVERE: Thread error: Java heap space
java.lang.OutOfMemoryError: Java heap space

2016年3月10日凌晨2:05:59 com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector运行
警告：com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1bc8a7f5--明显的僵局！！！为未分配的挂起线程创建紧急线程
任务！
2016年3月10日凌晨3:38:38 com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector运行
警告：com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1bc8a7f5--明显的僵局！！！完成状态：
托管线程：3个
活动线程：1
活动任务：
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@65d8b232（com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0）
未决任务：
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@359acb0d
池线程堆栈跟踪：
线程[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5，main]
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run（ThreadPoolAsynchronousRunner.java:560）
线程[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5，main]
java.lang.Object.wait（本机方法）
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run（ThreadPoolAsynchronousRunner.java:534）
线程[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5，main]
java.lang.Object.wait（本机方法）
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run（ThreadPoolAsynchronousRunner.java:534）
3月10日星期四04:38:34 UTC 2016[客户端循环]错误-选择异常，不应发生
java.lang.OutOfMemoryError:java堆空间
位于java.io.ByteArrayOutputStream。（ByteArrayOutputStream.java:77）
位于sun.security.ssl.OutputRecord.（OutputRecord.java:76）
位于sun.security.ssl.EngineOutputRecord.（EngineOutputRecord.java:65）
位于sun.security.ssl.HandshakeOutStream（HandshakeOutStream.java:63）
位于sun.security.ssl.Handshaker.activate（Handshaker.java:514）
在sun.security.ssl.SSLEngineImpl.kickstartHandshake（SSLEngineImpl.java:717）
位于sun.security.ssl.sslenginimpl.beginHandshake（sslenginimpl.java:743）
位于org.httpkit.client.HttpClient.finishConnect（HttpClient.java:310）
位于org.httpkit.client.HttpClient.run（HttpClient.java:375）
运行（Thread.java:745）
2016年3月10日上午4:56:34巴伦
严重：线程错误：Java堆空间
java.lang.OutOfMemoryError:java堆空间
2016年3月10日上午5:00:43巴伦
严重：线程错误：Java堆空间
java.lang.OutOfMemoryError:java堆空间
2016年3月10日上午4:58:25巴伦
严重：线程错误：Java堆空间
java.lang.OutOfMemoryError:java堆空间

我不知道失败的原因是什么。可能有太多的闭包被保留，或者资源逐渐泄漏，或者线程不足

问题

每秒发出50个HTTP请求（每个请求可能需要200毫秒），这意味着在任何给定时间都可能有100个请求在传输中，这听起来像是一个过重的负担吗

如何以处理吞吐量和健壮性的方式实现这一点

编辑

---

YourKit profiler告诉我，通过

org.httpkit.client.Handler

s通过

java.util.concurrent.FutureTask

s，我有大约2GB的

char[]

s，这表明对旧处理程序的引用（即请求）以某种方式被保留。尝试使用回调的全部原因是为了避免这种情况（尽管它们可能会以某种方式陷入闭包中）

方法A的替代方法（

deref

ing在go块中返回的HTTP工具包）可能是可行的，只是这样做不会导致将来阻塞core.async处理程序线程，您可以通过结合httpkit的回调和core.async来实现这一点：

(defn handle-event
 "Return a core.async channel that will contain the result of making both HTTP call A and B."
  [event-data]
  (let [event-a-chan (clojure.core.async/chan)
        event-b-chan (clojure.core.async/chan)
        return-chan (clojure.core.async/chan)]
    (org.httpkit.client/request "https://event-a-call"
                                {:method :get :params {"param1-k" "param1-v"}}
                                (fn [resp]
                                  (clojure.core.async/put! event-a-chan resp)))
    (org.httpkit.client/request "https://event-b-call"
                                {:method :get :params {"param1-k" "param1-v"}}
                                (fn [resp]
                                  (clojure.core.async/put! event-b-chan resp)))
    (clojure.core.async/go
      (clojure.core.async/>! return-chan {:event-a-response (clojure.core.async/<! event-a-chan)
                                          :event-b-response (clojure.core.async/<! event-b-chan)}))
    return-chan))

（defn句柄事件
“返回一个core.async通道，该通道将包含进行HTTP调用a和B的结果。”
[事件数据]
（let[event-a-chan（clojure.core.async/chan）
event-b-chan（clojure.core.async/chan）
返回chan（clojure.core.async/chan）]
（org.httpkit.client/request）https://event-a-call"
{：方法：get:params{“param1-k”“param1-v”}
（fn[resp]
（clojure.core.async/put！event-a-c）
(ns concur-req.core
  (require [clojure.core.async :as async]
           [cheshire.core :refer [decode]]
           [org.httpkit.client :as http]))

(defn url-of
  [id]
  ;; this service responds within 100-200ms
  (str "http://localhost:28080/" id ".json"))

(defn retrieve-json-async
  [url c]
  (http/get url nil
            (fn [{body :body status :status :as resp}]
              (if (= 200 status)
                (async/put! c (decode body true))
                (println "ERROR:" resp))
              (async/close! c))))

(defn run [parallelism stop-chan]
  (let [;; allocate half of the parallelism to each step
        step1-n    (int (max (/ parallelism 2) 1))
        step2-n    step1-n
        ;; buffer to take ids, transform them into urls
        step1-chan (async/chan step1-n (map url-of))
        ;; buffer for result of pulling urls from step1, xform by extracting :next url
        step2-chan (async/chan step2-n (map :next))
        ;; buffer to count completed results
        out-chan   (async/chan 1 (map (constantly 1)))
        ;; for delivering the final result
        final-chan (async/chan)
        start-time (System/currentTimeMillis)]

    ;; process URLs from step1 and put the result in step2
    (async/pipeline-async step1-n step2-chan retrieve-json-async step1-chan)
    ;; process URLs from step2 and put the result in out
    (async/pipeline-async step2-n out-chan retrieve-json-async step2-chan)

    ;; keep the input channel full until stop-chan is closed.
    (async/go-loop []
      (let [[v c] (async/alts! [stop-chan [step1-chan "1"]])]
        (if (= c stop-chan)
          (async/close! step1-chan)
          (recur))))

    ;; count messages on out-chan until the pipeline is closed, printing
    ;; status message every second
    (async/go-loop [status-timer (async/timeout 1000) subt 0 accu 0]
      (let [[v c] (async/alts! [status-timer out-chan])]
        (cond (= c status-timer)
              (do (println subt "records...")
                  (recur (async/timeout 1000) 0 (+ subt accu)))

              (nil? v)
              (async/>! final-chan (+ subt accu))

              :else
              (recur status-timer (+ v subt) accu))))

    ;; block until done, then emit final report.
    (let [final-total (async/<!! final-chan)
          elapsed-ms  (- (System/currentTimeMillis) start-time)
          elapsed-s   (/ elapsed-ms 1000.0)]
      (print (format "Processed %d records with parallelism %d in %.3f seconds (%d/sec)\n"
                     final-total parallelism elapsed-s
                     (int (/ final-total elapsed-s)))))))

(defn run-for
  [seconds parallelism]
  (let [stop-chan (async/chan)]
    (future
      (Thread/sleep (* seconds 1000))
      (async/close! stop-chan))
    (run parallelism stop-chan)))

(do
  ;; Warm up the connection pool, avoid somaxconn problems...
  (doseq [p (map #(* 20 (inc %)) (range 25))]
    (run-for 1 p))
  (run-for (* 60 60 6) 500))