使用多个并发tls拨号器，golang中的常驻内存不断增加

好吧，这已经困扰了我好几个星期了，我不知道我错过了什么，这个漏洞在哪里，或者它是否存在。我的工作量相当简单。获取URL列表，旋转一个goroutine池，从通道中提取URL，并使用tls.Dialer创建到它们的tls连接。下面是内存图的快照，显示了代码的持续上升和POC

我的猜测是，这与tls包完成的分配有关，因为它似乎只会爬升它连接到的更“成功”的URL。也就是说，如果它们中的大多数没有连接，我就看不到内存的稳定增长

以下是运行中途的输出结果：

Showing nodes accounting for 190.70MB, 95.58% of 199.53MB total
Dropped 34 nodes (cum <= 1MB)
Showing top 20 nodes out of 77
   flat  flat%   sum%        cum   cum%
   51.52MB 25.82% 25.82%    51.52MB 25.82%  runtime.malg
   24.10MB 12.08% 37.90%    24.10MB 12.08%  bytes.makeSlice
   17.07MB  8.55% 46.45%    41.17MB 20.63%  crypto/tls.(*Conn).readHandshake
   15MB  7.52% 53.97%    78.85MB 39.52%  crypto/tls.dial
   11MB  5.51% 59.48%    11.50MB  5.76%  net.(*netFD).connect
   10MB  5.01% 64.50%    15.42MB  7.73%  context.WithDeadline
   9MB  4.51% 69.01%        9MB  4.51%  net.newFD (inline)
   8MB  4.01% 73.02%    10.84MB  5.43%  time.AfterFunc
   7MB  3.51% 76.53%    52.93MB 26.53%  net.(*Dialer).DialContext
   5.50MB  2.76% 79.28%     5.50MB  2.76%  context.(*cancelCtx).Done
   5MB  2.51% 81.79%    84.35MB 42.28%  main.main.func3
   5MB  2.51% 84.30%        5MB  2.51%  net.(*netFD).connect.func2
   4.50MB  2.26% 86.55%     4.50MB  2.26%  time.goFunc
   4MB  2.01% 88.56%        4MB  2.01%  crypto/tls.Client (inline)
   3.16MB  1.58% 90.14%     3.16MB  1.58%  main.main
   2.84MB  1.42% 91.56%     2.84MB  1.42%  time.startTimer
   2.50MB  1.25% 92.82%     2.50MB  1.25%  crypto/aes.(*aesCipherGCM).NewGCM
   2.50MB  1.25% 94.07%     2.50MB  1.25%  net.(*Resolver).internetAddrList.func1
   1.50MB  0.75% 94.82%     1.50MB  0.75%  crypto/tls.(*Config).Clone
   1.50MB  0.75% 95.58%     1.50MB  0.75%  crypto/aes.newCipher

显示占190.70MB的节点，占199.53MB总数的95.58%
删除了34个节点（cum结果是
//用连接做点什么
中的代码比我想象的更重要。即使在tls拨号级别，你也必须读取“body”.我现在的假设显然是错误的，那就是tls.Dial只是建立了连接，而且由于还没有发送
GET/http1.1
请求，因此不需要从线路上读取任何数据。这导致所有那些充满服务器响应的缓冲区都闲置了

，
ioutil.ReadAll（tConn）

在一行中解决了所有问题。同时，我觉得自己更聪明，也更笨。顺便说一下，在这个级别上，如果服务器响应缓慢，

ReadAll（）

可能会挂起很长一段时间。

tConn.SetReadDeadline（time.Now（）.Add（time.Second*timeout））

也解决了这个问题。

65536的工作人员实在太多了。他们每个人“开始工作”（并开始分配更多内存）可能需要一段时间，因为他们在争夺网络带宽等资源。还有，你的文件有多大，你是如何读取的？嗯，减少工作进程肯定会降低总内存使用量，只是因为goroutine的减少，但这仍然留下了为什么它会增长的谜团。即使是很多文件，我也希望最终会稳定下来。我使用了一些e goroutines在做什么，没有人坐在那里等着做什么，所以我认为他们“开始行动”马上，这个文件大约有7877302个URL，我正在使用一个bufio.Scanner发送到一个通道。为什么在开始将字符串发送到通道之前需要等待初始化所有65536个工作程序goroutine？我有点好奇为什么要说“http客户机”，因为您不使用http客户端，而是使用tls，这是https下面的一层。http包在执行Get请求时自动使用tls。http客户端不正确，我同意，对此表示抱歉。我实际上根本不需要响应中的字节，只需要成功的tls连接中的数据，所以我将级别降低到jus不要创建tls连接。我将改写问题。

package main

import (
    "crypto/tls"
    "net"
    "sync"
    "time"
)

func connectToTarget(targetString string, dialer *net.Dialer, config *tls.Config) {
    tConn, err := tls.DialWithDialer(dialer,"tcp", targetString, config)
    if err == nil {
        //do something with connection
        tConn.Close()
    }
}

func main() {
    workers := 256 * 256 //65536
    tlsConfig := &tls.Config{
        InsecureSkipVerify: true,
    }
    dialer := &net.Dialer{
        FallbackDelay: -1,
        KeepAlive:     -1,
        Timeout:       time.Duration(60) * time.Second,
    }

    targetsChan := make(chan string, workers)
    var workerDone sync.WaitGroup
    workerDone.Add(workers)

    for i := 0; i < workers; i++ {
        go func(functionWg *sync.WaitGroup, dialer *net.Dialer, tlsConfig *tls.Config, targets chan string) {
            for targetToConnect := range targets {
                connectToTarget(targetToConnect, dialer, tlsConfig)
            }

            functionWg.Done()
        }(&workerDone, dialer, tlsConfig,targetsChan)
    }

    targets := []string{} //in the actual code this reads from a file containing the list since it is large
    for _,target := range targets {
        targetsChan <- target
    }


    close(targetsChan)
    workerDone.Wait()
}

Showing nodes accounting for 329.76MB, 83.32% of 395.77MB total
Dropped 57 nodes (cum <= 1.98MB)
  flat  flat%   sum%        cum   cum%
  199.43MB 50.39% 50.39%   199.43MB 50.39%  bytes.makeSlice
  80.80MB 20.42% 70.81%   280.22MB 70.80%  crypto/tls.(*Conn).readHandshake
  28.02MB  7.08% 77.89%    28.02MB  7.08%  crypto/tls.Client (inline)
  18.01MB  4.55% 82.44%    18.01MB  4.55%  crypto/aes.(*aesCipherGCM).NewGCM
  11MB  2.78% 85.22%       11MB  2.78%  crypto/aes.newCipher
  9.50MB  2.40% 87.62%     9.50MB  2.40%  crypto/tls.(*Config).Clone
  -8MB  2.02% 85.60%    15.53MB  3.92%  crypto/tls.dial
  -5.50MB  1.39% 84.21%    -5.50MB  1.39%  net.(*netFD).connect
  -5MB  1.26% 82.94%    -5.50MB  1.39%  context.WithDeadline
  -4.50MB  1.14% 81.81%   -11.50MB  2.91%  net.(*Dialer).DialContext
  3.50MB  0.88% 82.69%     3.50MB  0.88%  net.sockaddrToTCP
  -3MB  0.76% 81.93%       -3MB  0.76%  time.AfterFunc
   2MB  0.51% 82.44%    17.53MB  4.43%  main.serverCert
   1.50MB  0.38% 82.82%        2MB  0.51%  crypto/tls.(*cipherSuiteTLS13).expandLabel
   1MB  0.25% 83.07%       18MB  4.55%  crypto/tls.aeadAESGCM
   1MB  0.25% 83.32%       10MB  2.53%  crypto/tls.aeadAESGCMTLS13
   0.50MB  0.13% 83.45%   201.93MB 51.02%  crypto/tls.(*Conn).readRecordOrCCS
   -0.50MB  0.13% 83.32%       -2MB  0.51%  net.(*sysDialer).dialSingle
   0     0% 83.32%   118.09MB 29.84%  bytes.(*Buffer).Grow (inline)
   0     0% 83.32%    81.33MB 20.55%  bytes.(*Buffer).Write
   0     0% 83.32%   199.43MB 50.39%  bytes.(*Buffer).grow
   0     0% 83.32%       11MB  2.78%  crypto/aes.NewCipher
   0     0% 83.32%    18.01MB  4.55%  crypto/cipher.NewGCM (inline)
   0     0% 83.32%    18.01MB  4.55%  crypto/cipher.newGCMWithNonceAndTagSize
   0     0% 83.32%   318.24MB 80.41%  crypto/tls.(*Conn).Handshake
   0     0% 83.32%   318.24MB 80.41%  crypto/tls.(*Conn).clientHandshake
   0     0% 83.32%     3.01MB  0.76%  crypto/tls.(*Conn).readChangeCipherSpec (inline)
   0     0% 83.32%   118.09MB 29.84%  crypto/tls.(*Conn).readFromUntil
   0     0% 83.32%   198.92MB 50.26%  crypto/tls.(*Conn).readRecord (inline)
   0     0% 83.32%    11.58MB  2.93%  crypto/tls.(*Conn).retryReadRecord
   0     0% 83.32%   154.61MB 39.06%  crypto/tls.(*clientHandshakeState).doFullHandshake
   0     0% 83.32%    22.51MB  5.69%  crypto/tls.(*clientHandshakeState).establishKeys
   0     0% 83.32%   180.12MB 45.51%  crypto/tls.(*clientHandshakeState).handshake
   0     0% 83.32%     3.01MB  0.76%  crypto/tls.(*clientHandshakeState).readFinished
   0     0% 83.32%       12MB  3.03%  crypto/tls.(*clientHandshakeStateTLS13).establishHandshakeKeys
   0     0% 83.32%   117.50MB 29.69%  crypto/tls.(*clientHandshakeStateTLS13).handshake
   0     0% 83.32%    92.92MB 23.48%  crypto/tls.(*clientHandshakeStateTLS13).readServerCertificate
   0     0% 83.32%    11.58MB  2.93%  crypto/tls.(*clientHandshakeStateTLS13).readServerParameters
   0     0% 83.32%    10.50MB  2.65%  crypto/tls.(*halfConn).setTrafficSecret
   0     0% 83.32%    15.53MB  3.92%  crypto/tls.DialWithDialer (inline)
   0     0% 83.32%        3MB  0.76%  crypto/tls.cipherAES
   0     0% 83.32%   318.24MB 80.41%  crypto/tls.dial.func2
   0     0% 83.32%    17.53MB  4.43%  main.main.func3
   0     0% 83.32%       -2MB  0.51%  net.(*sysDialer).dialSerial
   0     0% 83.32%       -2MB  0.51%  net.internetSocket
   0     0% 83.32%       -2MB  0.51%  net.socket