Go 如何使此脚本多线程
我使用代码的方式是从一个文件中传递一个url列表,并用我的代码运行每个url,代码使用bufio,因此我可以通过管道从文件中传递url 命令:cat test2.txt |/mygofileGo 如何使此脚本多线程,go,Go,我使用代码的方式是从一个文件中传递一个url列表,并用我的代码运行每个url,代码使用bufio,因此我可以通过管道从文件中传递url 命令:cat test2.txt |/mygofile https://beap.adss.yahoo.com/ https://id.answers.yahoo.com/search https://brokenurl https://id.answers.yahoo.com/KnowledgeSearchService https://id.answers.
https://beap.adss.yahoo.com/
https://id.answers.yahoo.com/search
https://brokenurl
https://id.answers.yahoo.com/KnowledgeSearchService
https://id.answers.yahoo.com/question/nextQuestion
示例URL的内容
package main
import (
"log"
"bufio"
"bytes"
"fmt"
"net/http"
"os"
"strings"
"crypto/tls"
)
func main() {
tr := http.DefaultTransport.(*http.Transport).Clone()
tr.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
client := &http.Client{Transport: tr}
passedUrl := bufio.NewScanner(os.Stdin)
var pUrl string
for passedUrl.Scan() {
fmt.Println(passedUrl.Text())
if strings.Contains(passedUrl.Text(), "://"){
pUrl = passedUrl.Text()
} else {
pUrl = "https://" + passedUrl.Text()
}
body := "<?xml version=\"1.0\"?>"
req, err := http.NewRequest("POST", pUrl, bytes.NewBuffer([]byte(body)))
if err != nil {
log.Println(err)
}
req.Header.Add("Content-Type", "application/xml; charset=utf-8")
resp, err := client.Do(req)
if err != nil {
log.Println(err)
continue // stop here and process next item
}
defer resp.Body.Close()
}
}
问题是,当我给出一个巨大的URL列表时,代码非常慢。是否有任何方法可以使代码多线程化或以简单的方式更快地创建工作进程池。worker goroutine从通道获取要处理的URL,并在通道关闭时退出。主goroutine将URL提供给频道,并在完成时关闭频道
tr := http.DefaultTransport.(*http.Transport).Clone()
tr.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
client := &http.Client{Transport: tr}
var wg sync.WaitGroup
work := make(chan string)
for i := 0; i < 10; i++ {
wg.Add()
go func() {
defer wg.Done()
for pUrl := range work {
fmt.Println(purl)
if !strings.Contains(purl, "://") {
pUrl = "https://" + passedUrl.Text()
}
body := "<?xml version=\"1.0\"?>"
req, err := http.NewRequest("POST", pUrl, strings.NewReader(body))
if err != nil {
log.Println(err)
continue
}
req.Header.Add("Content-Type", "application/xml; charset=utf-8")
resp, err := client.Do(req)
if err != nil {
log.Println(err)
continue // stop here and process next item
}
// do something with resp
resp.Body.Close()
}
}()
}
passedUrl := bufio.NewScanner(os.Stdin)
for passedUrl.Scan() {
work <- passedUrl.Text()
}
close(work) // signal workers to exit
wg.Wait() // wait for workers to exit
我想向它传递大约1000个URL..一个关于术语的小提示:那不是一个脚本。Go是一种编译语言。脚本没有编译。@CeriseLimón你能告诉我怎么做吗?@CeriseLimón我希望它现在看起来不错?我编辑了问题中的代码。。你能帮助我如何让脚本成为多线程的吗?我不知道围棋中使用的术语是否正确。。命令行参数./mygofile.go:23:12:调用wg时没有足够的参数。Add具有want int./mygofile.go:27:26:未定义:purl./mygofile.go:28:35:未定义:purl./mygofile.go:29:38:未定义:passedurl是否有人可以帮助将1作为参数传递给wg.Add