Concurrency 为什么函数提前返回?
我刚刚开始学习围棋,并一直在努力完成整个巡回赛。最后一个练习是编辑一个web爬虫,使其并行爬行且不重复 以下是该练习的链接: 这是代码。我只改变了爬网和主功能。所以我会把它们贴出来保持整洁Concurrency 为什么函数提前返回?,concurrency,go,Concurrency,Go,我刚刚开始学习围棋,并一直在努力完成整个巡回赛。最后一个练习是编辑一个web爬虫,使其并行爬行且不重复 以下是该练习的链接: 这是代码。我只改变了爬网和主功能。所以我会把它们贴出来保持整洁 // Crawl uses fetcher to recursively crawl // pages starting with url, to a maximum of depth. var used = make(map[string]bool) var urlchan
// Crawl uses fetcher to recursively crawl
// pages starting with url, to a maximum of depth.
var used = make(map[string]bool)
var urlchan = make(chan string)
func Crawl(url string, depth int, fetcher Fetcher) {
// TODO: Fetch URLs in parallel.
// Done: Don't fetch the same URL twice.
// This implementation doesn't do either:
done := make(chan bool)
if depth <= 0 {
return
}
body, urls, err := fetcher.Fetch(url)
if err != nil {
fmt.Println(err)
return
}
fmt.Printf("\nfound: %s %q\n\n", url, body)
go func() {
for _, i := range urls {
urlchan <- i
}
done <- true
}()
for u := range urlchan {
if used[u] == false {
used[u] = true
go Crawl(u, depth-1, fetcher)
}
if <-done == true {
break
}
}
return
}
func main() {
used["http://golang.org/"] = true
Crawl("http://golang.org/", 4, fetcher)
}
只有当我试图使程序并行运行时,才会发生这种情况。如果我让它线性运行,那么所有的URL都能正确找到
注意:如果我做得不对(我的意思是并行性),那么我道歉。- 小心戈鲁廷李>
- 因为当主例程或
func返回时,所有其他go例程都将立即终止李>main()
- 您的
似乎是递归的,但它不是,这意味着它将立即返回,而不是等待其他Crawl()
例程。您知道,如果第一个由Crawl()
调用的main()
函数返回,那么Crawl()
函数将认为其任务已完成李>main()
- 您可以让
func等待最后一个main()
返回。Crawl()
软件包或sync
会有所帮助chan
- 您可能可以看看的上一个解决方案,我在几个月前做过:
var store map[string]bool func Krawl(url string, fetcher Fetcher, Urls chan []string) { body, urls, err := fetcher.Fetch(url) if err != nil { fmt.Println(err) } else { fmt.Printf("found: %s %q\n", url, body) } Urls <- urls } func Crawl(url string, depth int, fetcher Fetcher) { Urls := make(chan []string) go Krawl(url, fetcher, Urls) band := 1 store[url] = true // init for level 0 done for i := 0; i < depth; i++ { for band > 0 { band-- next := <- Urls for _, url := range next { if _, done := store[url] ; !done { store[url] = true band++ go Krawl(url, fetcher, Urls) } } } } return } func main() { store = make(map[string]bool) Crawl("http://golang.org/", 4, fetcher) }
var存储映射[string]bool func-Krawl(url字符串、抓取器抓取器、url[]字符串){ 正文,url,错误:=fetcher.Fetch(url) 如果错误!=零{ fmt.Println(错误) }否则{ fmt.Printf(“找到:%s%q\n”,url,正文) } URL 0{ 乐队-- 下一个:=是的,这是罗布·派克解释戈罗廷和梅因
var store map[string]bool func Krawl(url string, fetcher Fetcher, Urls chan []string) { body, urls, err := fetcher.Fetch(url) if err != nil { fmt.Println(err) } else { fmt.Printf("found: %s %q\n", url, body) } Urls <- urls } func Crawl(url string, depth int, fetcher Fetcher) { Urls := make(chan []string) go Krawl(url, fetcher, Urls) band := 1 store[url] = true // init for level 0 done for i := 0; i < depth; i++ { for band > 0 { band-- next := <- Urls for _, url := range next { if _, done := store[url] ; !done { store[url] = true band++ go Krawl(url, fetcher, Urls) } } } } return } func main() { store = make(map[string]bool) Crawl("http://golang.org/", 4, fetcher) }