Python 如何使用线程处理多个GET请求并进行比较
我一直在试图找出如何加快速度，并获得一些线程方面的知识 我一直在尝试创建一个函数，其中我放置了两个GET请求。对于每个链接，我会刮取一些数据，然后将其保存到一个返回的列表中，然后使用该列表进行比较，以查看其中一个链接中是否出现了新链接：Python 如何使用线程处理多个GET请求并进行比较,python,multithreading,for-loop,Python,Multithreading,For Loop,我一直在试图找出如何加快速度，并获得一些线程方面的知识 我一直在尝试创建一个函数，其中我放置了两个GET请求。对于每个链接，我会刮取一些数据，然后将其保存到一个返回的列表中，然后使用该列表进行比较，以查看其中一个链接中是否出现了新链接： """ def getScrapeLinks(self, siteURL): response = requests.get( siteURL, timeout=5
"""
def getScrapeLinks(self, siteURL):
response = requests.get(
siteURL,
timeout=5
)
if response.ok:
bs4 = soup(response.text, 'lxml')
links = ['{}'.format( raw_product.find('a').get('href')) for
raw_product in bs4.find_all('div', {'class': 'test'})]
return links
"""
def pollNewProducts(self, storeClass):
# storeClass.siteCatalog = ["https://www.google.com", "https://www.facebook.com"]
LinksLists = reduce(operator.add, [getScrapeLinks(getLinks) for getLinks in storeClass.siteCatalog])
while True:
newLinksLists = reduce(operator.add,
[getScrapeLinks(getLinks) for getLinks in storeClass.siteCatalog]
)
for URL in newLinksLists:
if URL not in LinksLists:
print("New link")
print(URL)
LinksLists.append(URL)
else:
print("Sleep to see new links!")
time.sleep(random.randint(2, 4))
我想知道，如果get请求中出现了新的URL，我如何能够单独运行每个链接，并且仍然能够比较和获取新的URL？根据我对问题的回答进行调整
您应该研究异步编程。与线程不同，异步代码在同一线程中运行，但它在事件循环中运行。当Python关键字
awaitclient sends request -> ... waiting for server reply ... <- server replies
client sends request -> switch operation -> ... wait ... <- server replies
client sends request -> switch operation -> ... wait ... <- server replies
client sends request -> switch operation -> ... wait ... <- server replies
...
Successfully got url http://www.google.com.br with response of length 12188.
Successfully got url http://www.google.it with response of length 12155.
Successfully got url https://www.t.co with response of length 0.
Successfully got url http://www.msn.com with response of length 46335.
Successfully got url http://www.chinadaily.com.cn with response of length 122053.
Successfully got url https://www.google.co.in with response of length 11557.
Successfully got url https://www.google.de with response of length 12135.
Successfully got url https://www.facebook.com with response of length 115258.
Successfully got url http://www.gmw.cn with response of length 120866.
Successfully got url https://www.google.co.uk with response of length 11540.
Successfully got url https://www.google.fr with response of length 12189.
Successfully got url http://www.google.es with response of length 12163.
Successfully got url http://www.google.co.id with response of length 12169.
Successfully got url https://www.bing.com with response of length 117915.
Successfully got url https://www.instagram.com with response of length 36307.
Successfully got url https://www.google.ru with response of length 12128.
Successfully got url http://www.googleusercontent.com with response of length 1561.
Successfully got url http://www.xinhuanet.com with response of length 179254.
Successfully got url http://www.google.ca with response of length 11592.
Successfully got url http://www.accuweather.com with response of length 269.
Successfully got url http://www.googleadservices.com with response of length 1561.
Successfully got url https://www.whatsapp.com with response of length 77951.
Successfully got url http://www.cntv.cn with response of length 3139.
Successfully got url http://www.google.com.au with response of length 11579.
Successfully got url https://www.example.com with response of length 1270.
Successfully got url http://www.google.co.th with response of length 12151.
Successfully got url https://www.amazon.com with response of length 465905.
Successfully got url https://www.wikipedia.org with response of length 76240.
Successfully got url https://www.google.co.kr with response of length 12211.
Successfully got url https://www.apple.com with response of length 63322.
Successfully got url http://www.uol.com.br with response of length 333257.
Successfully got url https://www.aliexpress.com with response of length 59742.
Successfully got url http://www.sohu.com with response of length 215201.
Successfully got url https://www.google.pl with response of length 12144.
Successfully got url https://www.googleweblight.com with response of length 0.
Successfully got url https://www.cnn.com with response of length 1138392.
Successfully got url https://www.google.com.ph with response of length 11561.
Successfully got url https://www.linkedin.com with response of length 71498.
Successfully got url https://www.naver.com with response of length 176038.
Successfully got url https://www.live.com with response of length 3667.
Successfully got url https://www.twitch.tv with response of length 61599.
Successfully got url http://www.163.com with response of length 696338.
Successfully got url https://www.ebay.com with response of length 307068.
Successfully got url https://www.wordpress.com with response of length 76680.
Successfully got url https://www.wikia.com with response of length 291400.
Successfully got url http://www.chrome.com with response of length 161223.
Successfully got url https://www.twitter.com with response of length 291741.
Successfully got url https://www.stackoverflow.com with response of length 105987.
Successfully got url https://www.netflix.com with response of length 83125.
Successfully got url https://www.tumblr.com with response of length 78110.
Successfully got url http://www.doubleclick.net with response of length 129901.
Successfully got url https://www.yahoo.com with response of length 531829.
Successfully got url http://www.soso.com with response of length 174.
Successfully got url https://www.microsoft.com with response of length 187549.
Successfully got url http://www.office.com with response of length 89556.
Successfully got url http://www.alibaba.com with response of length 167978.
Successfully got url https://www.reddit.com with response of length 483295.
Successfully got url http://www.outbrain.com with response of length 24432.
Successfully got url http://www.tianya.cn with response of length 7941.
Successfully got url https://www.baidu.com with response of length 156768.
Successfully got url http://www.diply.com with response of length 3074314.
Successfully got url http://www.blogspot.com with response of length 94478.
Successfully got url http://www.popads.net with response of length 14548.
Successfully got url http://www.answers.yahoo.com with response of length 104726.
Successfully got url http://www.blogger.com with response of length 94478.
Successfully got url http://www.imgur.com with response of length 4008.
Successfully got url http://www.qq.com with response of length 244841.
Successfully got url http://www.paypal.com with response of length 45587.
Successfully got url http://www.pinterest.com with response of length 45692.
Successfully got url http://www.github.com with response of length 86917.
Successfully got url http://www.zhihu.com with response of length 31473.
Successfully got url http://www.go.com with response of length 594291.
Successfully got url http://www.fc2.com with response of length 34546.
Successfully got url https://www.amazon.de with response of length 439209.
Successfully got url https://www.youtube.com with response of length 439571.
Successfully got url http://www.bbc.co.uk with response of length 321966.
Successfully got url http://www.tmall.com with response of length 234388.
Successfully got url http://www.imdb.com with response of length 289339.
Successfully got url http://www.dropbox.com with response of length 103714.
Successfully got url http://www.bilibili.com with response of length 50959.
Successfully got url http://www.jd.com with response of length 18105.
Successfully got url http://www.yahoo.co.jp with response of length 18565.
Successfully got url https://www.amazon.co.jp with response of length 479721.
Successfully got url http://www.craigslist.org with response of length 59372.
Successfully got url https://www.360.cn with response of length 74502.
Successfully got url http://www.ok.ru with response of length 170516.
Successfully got url https://www.amazon.in with response of length 460696.
Successfully got url http://www.booking.com with response of length 408992.
Successfully got url http://www.yandex.ru with response of length 116661.
Successfully got url http://www.nicovideo.jp with response of length 107271.
Successfully got url http://www.onet.pl with response of length 720657.
Successfully got url http://www.alipay.com with response of length 21698.
Successfully got url https://www.amazon.co.uk with response of length 443607.
Successfully got url http://www.sina.com.cn with response of length 579107.
Successfully got url http://www.hao123.com with response of length 295213.
Successfully got url http://www.pixnet.net with response of length 6295.
Successfully got url http://www.coccoc.com with response of length 45822.
Successfully got url http://www.taobao.com with response of length 393128.
Successfully got url http://www.weibo.com with response of length 95482.
Successfully got url http://www.youku.com with response of length 762485.
Finalized all. ret is a list of len 100 outputs.
Took 3.899034023284912 seconds to pull 100 websites.
- 打印语句（是，包括上面代码中的语句）
- 到达距离您地理位置更远的服务器
上面的例子有上面的两个例子，因此它可以说是做你所要求的事情的最不优化的方式。但是，我相信这是一个很好的开始。我建议您使用异步编程，而不是线程。哦，您有没有任何例子说明它的外观，以及为什么要使用thst@FelipeYes当然，请稍等。
等待asyncio.gather（*[get（url）用于url中的url]）get（）resp=wait response.read（）waitget（）get（）get（）asyncio