如何使用python proxybroker方法从本地文件夹获取和旋转代理
我尝试使用一个脚本从在线源抓取和旋转公共代理(来自defirent网站的免费代理列表),我使用proxybroker工具(python)直到现在一切都很好,但我正在寻找方法如何从我的语言环境机器中的文本或csv文件中绘制我的私有代理列表,并每x次旋转一次。 服侍 运行本地代理服务器,将传入请求分发到已找到的具有高匿名性的HTTP(S)代理池:如何使用python proxybroker方法从本地文件夹获取和旋转代理,python,proxy,proxies,proxybroker,Python,Proxy,Proxies,Proxybroker,我尝试使用一个脚本从在线源抓取和旋转公共代理(来自defirent网站的免费代理列表),我使用proxybroker工具(python)直到现在一切都很好,但我正在寻找方法如何从我的语言环境机器中的文本或csv文件中绘制我的私有代理列表,并每x次旋转一次。 服侍 运行本地代理服务器,将传入请求分发到已找到的具有高匿名性的HTTP(S)代理池: $ proxybroker serve --host 127.0.0.1 --port 8888 --types HTTP HTTPS --lvl Hig
$ proxybroker serve --host 127.0.0.1 --port 8888 --types HTTP HTTPS --lvl High
对外部代理的传入请求。”“”
import asyncio
import aiohttp
from proxybroker import Broker
async def get_pages(urls, proxy_url):
tasks = [fetch(url, proxy_url) for url in urls]
for task in asyncio.as_completed(tasks):
url, content = await task
print('Done! url: %s; content: %.100s' % (url, content))
async def fetch(url, proxy_url):
resp = None
try:
async with aiohttp.ClientSession() as session:
async with session.get(url, proxy=proxy_url) as response:
resp = await response.read()
except (aiohttp.errors.ClientOSError, aiohttp.errors.ClientResponseError,
aiohttp.errors.ServerDisconnectedError) as e:
print('Error. url: %s; error: %r' % (url, e))
finally:
return (url, resp)
def main():
host, port = '127.0.0.1', 8888 # by default
loop = asyncio.get_event_loop()
types = [('HTTP', 'High'), 'HTTPS', 'CONNECT:80']
codes = [200, 301, 302]
broker = Broker(max_tries=1, loop=loop)
# Broker.serve() also supports all arguments that are accepted
# Broker.find() method: data, countries, post, strict, dnsbl.
broker.serve(host=host, port=port, types=types, limit=10, max_tries=3,
prefer_connect=True, min_req_proxy=5, max_error_rate=0.5,
max_resp_time=8, http_allowed_codes=codes, backlog=100)
urls = ['http://httpbin.org/get', 'https://httpbin.org/get',
'http://httpbin.org/redirect/1', 'http://httpbin.org/status/404']
proxy_url = 'http://%s:%d' % (host, port)
loop.run_until_complete(get_pages(urls, proxy_url))
broker.stop()
if __name__ == '__main__':
main()