Python 在远程服务器上设置并发Selenium webdriver调用时出现的问题
当我在个人计算机上运行以下代码时,它运行正常:Python 在远程服务器上设置并发Selenium webdriver调用时出现的问题,python,multithreading,selenium,selenium-webdriver,Python,Multithreading,Selenium,Selenium Webdriver,当我在个人计算机上运行以下代码时,它运行正常: from selenium import webdriver from concurrent.futures import ThreadPoolExecutor def open_driver(num): options = webdriver.firefox.options.Options() options.headless = True print("Starting") with webdriver.Fir
from selenium import webdriver
from concurrent.futures import ThreadPoolExecutor
def open_driver(num):
options = webdriver.firefox.options.Options()
options.headless = True
print("Starting")
with webdriver.Firefox() as driver:
pass
print("Ending")
return num
if __name__ == '__main__':
with ThreadPoolExecutor(6) as executor:
results = executor.map(open_driver,list(range(6)))
print(list(results))
输出:
StartingStarting
Starting
Starting
StartingStarting
Ending
Ending
Ending
Ending
Ending
Ending
[0, 1, 2, 3, 4, 5]
但是,当我尝试在远程DigitalOcean服务器(运行Ubuntu 18.04版)上运行相同的程序时,会产生以下错误:
Traceback (most recent call last):
File "multi_drive.py", line 32, in <module>
print(list(results))
File "/usr/lib/python3.6/concurrent/futures/_base.py", line 586, in result_iterator
yield fs.pop().result()
File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
result = self.fn(*self.args, **self.kwargs)
File "multi_drive.py", line 24, in open_driver
with webdriver.Firefox(options = options, executable_path = '/home/ubuntu/.local/bin/geckodriver') as browser:
File "/home/ubuntu/testing/venv/lib/python3.6/site-packages/selenium/webdriver/firefox/webdriver.py", line 174, in __init__
keep_alive=True)
File "/home/ubuntu/testing/venv/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "/home/ubuntu/testing/venv/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/home/ubuntu/testing/venv/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/home/ubuntu/testing/venv/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: invalid argument: can't kill an exited process
程序按预期运行(减去并发):
当我将池计数设置为2时,程序也会运行而不会出现错误,因此可能会出现一定程度的并发性(对于没有webdriver调用的代码,max_workers=6可以正常运行)
max_workers=3似乎是程序开始失败的阈值
(不确定是否相关,但此处的“cat/proc/sys/kernel/threads max”生成7677)
这是服务器的限制吗?(软件或硬件?)如何检查?否则,这里到底发生了什么??任何帮助都将不胜感激。多谢各位
with ThreadPoolExecutor(1) as executor:
Starting
Finally
Starting
Finally
Starting
Finally
Starting
Finally
Starting
Finally
Starting
Finally
[0, 1, 2, 3, 4, 5]