Warning: file_get_contents(/data/phpspider/zhask/data//catemap/8/python-3.x/15.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181

Warning: file_get_contents(/data/phpspider/zhask/data//catemap/8/selenium/4.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Python 3.x 使用Chrome问题在Python3中多处理selenium_Python 3.x_Selenium_Selenium Webdriver_Multiprocessing - Fatal编程技术网

Python 3.x 使用Chrome问题在Python3中多处理selenium

Python 3.x 使用Chrome问题在Python3中多处理selenium,python-3.x,selenium,selenium-webdriver,multiprocessing,Python 3.x,Selenium,Selenium Webdriver,Multiprocessing,我正在尝试从一个网站上抓取数据,并尝试运行多个chrome浏览器来同时下载这些文件,以加快下载过程。如果使用单个窗口,则此脚本运行良好。然而,我遇到了两个问题- a) 许多浏览器窗口不会关闭。 b) 虽然程序确实会运行并下载文件一段时间,但它会在一段时间后停止。错误消息-'Error:shader\u disk\u cache.cc(238)]未能创建着色器缓存条目-2' 我的chromedriver位于'D:\401\401k' 脚本- '''Downloading 5500 forms fr

我正在尝试从一个网站上抓取数据,并尝试运行多个chrome浏览器来同时下载这些文件,以加快下载过程。如果使用单个窗口,则此脚本运行良好。然而,我遇到了两个问题-

a) 许多浏览器窗口不会关闭。 b) 虽然程序确实会运行并下载文件一段时间,但它会在一段时间后停止。错误消息-'Error:shader\u disk\u cache.cc(238)]未能创建着色器缓存条目-2'

我的chromedriver位于'D:\401\401k'

脚本-

'''Downloading 5500 forms from ERISA'''

#Import Library
from selenium import webdriver
from selenium.webdriver import ActionChains
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time, os, timeit, shutil
import pandas as pd
from multiprocessing import Pool

#Clean up download folder
if os.path.exists('D:/Form5500_Downloads'):    
    shutil.rmtree('D:/Form5500_Downloads')
    os.makedirs('D:/Form5500_Downloads')
else:
    os.makedirs('D:/Form5500_Downloads')

'''Function to download a single form using ACK ID'''
def download_form(ackid):
    #Setting Chrome preferences
    chromeOptions = webdriver.ChromeOptions()
    prefs = {"download.default_directory" : "D:/Form5500_Downloads"} #Download folder
    chromeOptions.add_experimental_option("prefs",prefs)
    path_to_chromedriver = 'D:/401/401k/chromedriver_2.35'
    browser = webdriver.Chrome(executable_path = r'D:\401\401k\chromedriver_2.35.exe', chrome_options=chromeOptions)
    browser.implicitly_wait(3) #Implicit wait untile the element appears

    # Open ERISA website
    url = 'https://www.efast.dol.gov/portal/app/disseminate?execution=e1s4#'
    browser.get(url)
    # Search for a form using ACK ID
    browser.find_element_by_css_selector('#ackId').send_keys(ackid)
    browser.find_element_by_css_selector('.ui-icon-search').click()

    #Check if the form exists - if NOT, exit the function
    try:
        browser.find_element_by_css_selector('#form\:filingTreeTable\:0\:einLnk').click() 
    except:
        browser.find_element_by_css_selector('#ackId').clear()# delete the ackid value
        browser.quit()
    #Wait until downloaded and rename using ackid
    print(ackid)
    while not os.path.exists("D:/Form5500_Downloads/filing.pdf"):
        time.sleep(5)
    os.rename("D:/Form5500_Downloads/filing.pdf","D:/Form5500_Downloads/"+ackid+".pdf")  
    browser.quit()

def main():

    '''Download'''
    # Get list of ackids from csv file
    df = pd.read_csv('D:/401/401k/F_SCH_H_2015_latest.csv',usecols=[0], nrows=10000) 
    ackid_list = df['ACK_ID'].tolist()

    if __name__ ==  '__main__':
        with Pool(10) as p:
            records = p.map(download_form, ackid_list)

main()

该问题通过以下方式解决:

    browser.stop_client()
    browser.close()
而不是

    browser.quit()
但我仍然不明白为什么这个黑客会奏效