Python 随机搜索CV和sklearn.externals.joblib.externals.loky.process\u executor.TerminatedWorkerError问题

我正在使用RandomizedSearchCV为我的XGBClassifier学习者找到最佳配置，并遇到TerminatedWorkerError错误

Python代码

rs = RandomizedSearchCV(XGBClassifier(), param_grid, cv=5, scoring='f1_micro', n_jobs=16, verbose=1, n_iter=50000)
rs.fit(X_tr, y_tr)

控制台输出

Fitting 5 folds for each of 50000 candidates, totalling 250000 fits
[Parallel(n_jobs=16)]: Using backend LokyBackend with 16 concurrent workers.
[Parallel(n_jobs=16)]: Done  18 tasks      | elapsed:   30.2s
[Parallel(n_jobs=16)]: Done 168 tasks      | elapsed:  2.4min
[Parallel(n_jobs=16)]: Done 418 tasks      | elapsed:  4.4min
[Parallel(n_jobs=16)]: Done 768 tasks      | elapsed:  8.7min
[Parallel(n_jobs=16)]: Done 1218 tasks      | elapsed: 14.5min
[Parallel(n_jobs=16)]: Done 1768 tasks      | elapsed: 19.9min
[Parallel(n_jobs=16)]: Done 2418 tasks      | elapsed: 27.8min
[Parallel(n_jobs=16)]: Done 3168 tasks      | elapsed: 37.1min
[Parallel(n_jobs=16)]: Done 4018 tasks      | elapsed: 47.0min
[Parallel(n_jobs=16)]: Done 4968 tasks      | elapsed: 57.9min
[Parallel(n_jobs=16)]: Done 6018 tasks      | elapsed: 69.6min
[Parallel(n_jobs=16)]: Done 7168 tasks      | elapsed: 82.3min
[Parallel(n_jobs=16)]: Done 8418 tasks      | elapsed: 96.4min
[Parallel(n_jobs=16)]: Done 9768 tasks      | elapsed: 110.2min
[Parallel(n_jobs=16)]: Done 11218 tasks      | elapsed: 126.7min
[Parallel(n_jobs=16)]: Done 12768 tasks      | elapsed: 143.4min
[Parallel(n_jobs=16)]: Done 14418 tasks      | elapsed: 162.3min
[Parallel(n_jobs=16)]: Done 16168 tasks      | elapsed: 183.0min
[Parallel(n_jobs=16)]: Done 18018 tasks      | elapsed: 203.9min
exception calling callback for <Future at 0x1980abf4908 state=finished raised TerminatedWorkerError>
Traceback (most recent call last):
  File "C:\Users\Huy.Nguyen\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\externals\loky\_base.py", line 625, in _invoke_callbacks
    callback(self)
  File "C:\Users\Huy.Nguyen\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\parallel.py", line 309, in __call__
    self.parallel.dispatch_next()
  File "C:\Users\Huy.Nguyen\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\parallel.py", line 731, in dispatch_next
    if not self.dispatch_one_batch(self._original_iterator):
  File "C:\Users\Huy.Nguyen\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\parallel.py", line 759, in dispatch_one_batch
    self._dispatch(tasks)
  File "C:\Users\Huy.Nguyen\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\parallel.py", line 716, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
  File "C:\Users\Huy.Nguyen\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\_parallel_backends.py", line 510, in apply_async
    future = self._workers.submit(SafeFunction(func))
  File "C:\Users\Huy.Nguyen\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\externals\loky\reusable_executor.py", line 151, in submit
    fn, *args, **kwargs)
  File "C:\Users\Huy.Nguyen\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\externals\loky\process_executor.py", line 1022, in submit
    raise self._flags.broken
sklearn.externals.joblib.externals.loky.process_executor.TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.
Traceback (most recent call last):
  File "main.py", line 117, in <module>
    rs.fit(X_tr, y_tr)
  File "C:\Users\Huy.Nguyen\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\model_selection\_search.py", line 680, in fit
    self._run_search(evaluate_candidates)
  File "C:\Users\Huy.Nguyen\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\model_selection\_search.py", line 1460, in _run_search
    random_state=self.random_state))
  File "C:\Users\Huy.Nguyen\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\model_selection\_search.py", line 669, in evaluate_candidates
    cv.split(X, y, groups)))
  File "C:\Users\Huy.Nguyen\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\parallel.py", line 930, in __call__
    self.retrieve()
  File "C:\Users\Huy.Nguyen\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\parallel.py", line 833, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "C:\Users\Huy.Nguyen\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\_parallel_backends.py", line 521, in wrap_future_result
    return future.result(timeout=timeout)
  File "C:\Users\Huy.Nguyen\AppData\Local\Programs\Python\Python36\lib\concurrent\futures\_base.py", line 432, in result
    return self.__get_result()
  File "C:\Users\Huy.Nguyen\AppData\Local\Programs\Python\Python36\lib\concurrent\futures\_base.py", line 384, in __get_result
    raise self._exception
  File "C:\Users\Huy.Nguyen\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\externals\loky\_base.py", line 625, in _invoke_callbacks
    callback(self)
  File "C:\Users\Huy.Nguyen\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\parallel.py", line 309, in __call__
    self.parallel.dispatch_next()
  File "C:\Users\Huy.Nguyen\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\parallel.py", line 731, in dispatch_next
    if not self.dispatch_one_batch(self._original_iterator):
  File "C:\Users\Huy.Nguyen\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\parallel.py", line 759, in dispatch_one_batch
    self._dispatch(tasks)
  File "C:\Users\Huy.Nguyen\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\parallel.py", line 716, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
  File "C:\Users\Huy.Nguyen\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\_parallel_backends.py", line 510, in apply_async
    future = self._workers.submit(SafeFunction(func))
  File "C:\Users\Huy.Nguyen\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\externals\loky\reusable_executor.py", line 151, in submit
    fn, *args, **kwargs)
  File "C:\Users\Huy.Nguyen\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\externals\loky\process_executor.py", line 1022, in submit
    raise self._flags.broken
sklearn.externals.joblib.externals.loky.process_executor.TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.

为50000名候选人每人试衣5次，共计250000次
[并行（n_jobs=16）]：使用后端LokyBackend和16个并发工作者。
[并行（n_作业=16）]：完成18项任务|已用时间：30.2秒
[并行（n_作业=16）]：完成168项任务|耗时：2.4分钟
[并行（n_作业=16）]：完成418项任务|耗时：4.4分钟
[并行（n_作业=16）]：完成768项任务|耗时：8.7分钟
[并行（n_作业=16）]：完成1218项任务|耗时：14.5分钟
[并行（n_作业=16）]：完成1768项任务|耗时：19.9分钟
[并行（n_作业=16）]：完成2418项任务|耗时：27.8分钟
[并行（n_作业=16）]：完成3168项任务|耗时：37.1分钟
[并行（n_作业=16）]：完成4018项任务|耗时：47.0分钟
[并行（n_作业=16）]：完成4968项任务|耗时：57.9分钟
[并行（n_作业=16）]：完成6018项任务|耗时：69.6分钟
[并行（n_作业=16）]：完成7168项任务|耗时：82.3分钟
[并行（n_作业=16）]：完成8418项任务|耗时：96.4分钟
[并行（n_作业=16）]：完成9768项任务|耗时：110.2分钟
[并行（n_作业=16）]：完成11218项任务|耗时：126.7分钟
[并行（n_作业=16）]：完成12768项任务|耗时：143.4分钟
[并行（n_作业=16）]：完成14418项任务|耗时：162.3分钟
[并行（n_作业=16）]：完成16168项任务|耗时：183.0分钟
[并行（n_作业=16）]：完成18018项任务|耗时：203.9分钟
调用的回调时发生异常
回溯（最近一次呼叫最后一次）：
文件“C:\Users\Huy.Nguyen\AppData\Local\Programs\Python36\lib\site packages\sklearn\externals\joblib\externals\loky\\ u base.py”，第625行，在调用调用中
回拨（自我）
文件“C:\Users\Huy.Nguyen\AppData\Local\Programs\Python36\lib\site packages\sklearn\externals\joblib\parallel.py”，第309行，在调用中__
self.parallel.dispatch_next（）
文件“C:\Users\Huy.Nguyen\AppData\Local\Programs\Python36\lib\site packages\sklearn\externals\joblib\parallel.py”，第731行，在dispatch\u next
如果不是self.dispatch\u一批（self.\u原始\u迭代器）：
文件“C:\Users\Huy.Nguyen\AppData\Local\Programs\Python36\lib\site packages\sklearn\externals\joblib\parallel.py”，第759行，一批发送
自我分配（任务）
文件“C:\Users\Huy.Nguyen\AppData\Local\Programs\Python36\lib\site packages\sklearn\externals\joblib\parallel.py”，第716行，在
作业=self.\u后端.apply\u异步（批处理，回调=cb）
文件“C:\Users\Huy.Nguyen\AppData\Local\Programs\Python36\lib\site packages\sklearn\externals\joblib\u parallel\u backends.py”，第510行，在apply\u async中
future=自我.\u workers.submit（安全功能（func））
文件“C:\Users\Huy.Nguyen\AppData\Local\Programs\Python36\lib\site packages\sklearn\externals\joblib\externals\loky\reusables\u executor.py”，第151行，提交
fn，*args，**kwargs）
文件“C:\Users\Huy.Nguyen\AppData\Local\Programs\Python36\lib\site packages\sklearn\externals\joblib\externals\loky\process\u executor.py”，第1022行，提交
举起自己的旗帜
sklearn.externals.joblib.externals.loky.process_executor.TerminatedWorkerError:由执行器管理的工作进程意外终止。这可能是由于调用函数时出现分段错误或内存使用过多导致操作系统杀死工作进程造成的。
回溯（最近一次呼叫最后一次）：
文件“main.py”，第117行，在
rs.fit（X_tr，y_tr）
文件“C:\Users\Huy.Nguyen\AppData\Local\Programs\Python\36\lib\site packages\sklearn\model\u selection\\u search.py”，第680行，fit
自我评估。运行搜索（评估候选人）
文件“C:\Users\Huy.Nguyen\AppData\Local\Programs\Python36\lib\site packages\sklearn\model\u selection\u search.py”，第1460行，在运行搜索中
随机状态=自身。随机状态）
文件“C:\Users\Huy.Nguyen\AppData\Local\Programs\Python\36\lib\site packages\sklearn\model\u selection\u search.py”，第669行，位于评估候选项中
cv.分割（X、y、组）
文件“C:\Users\Huy.Nguyen\AppData\Local\Programs\Python36\lib\site packages\sklearn\externals\joblib\parallel.py”，第930行，在调用中__
self.retrieve（）
文件“C:\Users\Huy.Nguyen\AppData\Local\Programs\Python36\lib\site packages\sklearn\externals\joblib\parallel.py”，第833行，检索
self.\u output.extend（job.get（timeout=self.timeout））
文件“C:\Users\Huy.Nguyen\AppData\Local\Programs\Python36\lib\site packages\sklearn\externals\joblib\u parallel\u backends.py”，第521行，在wrap\u future\u result中
返回future.result（超时=超时）
结果文件“C:\Users\Huy.Nguyen\AppData\Local\Programs\Python\Python36\lib\concurrent\futures\u base.py”，第432行
返回self.\u获取\u结果（）
文件“C:\Users\Huy.Nguyen\AppData\Local\Programs\Python\Python36\lib\concurrent\futures\\u base.py”，第384行，在“获取结果”中
提出自己的意见
文件“C:\Users\Huy.Nguyen\AppData\Local\Programs\Python36\lib\site packages\sklearn\externals\joblib\externals\loky\\ u base.py”，第625行，在调用调用中
回拨（自我）
文件“C:\Users\Huy.Nguyen\AppData\Local\Programs\Python36\lib\site packages\sklearn\externals\joblib\parallel.py”，第309行，在调用中__
self.parallel.dispatch_next（）
文件“C:\Users\Huy.Nguyen\AppData\Local\Programs\Python36\lib\site packages\sklearn\externals\joblib\parallel.py”，第731行，在dispatch\u next
如果不是self.dispatch\u一批（self.\u原始\u迭代器）：
文件“C:\Users\Huy.Nguyen\AppData\Local\Programs\Python36\lib\site packages\sklearn\externals\joblib\parallel.py”，第759行，一批发送
自我分配（任务）
文件“C:\Users\Huy.Nguyen\AppData\Local\Programs\Python36\lib\site packages\sklearn\externals\joblib\parallel.py”，第716行，在