Python 如何使多个MLFlow并行运行?
我不太熟悉Python中的并行化,在尝试并行地在多个训练折叠上训练模型时遇到了一个错误。以下是我的代码的简化版本:Python 如何使多个MLFlow并行运行?,python,pyspark,parallel-processing,mlflow,Python,Pyspark,Parallel Processing,Mlflow,我不太熟悉Python中的并行化,在尝试并行地在多个训练折叠上训练模型时遇到了一个错误。以下是我的代码的简化版本: def train_test_model(fold): # here I train the model etc... # now I want to save the parameters and metrics with mlflow.start_run(): mlflow.log_param("run_name&qu
def train_test_model(fold):
# here I train the model etc...
# now I want to save the parameters and metrics
with mlflow.start_run():
mlflow.log_param("run_name", run_name)
mlflow.log_param("modeltype", modeltype)
# and so on...
if __name__=="__main__":
pool = ThreadPool(processes = num_trials)
# run folds in parallel
pool.map(lambda fold:train_test_model(fold), folds)
我得到以下错误:
Exception: Run with UUID 23e9bb6d22674a518e48af9c51252860 is already active. To start a new run, first end the current run with mlflow.end_run(). To start a nested run, call start_run with nested=True
说明
mlflow.start\u run()
启动新的运行并使其处于活动状态,这是我的问题的根源。每个线程为其相应折叠启动一个MLFlow运行,并使其处于活动状态,同时我需要并行运行,即所有运行都处于活动状态(?)并保存相应折叠的参数/度量。我怎样才能解决这个问题呢?我找到了一个解决方案,也许对其他人有用。您可以在此处通过代码示例查看详细信息: