Warning: file_get_contents(/data/phpspider/zhask/data//catemap/2/python/308.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Python 如何使多个MLFlow并行运行?_Python_Pyspark_Parallel Processing_Mlflow - Fatal编程技术网

Python 如何使多个MLFlow并行运行?

Python 如何使多个MLFlow并行运行?,python,pyspark,parallel-processing,mlflow,Python,Pyspark,Parallel Processing,Mlflow,我不太熟悉Python中的并行化,在尝试并行地在多个训练折叠上训练模型时遇到了一个错误。以下是我的代码的简化版本: def train_test_model(fold): # here I train the model etc... # now I want to save the parameters and metrics with mlflow.start_run(): mlflow.log_param("run_name&qu

我不太熟悉Python中的并行化,在尝试并行地在多个训练折叠上训练模型时遇到了一个错误。以下是我的代码的简化版本:

def train_test_model(fold):
    # here I train the model etc...
    
    # now I want to save the parameters and metrics
    with mlflow.start_run():
        mlflow.log_param("run_name", run_name)
        mlflow.log_param("modeltype", modeltype)
        # and so on...

if __name__=="__main__":
    pool = ThreadPool(processes = num_trials)
    # run folds in parallel
    pool.map(lambda fold:train_test_model(fold), folds)
我得到以下错误:

Exception: Run with UUID 23e9bb6d22674a518e48af9c51252860 is already active. To start a new run, first end the current run with mlflow.end_run(). To start a nested run, call start_run with nested=True

说明
mlflow.start\u run()
启动新的运行并使其处于活动状态,这是我的问题的根源。每个线程为其相应折叠启动一个MLFlow运行,并使其处于活动状态,同时我需要并行运行,即所有运行都处于活动状态(?)并保存相应折叠的参数/度量。我怎样才能解决这个问题呢?

我找到了一个解决方案,也许对其他人有用。您可以在此处通过代码示例查看详细信息: