Python 如何在google cloudbuild步骤中持久化变量?

Python 如何在google cloudbuild步骤中持久化变量?,python,docker,google-cloud-platform,kubeflow-pipelines,google-ai-platform,Python,Docker,Google Cloud Platform,Kubeflow Pipelines,Google Ai Platform,我有一个cloudbuild.json,用于将管道上传到gcp kubeflow。现在,我想添加另一个步骤,在这个步骤中,我想获取最新的管道id,然后作为实验运行管道。 因此,我的主要问题是如何在后续步骤中获取管道id。我已经编写了一个小脚本来获取最新的管道id,并将其添加为从docker运行的一个步骤,但现在我不确定如何获取这个管道id 这是我的coudbuild.json { "steps": [ { "name&

我有一个cloudbuild.json,用于将管道上传到gcp kubeflow。现在,我想添加另一个步骤,在这个步骤中,我想获取最新的管道id,然后作为实验运行管道。 因此,我的主要问题是如何在后续步骤中获取管道id。我已经编写了一个小脚本来获取最新的管道id,并将其添加为从docker运行的一个步骤,但现在我不确定如何获取这个管道id

这是我的coudbuild.json

{
    "steps": [
        {
            "name": "gcr.io/cloud-builders/docker",
            "args": [
                "build",
                "-t",
                "trainer_image",
                "."
            ],
            "dir": "./trainer_image/"
        },
        {
            "name": "gcr.io/cloud-builders/docker",
            "args": [
                "build",
                "-t",
                "base_image",
                "."
            ],
            "dir": "./base_image/"
        },
        {
            "name": "gcr.io/dmgcp-pkg-internal-poc-oct-04/kfp-cli",
            "args": [
                "-c",
                "dsl-compile --py covertype_training_pipeline.py --output covertype_training_pipeline.yaml"
            ],
            "env": [
                "BASE_IMAGE=gcr.io/dmgcp-pkg-internal-poc-oct-04/base_image:test",
                "TRAINER_IMAGE=gcr.io/dmgcp-pkg-internal-poc-oct-04/trainer_image:test",
                "RUNTIME_VERSION=1.15",
                "PYTHON_VERSION=3.7",
                "COMPONENT_URL_SEARCH_PREFIX=https://raw.githubusercontent.com/kubeflow/pipelines/0.2.5/components/gcp/",
                "USE_KFP_SA=False"
            ],
            "dir": "./pipeline/"
        },
        {
            "name": "gcr.io/dmgcp-pkg-internal-poc-oct-04/kfp-cli",
            "args": [
                "-c",
                "kfp --endpoint 66df1d31e46e6510-dot-us-central2.pipelines.googleusercontent.com pipeline upload -p credit_fraud_training_test covertype_training_pipeline.yaml"
            ],
            "dir": "./pipeline/"
        },
        {
            "name": "gcr.io/cloud-builders/docker",
            "args": [
                "build",
                "-t",
                "id_image",
                "."
            ],
            "dir": "./id_image/"
        }
    ],
    "images": [
        "gcr.io/dmgcp-pkg-internal-poc-oct-04/trainer_image:test",
        "gcr.io/dmgcp-pkg-internal-poc-oct-04/base_image:test"
    ]
}

下面是我获取最新管道id的python脚本

import kfp
client = kfp.Client(host='66df1d31e46e6510-dot-us-central2.pipelines.googleusercontent.com')
pipelines = client.list_pipelines()
total_pipeline = len(pipelines.pipelines)
latest_pipeline_id = pipelines.pipelines[total_pipeline-1].id
with open('/workspace/pipeline.txt') as file:
    file.write(latest_pipeline_id)

不能在两个步骤之间持久化变量。您只能保存文件。然而,在一个步骤中,您可以重用局部变量

- name: "gcr.io/cloud-builders/docker",
  entrypoint: "bash"
  args: 
   - "-c"
   - |
      export DOCKER_VERSION=$(docker version)
      echo $$DOCKER_VERSION #Reuse local variable
      echo $$DOCKER_VERSION > DOCKER_VERSION.txt #Save the variable for the next step
- name: "gcr.io/cloud-builders/docker",
  entrypoint: "bash"
  args: 
   - "-c"
   - |
      export DOCKER_VERSION=$(cat DOCKER_VERSION.txt) #Get the value saved in file
      echo $$DOCKER_VERSION #Reuse local variable

不能在两个步骤之间持久化变量。您只能保存文件。然而,在一个步骤中,您可以重用局部变量

- name: "gcr.io/cloud-builders/docker",
  entrypoint: "bash"
  args: 
   - "-c"
   - |
      export DOCKER_VERSION=$(docker version)
      echo $$DOCKER_VERSION #Reuse local variable
      echo $$DOCKER_VERSION > DOCKER_VERSION.txt #Save the variable for the next step
- name: "gcr.io/cloud-builders/docker",
  entrypoint: "bash"
  args: 
   - "-c"
   - |
      export DOCKER_VERSION=$(cat DOCKER_VERSION.txt) #Get the value saved in file
      echo $$DOCKER_VERSION #Reuse local variable