Google cloud dataflow 从Python云函数调用dataflow flex模板时出错

我尝试使用以下代码从Python云函数调用Dataflow flex模板：

from googleapiclient.discovery import build

dataflow = build('dataflow', 'v1b3')
request = dataflow.projects().templates().launch(
    projectId=projectid,
    gcsPath=template,
    body={
        'jobName': job_name,
        'parameters': parameters,
    }
)
print(f"Start to execute Dataflow Job with name: {job_name}")
return request.execute()

如果引用经典模板，此代码也可以工作，但当引用flex模板时，它会出错，并显示以下详细信息：

Traceback (most recent call last): File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker_v2.py", line 449, in run_background_function _function_handler.invoke_user_function(event_object) File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker_v2.py", line 268, in invoke_user_function return call_user_function(request_or_event) File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker_v2.py", line 265, in call_user_function event_context.Context(**request_or_event.context)) File "/user_code/main.py", line 41, in trigger_cleaning response = call_dataflow(project_id, job_name, template, parameters) File "/user_code/utils/gcp_utils.py", line 58, in call_dataflow 'parameters': parameters, File "/env/local/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper return wrapped(*args, **kwargs) File "/env/local/lib/python3.7/site-packages/googleapiclient/http.py", line 915, in execute raise HttpError(resp, content, uri=self.uri) googleapiclient.errors.HttpError: <HttpError 400 when requesting https://dataflow.googleapis.com/v1b3/projects/concise-flame-279117/templates:launch?gcsPath=gs%3A%2F%2Filan-artefacts%2Ftemplates%2Ftemplate666.json&alt=json returned "(145ec02dfb186de2): There is no support for job type with environment version . Please try upgrading the SDK to the latest version. You can find the instructions on installing the latest SDK at https://cloud.google.com/dataflow/docs/guides/installing-beam-sdk. If that doesn't work, please contact the Cloud Dataflow team for assistance at https://cloud.google.com/dataflow/support.

---

嗯，看起来运行flex模板有一个单独的端点/调用：。这对我很有用：

数据流=生成“数据流”，“v1b3”，缓存\u发现=False 请求=dataflow.projects.locations.flexTemplates.launch projectd='my-project'， 位置='us-central1'， 身体={ “启动参数”：{ “jobName”：“testing-1234”， “参数”：{ 'some_parameter'：'hello'， }, “containerSpecGcsPath”：“gs://my storage bucket/flex\u templates/do\u something.json” } }

---

以下代码解决了该问题：

dataflow = build('dataflow', 'v1b3')
request = dataflow.projects().locations().flexTemplates().launch(
    projectId=projectid,
    location=location,
    body={
        'launchParameter': {
            'jobName': job_name,
            'containerSpecGcsPath': template,
            'parameters': parameters
        }
    }
)
print(f"Start to execute Dataflow Job with name: {job_name}")
return request.execute()

dataflow = build('dataflow', 'v1b3')
request = dataflow.projects().locations().flexTemplates().launch(
    projectId=projectid,
    location=location,
    body={
        'launchParameter': {
            'jobName': job_name,
            'containerSpecGcsPath': template,
            'parameters': parameters
        }
    }
)
print(f"Start to execute Dataflow Job with name: {job_name}")
return request.execute()

---

创建的数据流模板具有最新的apache beam[gcp]版本2.24.0。如果使用适当的服务帐户从本地python脚本运行此脚本，则会发生相同的错误。我收到了相同的错误，可能需要向Google提交一个案例。为了便于跟踪，请将您的更新和解决方案作为答案发布，并接受它。这样，您将更好地帮助有相同问题的未来成员。