Google cloud platform 如何在beam管道中设置GOOGLE_应用程序_凭据以解决拒绝访问错误?
我试图使用数据流管道从文件中插入bigquery中的数据。下面的代码给出了拒绝访问错误 我还通过环境变量设置了应用程序凭据。我开始出错,上面写着 应用程序默认凭据不可用 我需要帮助来解决这个问题。提前谢谢。请查找以下代码:Google cloud platform 如何在beam管道中设置GOOGLE_应用程序_凭据以解决拒绝访问错误?,google-cloud-platform,google-cloud-dataflow,Google Cloud Platform,Google Cloud Dataflow,我试图使用数据流管道从文件中插入bigquery中的数据。下面的代码给出了拒绝访问错误 我还通过环境变量设置了应用程序凭据。我开始出错,上面写着 应用程序默认凭据不可用 我需要帮助来解决这个问题。提前谢谢。请查找以下代码: from __future__ import absolute_import import argparse import logging import re import apache_beam as beam import os from apache_beam.opti
from __future__ import absolute_import
import argparse
import logging
import re
import apache_beam as beam
import os
from apache_beam.options.pipeline_options import PipelineOptions
class DataIngestion:
def parse_method(self, string_input):
values = re.split(",",
re.sub('\r\n', '', re.sub(u'"', '', string_input)))
row = dict(
zip(('state', 'gender', 'year', 'name', 'number', 'created_date'),
values))
return row
def run(argv=None):
"""The main function which creates the pipeline and runs it."""
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "LocalPath\FileName.json"
parser = argparse.ArgumentParser()
parser.add_argument(
'--input',
dest='input',
required=False,
help='Input file to read. This can be a local file or '
'a file in a Google Storage Bucket.',
default='gs://python-dataflow-example/data_files/head_usa_names.csv')
parser.add_argument('--output',
dest='output',
required=False,
help='Output BQ table to write results to.',
default='lake.usa_names')
# Parse arguments from the command line.
known_args, pipeline_args = parser.parse_known_args(argv)
pipeline_args.extend([
'--runner=DirectRunner',
'--project=projectID',
'--staging_location=staging_location',
'--temp_location=temp_location',
])
data_ingestion = DataIngestion()
p = beam.Pipeline(options=PipelineOptions(pipeline_args))
(p
| 'Read from a File' >> beam.io.ReadFromText(known_args.input, skip_header_lines=1)
| 'String To BigQuery Row' >>
beam.Map(lambda s: data_ingestion.parse_method(s))
| 'Write to BigQuery' >> beam.io.Write(
beam.io.BigQuerySink(
known_args.output,
schema='state:STRING,gender:STRING,year:STRING,name:STRING,'
'number:STRING,created_date:STRING',
create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
write_disposition=beam.io.BigQueryDisposition.WRITE_TRUNCATE)))
p.run().wait_until_finish()
if __name__ == '__main__':
logging.getLogger().setLevel(logging.INFO)
run()
您可以删除代码以在beam作业中设置凭据,并在运行作业之前尝试以下命令
gcloud auth application-default login
上面的命令将设置作业可见的GOOGLE_应用程序_凭据