Warning: file_get_contents(/data/phpspider/zhask/data//catemap/9/google-cloud-platform/3.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Google cloud platform 如何在beam管道中设置GOOGLE_应用程序_凭据以解决拒绝访问错误?_Google Cloud Platform_Google Cloud Dataflow - Fatal编程技术网

Google cloud platform 如何在beam管道中设置GOOGLE_应用程序_凭据以解决拒绝访问错误?

Google cloud platform 如何在beam管道中设置GOOGLE_应用程序_凭据以解决拒绝访问错误?,google-cloud-platform,google-cloud-dataflow,Google Cloud Platform,Google Cloud Dataflow,我试图使用数据流管道从文件中插入bigquery中的数据。下面的代码给出了拒绝访问错误 我还通过环境变量设置了应用程序凭据。我开始出错,上面写着 应用程序默认凭据不可用 我需要帮助来解决这个问题。提前谢谢。请查找以下代码: from __future__ import absolute_import import argparse import logging import re import apache_beam as beam import os from apache_beam.opti

我试图使用数据流管道从文件中插入bigquery中的数据。下面的代码给出了拒绝访问错误

我还通过环境变量设置了应用程序凭据。我开始出错,上面写着

应用程序默认凭据不可用

我需要帮助来解决这个问题。提前谢谢。请查找以下代码:

from __future__ import absolute_import
import argparse
import logging
import re
import apache_beam as beam
import os
from apache_beam.options.pipeline_options import PipelineOptions

class DataIngestion:

    def parse_method(self, string_input):        
        values = re.split(",",
                          re.sub('\r\n', '', re.sub(u'"', '', string_input)))
        row = dict(
            zip(('state', 'gender', 'year', 'name', 'number', 'created_date'),
                values))
        return row

def run(argv=None):
    """The main function which creates the pipeline and runs it."""
    os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "LocalPath\FileName.json"
    parser = argparse.ArgumentParser()
    parser.add_argument(
        '--input',
        dest='input',
        required=False,
        help='Input file to read. This can be a local file or '
        'a file in a Google Storage Bucket.',
        default='gs://python-dataflow-example/data_files/head_usa_names.csv')
    parser.add_argument('--output',
                        dest='output',
                        required=False,
                        help='Output BQ table to write results to.',
                        default='lake.usa_names')

    # Parse arguments from the command line.
    known_args, pipeline_args = parser.parse_known_args(argv)
    pipeline_args.extend([
        '--runner=DirectRunner',
        '--project=projectID',
        '--staging_location=staging_location',
        '--temp_location=temp_location',
    ])

    data_ingestion = DataIngestion()    
    p = beam.Pipeline(options=PipelineOptions(pipeline_args))
    (p     
     | 'Read from a File' >> beam.io.ReadFromText(known_args.input, skip_header_lines=1)     
     | 'String To BigQuery Row' >>
     beam.Map(lambda s: data_ingestion.parse_method(s))
     | 'Write to BigQuery' >> beam.io.Write(
         beam.io.BigQuerySink(           
             known_args.output,            
             schema='state:STRING,gender:STRING,year:STRING,name:STRING,'
             'number:STRING,created_date:STRING',            
             create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,            
             write_disposition=beam.io.BigQueryDisposition.WRITE_TRUNCATE)))
    p.run().wait_until_finish()
if __name__ == '__main__':
    logging.getLogger().setLevel(logging.INFO)
    run()

您可以删除代码以在beam作业中设置凭据,并在运行作业之前尝试以下命令

gcloud auth application-default login
上面的命令将设置作业可见的GOOGLE_应用程序_凭据