Java 如何从位于Google存储的文本文件写入Bitquery表？
我是apache beam和Google cloud dataflow的新手，我想编写一个程序，使用Java SDK从位于Google存储的文本文件中读取数据，并将数据插入Google Bigquery 我已经为这个问题编写了一个代码，但它在读取数据时显示了errorJava 如何从位于Google存储的文本文件写入Bitquery表？,java,google-bigquery,google-cloud-dataflow,apache-beam,Java,Google Bigquery,Google Cloud Dataflow,Apache Beam,我是apache beam和Google cloud dataflow的新手，我想编写一个程序，使用Java SDK从位于Google存储的文本文件中读取数据，并将数据插入Google Bigquery 我已经为这个问题编写了一个代码，但它在读取数据时显示了errorerror，错误消息：JSON表遇到了太多错误，放弃了。 public class StarterPipeline { private static final Logger LOG = LoggerFactory.getLog
error，错误消息：JSON表遇到了太多错误，放弃了。public class StarterPipeline {
private static final Logger LOG = LoggerFactory.getLogger(StarterPipeline.class);
public static void main(String[] args) {
PipelineOptions options = PipelineOptionsFactory.create();
options.setTempLocation("gs://com_example_gcptraining/assignment_1/temp");
Pipeline p = Pipeline.create(options);
PCollection<String> input = p.apply(TextIO.read().from("gs://com_example_gcptraining/assignment_1/emp_details.txt"));
PCollection<TableRow> output = input.apply(ParDo.of(new DoFn<String, TableRow>() {
@ProcessElement
public void processElement(ProcessContext c) {
StringTokenizer tokenizer = new StringTokenizer(c.element(),",");
c.output(new TableRow()
.set("id", Integer.parseInt(tokenizer.nextToken()))
.set("emp_name",tokenizer.nextToken())
.set("designation", tokenizer.nextToken())
.set("dob",Date.parseDate(tokenizer.nextToken()))
.set("contact_no",Long.parseLong(tokenizer.nextToken()))
.set("location",tokenizer.nextToken())
.set("salary",Float.parseFloat(tokenizer.nextToken())));
}
}));
String tableSpec = "gcp-training-246913:com_example_gcptraining.employee";
output.apply(BigQueryIO.writeTableRows().to(tableSpec).withCreateDisposition(CreateDisposition.CREATE_NEVER).withWriteDisposition(WriteDisposition.WRITE_APPEND));
p.run().waitUntilFinish();
}
}
公共类启动程序管道{
私有静态最终记录器LOG=LoggerFactory.getLogger（StarterPipeline.class）；
公共静态void main（字符串[]args）{
PipelineOptions=PipelineOptionsFactory.create（）；
options.setTempLocation（“gs://com\u example\u gcptraining/assignment\u 1/temp”）；
Pipeline p=Pipeline.create（选项）；
PCollection input=p.apply（TextIO.read（）。来自（“gs://com\u example\u gcptraining/assignment\u 1/emp\u details.txt”）；
PCollection输出=input.apply（ParDo.of（new DoFn（））{
@过程元素
公共void processElement（ProcessContext c）{
StringTokenizer tokenizer=新的StringTokenizer（c.element（），“，”）；
c、 输出（新表行（）
.set（“id”，Integer.parseInt（tokenizer.nextToken（）））
.set（“emp_name”，tokenizer.nextToken（））
.set（“指定”，标记器.nextToken（）
.set（“dob”，Date.parseDate（tokenizer.nextToken（）））
.set（“contact_no”，Long.parseLong（tokenizer.nextToken（）））
.set（“位置”，tokenizer.nextToken（））
.set（“工资”，Float.parseFloat（tokenizer.nextToken（））；
}
}));
String tableSpec=“gcp-training-246913:com\u example\u gcptraining.employee”；
output.apply（BigQueryIO.writeTableRows（）.to（tableSpec）.withCreateDisposition（CreateDisposition.CREATE_NEVER）.withWriteDisposition（WriteDisposition.WRITE_APPEND））；
p、 run（）.waitUntilFinish（）；
}
}
Jul 21, 2019 2:02:47 PM org.apache.beam.sdk.io.FileBasedSource getEstimatedSizeBytes
INFO: Filepattern gs://com_example_gcptraining/assignment_1/emp_details.txt matched 1 files with total size 682
Jul 21, 2019 2:02:47 PM org.apache.beam.sdk.io.FileBasedSource split
INFO: Splitting filepattern gs://com_example_gcptraining/assignment_1/emp_details.txt into bundles of size 170 took 67 ms and produced 1 files and 4 bundles
Jul 21, 2019 2:02:48 PM org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$4 getTempFilePrefix
INFO: Writing BigQuery temporary files to gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/ before loading them.
Jul 21, 2019 2:02:49 PM org.apache.beam.sdk.io.gcp.bigquery.TableRowWriter <init>
INFO: Opening TableRowWriter to gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/57558a8c-977e-478d-a44c-7863fb230af2.
Jul 21, 2019 2:02:49 PM org.apache.beam.sdk.io.gcp.bigquery.TableRowWriter <init>
INFO: Opening TableRowWriter to gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5.
Jul 21, 2019 2:02:49 PM org.apache.beam.sdk.io.gcp.bigquery.TableRowWriter <init>
INFO: Opening TableRowWriter to gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d73bb64e-cced-4016-9529-12df5bed8b9e.
Jul 21, 2019 2:02:49 PM org.apache.beam.sdk.io.gcp.bigquery.TableRowWriter <init>
INFO: Opening TableRowWriter to gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/be562c41-3aee-4c98-b1de-3e10691e3454.
Jul 21, 2019 2:02:50 PM org.apache.beam.sdk.io.gcp.bigquery.WriteTables load
INFO: Loading 4 files into {datasetId=com_example_gcptraining, projectId=gcp-training-246913, tableId=employee} using job {jobId=beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-0, location=US, projectId=gcp-training-246913}, attempt 0
Jul 21, 2019 2:02:50 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl startJob
INFO: Started BigQuery job: {jobId=beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-0, location=US, projectId=gcp-training-246913}.
bq show -j --format=prettyjson --project_id=gcp-training-246913 beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-0
Jul 21, 2019 2:02:50 PM org.apache.beam.sdk.io.gcp.bigquery.WriteTables load
INFO: Load job {jobId=beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-0, location=US, projectId=gcp-training-246913} started
Jul 21, 2019 2:02:51 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: Still waiting for BigQuery job beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-0, currently in status {"state":"RUNNING"}
bq show -j --format=prettyjson --project_id=gcp-training-246913 beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-0
Jul 21, 2019 2:02:52 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: BigQuery job {jobId=beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-0, location=US, projectId=gcp-training-246913} completed in state DONE
Jul 21, 2019 2:02:52 PM org.apache.beam.sdk.io.gcp.bigquery.WriteTables load
INFO: Load job {jobId=beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-0, location=US, projectId=gcp-training-246913} failed, will retry: {"errorResult":{"location":"gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5","message":"Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details.","reason":"invalid"},"errors":[{"location":"gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5","message":"Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details.","reason":"invalid"},{"message":"Error while reading data, error message: JSON processing encountered too many errors, giving up. Rows: 1; errors: 1; max bad: 0; error percent: 0","reason":"invalid"},{"location":"gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5","message":"Error while reading data, error message: JSON parsing error in row starting at position 0: JSON object specified for non-record field: dob","reason":"invalid"}],"state":"DONE"}
Jul 21, 2019 2:02:52 PM org.apache.beam.sdk.io.gcp.bigquery.WriteTables load
INFO: Loading 4 files into {datasetId=com_example_gcptraining, projectId=gcp-training-246913, tableId=employee} using job {jobId=beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-1, location=US, projectId=gcp-training-246913}, attempt 1
Jul 21, 2019 2:02:53 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl startJob
INFO: Started BigQuery job: {jobId=beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-1, location=US, projectId=gcp-training-246913}.
bq show -j --format=prettyjson --project_id=gcp-training-246913 beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-1
Jul 21, 2019 2:02:53 PM org.apache.beam.sdk.io.gcp.bigquery.WriteTables load
INFO: Load job {jobId=beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-1, location=US, projectId=gcp-training-246913} started
Jul 21, 2019 2:02:53 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: Still waiting for BigQuery job beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-1, currently in status {"state":"RUNNING"}
bq show -j --format=prettyjson --project_id=gcp-training-246913 beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-1
Jul 21, 2019 2:02:54 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: BigQuery job {jobId=beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-1, location=US, projectId=gcp-training-246913} completed in state DONE
Jul 21, 2019 2:02:54 PM org.apache.beam.sdk.io.gcp.bigquery.WriteTables load
INFO: Load job {jobId=beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-1, location=US, projectId=gcp-training-246913} failed, will retry: {"errorResult":{"location":"gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5","message":"Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details.","reason":"invalid"},"errors":[{"location":"gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5","message":"Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details.","reason":"invalid"},{"message":"Error while reading data, error message: JSON processing encountered too many errors, giving up. Rows: 1; errors: 1; max bad: 0; error percent: 0","reason":"invalid"},{"location":"gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5","message":"Error while reading data, error message: JSON parsing error in row starting at position 0: JSON object specified for non-record field: dob","reason":"invalid"}],"state":"DONE"}
Jul 21, 2019 2:02:54 PM org.apache.beam.sdk.io.gcp.bigquery.WriteTables load
INFO: Loading 4 files into {datasetId=com_example_gcptraining, projectId=gcp-training-246913, tableId=employee} using job {jobId=beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-2, location=US, projectId=gcp-training-246913}, attempt 2
Jul 21, 2019 2:02:54 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl startJob
INFO: Started BigQuery job: {jobId=beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-2, location=US, projectId=gcp-training-246913}.
bq show -j --format=prettyjson --project_id=gcp-training-246913 beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-2
Jul 21, 2019 2:02:54 PM org.apache.beam.sdk.io.gcp.bigquery.WriteTables load
INFO: Load job {jobId=beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-2, location=US, projectId=gcp-training-246913} started
Jul 21, 2019 2:02:54 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: Still waiting for BigQuery job beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-2, currently in status {"state":"RUNNING"}
bq show -j --format=prettyjson --project_id=gcp-training-246913 beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-2
Jul 21, 2019 2:02:55 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: BigQuery job {jobId=beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-2, location=US, projectId=gcp-training-246913} completed in state DONE
Jul 21, 2019 2:02:55 PM org.apache.beam.sdk.io.gcp.bigquery.WriteTables load
INFO: Load job {jobId=beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-2, location=US, projectId=gcp-training-246913} failed, will not retry: {"errorResult":{"location":"gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5","message":"Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details.","reason":"invalid"},"errors":[{"location":"gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5","message":"Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details.","reason":"invalid"},{"message":"Error while reading data, error message: JSON processing encountered too many errors, giving up. Rows: 1; errors: 1; max bad: 0; error percent: 0","reason":"invalid"},{"location":"gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5","message":"Error while reading data, error message: JSON parsing error in row starting at position 0: JSON object specified for non-record field: dob","reason":"invalid"}],"state":"DONE"}
Exception in thread "main" org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.RuntimeException: Failed to create load job with id prefix beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000, reached max retries: 3, last failed load job: {
"configuration" : {
"load" : {
"createDisposition" : "CREATE_NEVER",
"destinationTable" : {
"datasetId" : "com_example_gcptraining",
"projectId" : "gcp-training-246913",
"tableId" : "employee"
},
"schema" : {
"fields" : [ {
"mode" : "REQUIRED",
"name" : "id",
"type" : "INTEGER"
}, {
"name" : "emp_name",
"type" : "STRING"
}, {
"name" : "designation",
"type" : "STRING"
}, {
"name" : "dob",
"type" : "DATE"
}, {
"name" : "contact_no",
"type" : "NUMERIC"
}, {
"name" : "location",
"type" : "STRING"
}, {
"name" : "salary",
"type" : "FLOAT"
} ]
},
"sourceFormat" : "NEWLINE_DELIMITED_JSON",
"sourceUris" : [ "gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/57558a8c-977e-478d-a44c-7863fb230af2", "gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d73bb64e-cced-4016-9529-12df5bed8b9e", "gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/be562c41-3aee-4c98-b1de-3e10691e3454", "gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5" ],
"writeDisposition" : "WRITE_APPEND"
},
"jobType" : "LOAD"
},
"etag" : "FFm2QxlI5DISFk5gFxWdbw==",
"id" : "gcp-training-246913:US.beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-2",
"jobReference" : {
"jobId" : "beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-2",
"location" : "US",
"projectId" : "gcp-training-246913"
},
"kind" : "bigquery#job",
"selfLink" : "https://www.googleapis.com/bigquery/v2/projects/gcp-training-246913/jobs/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-2?location=US",
"statistics" : {
"creationTime" : "1563717774452",
"endTime" : "1563717775186",
"startTime" : "1563717774633"
},
"status" : {
"errorResult" : {
"location" : "gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5",
"message" : "Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details.",
"reason" : "invalid"
},
"errors" : [ {
"location" : "gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5",
"message" : "Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details.",
"reason" : "invalid"
}, {
"message" : "Error while reading data, error message: JSON processing encountered too many errors, giving up. Rows: 1; errors: 1; max bad: 0; error percent: 0",
"reason" : "invalid"
}, {
"location" : "gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5",
"message" : "Error while reading data, error message: JSON parsing error in row starting at position 0: JSON object specified for non-record field: dob",
"reason" : "invalid"
} ],
"state" : "DONE"
},
"user_email" : "shubham.naphade@outlook.in"
}.
at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:349)
at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:319)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:210)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:66)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:311)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:297)
at com.example.gcptraining.StarterPipeline.main(StarterPipeline.java:75)
Caused by: java.lang.RuntimeException: Failed to create load job with id prefix beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000, reached max retries: 3, last failed load job: {
"configuration" : {
"load" : {
"createDisposition" : "CREATE_NEVER",
"destinationTable" : {
"datasetId" : "com_example_gcptraining",
"projectId" : "gcp-training-246913",
"tableId" : "employee"
},
"schema" : {
"fields" : [ {
"mode" : "REQUIRED",
"name" : "id",
"type" : "INTEGER"
}, {
"name" : "emp_name",
"type" : "STRING"
}, {
"name" : "designation",
"type" : "STRING"
}, {
"name" : "dob",
"type" : "DATE"
}, {
"name" : "contact_no",
"type" : "NUMERIC"
}, {
"name" : "location",
"type" : "STRING"
}, {
"name" : "salary",
"type" : "FLOAT"
} ]
},
"sourceFormat" : "NEWLINE_DELIMITED_JSON",
"sourceUris" : [ "gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/57558a8c-977e-478d-a44c-7863fb230af2", "gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d73bb64e-cced-4016-9529-12df5bed8b9e", "gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/be562c41-3aee-4c98-b1de-3e10691e3454", "gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5" ],
"writeDisposition" : "WRITE_APPEND"
},
"jobType" : "LOAD"
},
"etag" : "FFm2QxlI5DISFk5gFxWdbw==",
"id" : "gcp-training-246913:US.beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-2",
"jobReference" : {
"jobId" : "beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-2",
"location" : "US",
"projectId" : "gcp-training-246913"
},
"kind" : "bigquery#job",
"selfLink" : "https://www.googleapis.com/bigquery/v2/projects/gcp-training-246913/jobs/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-2?location=US",
"statistics" : {
"creationTime" : "1563717774452",
"endTime" : "1563717775186",
"startTime" : "1563717774633"
},
"status" : {
"errorResult" : {
"location" : "gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5",
"message" : "Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details.",
"reason" : "invalid"
},
"errors" : [ {
"location" : "gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5",
"message" : "Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details.",
"reason" : "invalid"
}, {
"message" : "Error while reading data, error message: JSON processing encountered too many errors, giving up. Rows: 1; errors: 1; max bad: 0; error percent: 0",
"reason" : "invalid"
}, {
"location" : "gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5",
"message" : "Error while reading data, error message: JSON parsing error in row starting at position 0: JSON object specified for non-record field: dob",
"reason" : "invalid"
} ],
"state" : "DONE"
},
"user_email" : "shubham.naphade@outlook.in"
}.
at org.apache.beam.sdk.io.gcp.bigquery.WriteTables.load(WriteTables.java:308)
at org.apache.beam.sdk.io.gcp.bigquery.WriteTables.access$600(WriteTables.java:80)
at org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn.processElement(WriteTables.java:159)
2019年7月21日下午2:02:47 org.apache.beam.sdk.io.FileBasedSource getEstimatedSizeBytes
信息：Filepattern gs://com\u example\u gcptraining/assignment\u 1/emp\u details.txt匹配了1个总大小为682的文件
2019年7月21日下午2:02:47 org.apache.beam.sdk.io.FileBasedSource split
信息：将filepattern gs://com\u example\u gcptraining/assignment\u 1/emp\u details.txt拆分为大小为170的文件包花费了67毫秒，生成了1个文件和4个文件包
2019年7月21日下午2:02:48 org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$4 getTempFilePrefix
信息：加载前，将BigQuery临时文件写入gs://com\U示例\U gcptraining/assignment\U 1/temp/BigQueryWriteTemp/beam\U load\U启动器管道SHUBHAMN0721140248CBE18BB0\U d76616a7bafb416fa8e4ff9db0f8a951/。
2019年7月21日下午2:02:49 org.apache.beam.sdk.io.gcp.bigquery.TableRowWriter
信息：打开TableRowWriter至gs://com\u示例\u gcptraining/assignment\u 1/temp/BigQueryWriteTemp/beam\u load\u启动器管线shubhamn0721140248CBE18BB0\u d76616a7bafb416fa8e4ff9db0f8a951/57558a8c-977e-478d-a44c-7863fb230af2。
2019年7月21日下午2:02:49 org.apache.beam.sdk.io.gcp.bigquery.TableRowWriter
信息：打开TableRowWriter至gs://com\u示例\u gcptraining/assignment\u 1/temp/BigQueryWriteTemp/beam\u load\u启动器管线shubhamn0721140248CBE18BB0\u D76616AFB416FA8E4FF9DB0F8A951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5。
2019年7月21日下午2:02:49 org.apache.beam.sdk.io.gcp.bigquery.TableRowWriter
信息：打开TableRowWriter至gs://com\u示例\u gcptraining/assignment\u 1/temp/BigQueryWriteTemp/beam\u load\u启动器管线shubhamn0721140248CBE18BB0\u D76616AFB416FA8E4FF9DB0F8A951/d73bb64e-cced-4016-9529-12df5bed8b9e。
2019年7月21日下午2:02:49 org.apache.beam.sdk.io.gcp.bigquery.TableRowWriter
信息：打开TableRowWriter至gs://com\u示例\u gcptraining/assignment\u 1/temp/BigQueryWriteTemp/beam\u load\u启动器管线shubhamn0721140248CBE18BB0\u d76616a7bafb416fa8e4ff9db0f8a951/be562c41-3aee-4c98-b1de-3e10691e3454。
2019年7月21日下午2:02:50 org.apache.beam.sdk.io.gcp.bigquery.WriteTables加载
信息：使用作业{jobId=beam\u load\u starterpipelineshubhamn0721140248cbe18bb0\u D76616AFB416FA8E4FF9DB0F8A951\u 0d1a010156cfb76d262315de02b986ab\u 00001\u00000-0，位置=projectId=gcp-training-246913}将4个文件加载到{datasetId=com\u示例\u GCPTTraining，projectId=gcp-training-246916FA8B8E4E4FF9DB0F8A951\u 00001\u00000-0，尝试
2019年7月21日下午2:02:50 org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl startJob
信息：已启动BigQuery作业：{jobId=beam_load_StarterPipelineshubham0721140248CBE18BB0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-0，位置=US，项目=gcp-training-246913}。
bq show-j--格式=prettyjson--项目id=gcp-training-246913梁荷载启动器管线SHUBHAMN0721140248CBE18BB0\U D76616AFB416FA8E4FF9DB0F8A951\U 0d1a010156cfb76d262315de02b986ab\U 00001\U00000-0
2019年7月21日下午2:02:50 org.apache.beam.sdk.io.gcp.bigquery.WriteTables加载
信息：装载作业{jobId=beam_Load_STARTER PIPELINES SHUBHAMN0721140248CBE18BB0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-0，位置=US，项目=gcp-training-246913}已启动
2019年7月21日下午2:02:51 org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
信息：仍在等待BigQuery作业beam_load_STARTER PIPELINE SHUBHAMN0721140248CBE18BB0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-0，当前处于状态{“状态”：“正在运行”
bq show-j--格式=prettyjson--项目id=gcp-training-246913梁荷载启动器管线SHUBHAMN0721140248CBE18BB0\U D76616AFB416FA8E4FF9DB0F8A951\U 0d1a010156cfb76d262315de02b986ab\U 00001\U00000-0
2019年7月21日下午2:02:52 org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
信息：BigQuery作业{jobId=beam_load_StarterPipelineshubham0721140248CBE18BB0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-0，位置=US，项目=gcp-training-246913}已在完成状态下完成
2019年7月21日下午2:02:52 org.apache.bea