如何在ApacheBeam中将嵌套json作为拼花(使用JavaSDK)?
我想在apache beam中为parquet编写一个嵌套的json,但我得到一个例外。我的输入json类似于:如何在ApacheBeam中将嵌套json作为拼花(使用JavaSDK)?,json,nested,apache-beam,parquet,Json,Nested,Apache Beam,Parquet,我想在apache beam中为parquet编写一个嵌套的json,但我得到一个例外。我的输入json类似于: {"event_common":{"event_type":"exposure","session_id":"1234"}} 根据官方文件,我的代码如下: public class send_data { public static void main(String[] args)
{"event_common":{"event_type":"exposure","session_id":"1234"}}
根据官方文件,我的代码如下:
public class send_data {
public static void main(String[] args) {
Schema SCHEMA = new Schema.Parser().parse(
"{\n" +
" \"type\" : \"record\",\n" +
" \"name\" : \"MyClass\",\n" +
" \"namespace\" : \"com.test.avro\",\n" +
" \"fields\" : [ {\n" +
" \"name\" : \"event_common\",\n" +
" \"type\" : {\n" +
" \"type\" : \"record\",\n" +
" \"name\" : \"event_common\",\n" +
" \"fields\" : [ {\n" +
" \"name\" : \"event_type\",\n" +
" \"type\" : \"string\"\n" +
" }, {\n" +
" \"name\" : \"session_id\",\n" +
" \"type\" : \"string\"\n" +
" } ]\n" +
" }\n" +
" } ]\n" +
"}"
);
String outputPath = "./output/parquet";
PipelineOptions options = PipelineOptionsFactory.create();
Pipeline pipeline = Pipeline.create(options);
pipeline.apply(TextIO.read().from("input_struct/nested_type.json"))
.apply("Retrieve inner message JSON", ParDo.of(new DoFn<String, GenericRecord>() {
@ProcessElement
public void processElement(ProcessContext c) throws JsonProcessingException {
ObjectMapper mapper = new ObjectMapper();
HashMap<String, Object> map = mapper.readValue(c.element().toString(),HashMap.class);
GenericData.Record osRecord = new GenericData.Record(SCHEMA);
map.forEach((k,v)->{
osRecord.put(k,v);
});
c.output(osRecord);
}
})).setCoder(AvroGenericCoder.of(SCHEMA))
.apply(FileIO.<GenericRecord>write()
.via(ParquetIO.sink(SCHEMA)).to(outputPath)
.withSuffix(".parquet"));
System.out.println(SCHEMA);
pipeline.run();
System.out.println("pipleline run success");
}
这个问题困扰了我很长时间,非常感谢您的帮助
Exception in thread "main" org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.ClassCastException: java.util.LinkedHashMap cannot be cast to org.apache.avro.generic.IndexedRecord in field event_common