Warning: file_get_contents(/data/phpspider/zhask/data//catemap/4/json/13.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
如何在ApacheBeam中将嵌套json作为拼花(使用JavaSDK)?_Json_Nested_Apache Beam_Parquet - Fatal编程技术网

如何在ApacheBeam中将嵌套json作为拼花(使用JavaSDK)?

如何在ApacheBeam中将嵌套json作为拼花(使用JavaSDK)?,json,nested,apache-beam,parquet,Json,Nested,Apache Beam,Parquet,我想在apache beam中为parquet编写一个嵌套的json,但我得到一个例外。我的输入json类似于: {"event_common":{"event_type":"exposure","session_id":"1234"}} 根据官方文件,我的代码如下: public class send_data { public static void main(String[] args)

我想在apache beam中为parquet编写一个嵌套的json,但我得到一个例外。我的输入json类似于:

{"event_common":{"event_type":"exposure","session_id":"1234"}}
根据官方文件,我的代码如下:

public class send_data {

public static void main(String[] args) {

    Schema SCHEMA = new Schema.Parser().parse(
                "{\n" +
                        "  \"type\" : \"record\",\n" +
                        "  \"name\" : \"MyClass\",\n" +
                        "  \"namespace\" : \"com.test.avro\",\n" +
                        "  \"fields\" : [ {\n" +
                        "    \"name\" : \"event_common\",\n" +
                        "    \"type\" : {\n" +
                        "      \"type\" : \"record\",\n" +
                        "      \"name\" : \"event_common\",\n" +
                        "      \"fields\" : [ {\n" +
                        "        \"name\" : \"event_type\",\n" +
                        "        \"type\" : \"string\"\n" +
                        "      }, {\n" +
                        "        \"name\" : \"session_id\",\n" +
                        "        \"type\" : \"string\"\n" +
                        "      } ]\n" +
                        "    }\n" +
                        "  } ]\n" +
                        "}"
    );
    String outputPath = "./output/parquet";
    PipelineOptions options = PipelineOptionsFactory.create();
    Pipeline pipeline = Pipeline.create(options);
    pipeline.apply(TextIO.read().from("input_struct/nested_type.json"))
            .apply("Retrieve inner message JSON", ParDo.of(new DoFn<String, GenericRecord>() {
                @ProcessElement
                public void processElement(ProcessContext c) throws JsonProcessingException {
                    ObjectMapper mapper = new ObjectMapper();
                    HashMap<String, Object> map = mapper.readValue(c.element().toString(),HashMap.class);
                    GenericData.Record osRecord = new GenericData.Record(SCHEMA);
                    map.forEach((k,v)->{
                        osRecord.put(k,v);
                    });
                    c.output(osRecord);
                }
            })).setCoder(AvroGenericCoder.of(SCHEMA))
            .apply(FileIO.<GenericRecord>write()
                .via(ParquetIO.sink(SCHEMA)).to(outputPath)
                .withSuffix(".parquet"));

    System.out.println(SCHEMA);
    pipeline.run();
    System.out.println("pipleline run success");
}
这个问题困扰了我很长时间,非常感谢您的帮助

Exception in thread "main" org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.ClassCastException: java.util.LinkedHashMap cannot be cast to org.apache.avro.generic.IndexedRecord in field event_common