Warning: file_get_contents(/data/phpspider/zhask/data//catemap/9/java/362.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
如何在Spark中将Kafka流转换为Json格式,以Java语言解析数据_Java_Json_Apache Spark - Fatal编程技术网

如何在Spark中将Kafka流转换为Json格式,以Java语言解析数据

如何在Spark中将Kafka流转换为Json格式,以Java语言解析数据,java,json,apache-spark,Java,Json,Apache Spark,我正在尝试将kafka流数据解析为JSON格式,以便能够解析实时传入的跨国数据以获得所需的逻辑,并进一步希望在Hbase表中更新它 1.传入的数据流将采用这种格式 2.我需要提取卡id,金额,邮政编码和交易 {“卡id”:348702330256514,“会员id”:00003749506290,“金额”: 9084849,“pos_id”:614677375609919,“postcode”:33946, “交易”:“11-02-2018 00:00:00”} 使用下面列出的代码创建Kafka

我正在尝试将kafka流数据解析为JSON格式,以便能够解析实时传入的跨国数据以获得所需的逻辑,并进一步希望在Hbase表中更新它

1.传入的数据流将采用这种格式

2.我需要提取
卡id
金额
邮政编码
交易

{“卡id”:348702330256514,“会员id”:00003749506290,“金额”: 9084849,“pos_id”:614677375609919,“postcode”:33946, “交易”:“11-02-2018 00:00:00”}

使用下面列出的代码创建Kafka Consumer,但不确定如何通过RDD将其进一步处理到Jason中

    Logger.getLogger("org").setLevel(Level.OFF);
    Logger.getLogger("akka").setLevel(Level.OFF);

    SparkConf sparkConf = new SparkConf().setAppName("KafkaSparkStreamingDemo").setMaster("local");

    JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, Durations.seconds(10));

    Map<String, Object> kafkaParams = new HashMap<>();
    kafkaParams.put("bootstrap.servers", "100.xx.xxx.xxx:9092");
    kafkaParams.put("key.deserializer", StringDeserializer.class);
    kafkaParams.put("value.deserializer", StringDeserializer.class);
    kafkaParams.put("group.id", "groupkafkaspark2");
    kafkaParams.put("auto.offset.reset", "earliest");
    kafkaParams.put("enable.auto.commit", true);

    Collection<String> topics = Arrays.asList("transactions-topic-verified");

    JavaDStream<ConsumerRecord<String, String>> stream = KafkaUtils.createDirectStream(jssc,
            LocationStrategies.PreferConsistent(),
            ConsumerStrategies.<String, String>Subscribe(topics, kafkaParams));
Logger.getLogger(“org”).setLevel(Level.OFF);
Logger.getLogger(“akka”).setLevel(Level.OFF);
SparkConf SparkConf=new SparkConf().setAppName(“KafkaSparkStreamingDemo”).setMaster(“本地”);
JavaStreamingContext jssc=新的JavaStreamingContext(sparkConf,Durations.seconds(10));
Map kafkaParams=新HashMap();
kafkaParams.put(“bootstrap.servers”,“100.xx.xxx.xxx:9092”);
kafkaParams.put(“key.deserializer”,StringDeserializer.class);
kafkaParams.put(“value.deserializer”,StringDeserializer.class);
kafkaParams.put(“group.id”、“groupkafkapark2”);
kafkaParams.put(“自动偏移重置”、“最早”);
kafkaParams.put(“enable.auto.commit”,true);
集合主题=Arrays.asList(“已验证的事务主题”);
JavaDStream=KafkaUtils.createDirectStream(jssc,
LocationStrategies.PreferConsistent(),
订阅(主题,卡夫卡帕兰));

我需要获取上述4个字段,然后通过查看类似数据上预先创建的hbase表对其进行处理。

这可以通过使用JSONObject和JSONPasrer类来完成