Apache spark Spark进程在客户端模式下无法从Kafka队列接收数据
我试图在中使用Thread client模式运行以下代码，但出现下面提到的readprocessor缓慢错误，但代码在本地模式下运行良好。任何一个指针都是非常感谢的 从Kafka队列接收数据的代码行：Apache spark Spark进程在客户端模式下无法从Kafka队列接收数据,apache-spark,yarn,spark-streaming,Apache Spark,Yarn,Spark Streaming,我试图在中使用Thread client模式运行以下代码，但出现下面提到的readprocessor缓慢错误，但代码在本地模式下运行良好。任何一个指针都是非常感谢的 从Kafka队列接收数据的代码行： JavaPairReceiverInputDStream<String, String> messages = KafkaUtils.createStream(jssc, String.class, String.class, StringDecoder.class, StringD
JavaPairReceiverInputDStream<String, String> messages = KafkaUtils.createStream(jssc, String.class, String.class, StringDecoder.class, StringDecoder.class, kafkaParams, kafkaTopicMap, StorageLevel.MEMORY_ONLY());
JavaDStream<String> lines = messages.map(new Function<Tuple2<String, String>, String>() {
public String call(Tuple2<String, String> tuple2) {
LOG.info(" &&&&&&&&&&&&&&&&&&&& Input json stream data " + tuple2._2);
return tuple2._2();
}
});
这里有什么问题？这不是错误。是一个
警告级别日志语句。请你把你的问题澄清一下好吗？
016-02-05 11:44:00 WARN DFSClient:975 - Slow ReadProcessor read fields took 30
011ms (threshold=30000ms); ack: seqno: 1960 reply: 0 reply: 0 reply: 0 downstrea
mAckTimeNanos: 1227280, targets: [DatanodeInfoWithStorage[10.0.0.245:50010,DS-a5
5d9212-3771-4936-bbe7-02035e7de148,DISK], DatanodeInfoWithStorage[10.0.0.243:500
10,DS-231b9915-c2e2-4392-b075-8a52ba1820ac,DISK], DatanodeInfoWithStorage[10.0.0
.244:50010,DS-6b8b5814-7dd7-4315-847c-b73bd375af0e,DISK]]
2016-02-05 11:44:00 INFO BlockManager:59 - Removing RDD 1954
2016-02-05 11:44:00 INFO MapPartitionsRDD:59 - Removing RDD 1955 from persisten