不兼容类型使用转换将JavaPairRDD连接到JavaPairDStream时出错
我试图使用transform将JavaPairRDD连接到JavaPairDStream中的RDD,但出现以下错误:不兼容类型使用转换将JavaPairRDD连接到JavaPairDStream时出错,java,apache-spark,spark-streaming,Java,Apache Spark,Spark Streaming,我试图使用transform将JavaPairRDD连接到JavaPairDStream中的RDD,但出现以下错误: incompatible types: no instance(s) of type variable(s) W exist so that org.apache.spark.api.java.JavaPairRDD<java.lang.String,scala.Tuple2<LogType,W>> conforms to org.apache.spark
incompatible types: no instance(s) of type variable(s) W exist so that org.apache.spark.api.java.JavaPairRDD<java.lang.String,scala.Tuple2<LogType,W>> conforms to org.apache.spark.api.java.JavaRDD<U>
不兼容类型:不存在类型变量W的实例,因此org.apache.spark.api.java.javapairdd符合org.apache.spark.api.java.JavaRDD
这是我的密码:
// Load full table from C*
JavaPairRDD<String,CassTableType> table = javaFunctions(
jssc
).cassandraTable(
"mykeyspace", "mytable", mapRowTo(CassTableType.class)
).keyBy(
t -> t.getLogId()
);
// Process stream
JavaPairDStream<String,LogType> logs = flumeStream.flatMapToPair(
flumeEvent -> {
List<Tuple2<String, LogType>> events = new LinkedList<>();
LogType log = LogType.parse(flumeEvent);
events.add(new Tuple2<String, LogType>(log.getLogID(), log));
return events;
}
);
// Join RDD's in logs DStream to table
JavaPairDStream<String, Tuple2<LogType, CassTableType>>
joinedRDD = logs.transform(
rdd -> rdd.join(table))
);
//从C加载完整的表*
javapairdd表=javaFunctions(
jssc
).卡桑德拉(
“mykeyspace”、“mytable”、mapRowTo(casstTableType.class)
)凯比先生(
t->t.getLogId()
);
//工艺流程
JavaPairDStream日志=flumeStream.flatMapToPair(
flumeEvent->{
列表事件=新建LinkedList();
LogType log=LogType.parse(flumeEvent);
添加(新的Tuple2(log.getLogID(),log));
返回事件;
}
);
//将日志数据流中的RDD连接到表
JavaPairDStream
joinedd=logs.transform(
rdd->rdd.join(表))
);
这是否意味着rdd是
JavaRDD
而不是javapairdd
根据我正在转换的JavaPairDStream
?如何实现此连接?使用transformToPair而不是transform解决了此问题