Scala 需要类类型,但在编码器中找不到
我正在尝试创建一个通用代码,以便从视图中使用spark sql进行读取。当T可以是运行时传递的任何对象时,我应该能够获得T的数据集 方法Scala 需要类类型,但在编码器中找不到,scala,apache-spark,generics,apache-spark-sql,Scala,Apache Spark,Generics,Apache Spark Sql,我正在尝试创建一个通用代码,以便从视图中使用spark sql进行读取。当T可以是运行时传递的任何对象时,我应该能够获得T的数据集 方法 def queryStream[T](sparkSession: SparkSession, query: String, filterParams: Array[String])(implicit tag: ClassTag[T]): Dataset[T] = sparkSession.sql(query) .as(Encoders.
def queryStream[T](sparkSession: SparkSession, query: String, filterParams: Array[String])(implicit tag: ClassTag[T]): Dataset[T] = sparkSession.sql(query)
.as(Encoders.bean[KafkaMessage](classOf[KafkaMessage]))
.map(f => {
init()
getObjectMapper.readValue(f.message, classTag[T].runtimeClass).asInstanceOf[T]
})(Encoders.bean[T](classOf[T]))
调用:
queryStream[Student](sparkSession,"select cast(value as string) as message,"
+ "cast(key as string) as messageKey, " +
"cast(partition as int) partition, " +
"cast(offset as long) offset from events",null)
卡夫卡消息:
class KafkaMessage {
@BeanProperty
var messageKey: String = _
@BeanProperty
var message: String = _
@BeanProperty
var partition: Int = _
@BeanProperty
var offset: Long = _
override def toString(): String = s"Message Key: $messageKey, Message: $message, Partition:$partition, Offset:$offset"
}