Warning: file_get_contents(/data/phpspider/zhask/data//catemap/2/scala/16.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Scala 需要类类型,但在编码器中找不到_Scala_Apache Spark_Generics_Apache Spark Sql - Fatal编程技术网

Scala 需要类类型,但在编码器中找不到

Scala 需要类类型,但在编码器中找不到,scala,apache-spark,generics,apache-spark-sql,Scala,Apache Spark,Generics,Apache Spark Sql,我正在尝试创建一个通用代码,以便从视图中使用spark sql进行读取。当T可以是运行时传递的任何对象时,我应该能够获得T的数据集 方法 def queryStream[T](sparkSession: SparkSession, query: String, filterParams: Array[String])(implicit tag: ClassTag[T]): Dataset[T] = sparkSession.sql(query) .as(Encoders.

我正在尝试创建一个通用代码,以便从视图中使用spark sql进行读取。当T可以是运行时传递的任何对象时,我应该能够获得T的数据集

方法

def queryStream[T](sparkSession: SparkSession, query: String, filterParams: Array[String])(implicit tag: ClassTag[T]): Dataset[T] = sparkSession.sql(query)
            .as(Encoders.bean[KafkaMessage](classOf[KafkaMessage]))
            .map(f => {
              init()
              getObjectMapper.readValue(f.message, classTag[T].runtimeClass).asInstanceOf[T]
            })(Encoders.bean[T](classOf[T]))
调用:

queryStream[Student](sparkSession,"select cast(value as string) as message,"
    + "cast(key as string) as messageKey, " +
      "cast(partition as int) partition, " +
      "cast(offset as long) offset from events",null)
卡夫卡消息:

class KafkaMessage {

  @BeanProperty
  var messageKey: String = _

  @BeanProperty
  var message: String = _

  @BeanProperty
  var partition: Int = _

  @BeanProperty
  var offset: Long = _

  override def toString(): String = s"Message Key: $messageKey, Message: $message, Partition:$partition, Offset:$offset"
}