Warning: file_get_contents(/data/phpspider/zhask/data//catemap/3/apache-spark/5.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Scala Spark流错误:java.lang.IllegalArgumentException:java.net.URISyntaxException:绝对URI中的相对路径:_Scala_Apache Spark_Apache Kafka_Spark Streaming_Databricks - Fatal编程技术网

Scala Spark流错误:java.lang.IllegalArgumentException:java.net.URISyntaxException:绝对URI中的相对路径:

Scala Spark流错误:java.lang.IllegalArgumentException:java.net.URISyntaxException:绝对URI中的相对路径:,scala,apache-spark,apache-kafka,spark-streaming,databricks,Scala,Apache Spark,Apache Kafka,Spark Streaming,Databricks,我正在尝试从卡夫卡·阿夫罗主题中传输数据 下面是我的代码片段: val sparkStreamingContext = new StreamingContext(sc, Durations.seconds(60)) val brokers = "Broker info" val schemaRegistryURL = "URL schema registry " val subjectValueName = "topicname" + "-value" val restService = new

我正在尝试从卡夫卡·阿夫罗主题中传输数据

下面是我的代码片段:

val sparkStreamingContext = new StreamingContext(sc, Durations.seconds(60))
val brokers = "Broker info"
val schemaRegistryURL = "URL schema registry "
val subjectValueName = "topicname" + "-value"
val restService = new RestService(schemaRegistryURL)
val valueRestResponseSchema = restService.getLatestVersion(subjectValueName)
val parser = new Schema.Parser
val topicValueAvroSchema: Schema = parser.parse(valueRestResponseSchema.getSchema)

val kafkaParam = new mutable.HashMap[String, String]()
kafkaParam.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, brokers)
kafkaParam.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer")
kafkaParam.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer")
kafkaParam.put(ConsumerConfig.GROUP_ID_CONFIG, "streaming-kafka")
kafkaParam.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,  "latest")
kafkaParam.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "true")

val topicList = List("topicname")
val messageStream = KafkaUtils.createDirectStream(sparkStreamingContext, LocationStrategies.PreferConsistent, ConsumerStrategies.Subscribe[String, String](topicList, kafkaParam))
val TRANSACTION_SCHEMA: StructType = SchemaConverters.toSqlType(topicValueAvroSchema).dataType.asInstanceOf[StructType]
messageStream.foreachRDD { rdd =>
    val streamData = spark.read.schema(TRANSACTION_SCHEMA).avro(rdd.map(x => x.value()).toString())
    streamData.repartition(1).write.format("com.databricks.spark.avro").mode("Append") saveAsTable ("tablename")
    }
}
sparkStreamingContext.start()
sparkStreamingContext.awaitTermination()
但我得到下面的错误,有人可以帮助解决这个问题

java.lang.IllegalArgumentException:java.net.URISyntaxException:绝对URI中的相对路径:MapPartitionsRDD[75]位于映射位置:54

Btw,如果您有Avro数据,您应该使用字符串反序列化器,但是错误似乎是围绕
saveAsTable
调用的配置问题,或者您忘记了架构注册表上的
http
address@cricket_007架构注册表URL工作正常我可以从中打印架构,不确定这里的问题是什么`val streamData=spark.read.schema(TRANSACTION_schema).avro(rdd.map(x=>x.value()).toString()`这是导致问题的原因。好吧,我没有使用
.avro
spark函数,但我知道它们不能使用模式注册表编码的avro-dataBtw,如果您有Avro数据,则应该使用字符串反序列化器,但错误似乎是围绕
saveAsTable
调用的配置问题,或者您忘记了架构注册表上的
http
address@cricket_007架构注册表URL工作正常我可以从中打印架构,不确定这里的问题是什么`val streamData=spark.read.schema(TRANSACTION_schema).avro(rdd.map(x=>x.value()).toString()`这是导致问题的原因。好吧,我没有使用
.avro
spark函数,但我知道它们不能处理模式注册表编码的avro数据