Apache spark 任务在spark中具有不可序列化的结果
我正在尝试使用cassandra驱动程序读取cassandra表。 这是代码Apache spark 任务在spark中具有不可序列化的结果,apache-spark,spark-cassandra-connector,Apache Spark,Spark Cassandra Connector,我正在尝试使用cassandra驱动程序读取cassandra表。 这是代码 val x = 1 to 2 val rdd = sc.parallelize(x) val query = "Select data from testkeyspace.testtable where id=%d" val cc = CassandraConnector(sc.getConf) val res1 = rdd.map{ it => cc.withSessio
val x = 1 to 2
val rdd = sc.parallelize(x)
val query = "Select data from testkeyspace.testtable where id=%d"
val cc = CassandraConnector(sc.getConf)
val res1 =
rdd.map{ it =>
cc.withSessionDo{ session =>
session.execute( query.format(it))
}
}
res1.take(1).foreach(println)
但是我得到的异常任务有一个不可序列化的结果
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1.0 in stage 24.0 (TID 77) had a not serializable result: com.datastax.driver.core.ArrayBackedResultSet$SinglePage
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1185)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1174)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1173)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1173)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:688)
at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1391)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
如何解决这个问题?我们转换中的不可序列化对象是从Cassandra返回的结果,这是查询结果上的一个iterable。 您通常希望将该集合具体化到RDD中 一种方法是询问该查询产生的所有记录:
session.execute( query.format(it)).all()
即使我做了所有这些,我也会得到同样的例外。“任务有一个不可序列化的结果”@Knight71,原因相同?这次是不同的数据类型
org.apache.spark.scheduler.TaskSetManager:stage 1.0(TID 2)中的Task 1.0有一个不可序列化的结果:com.datastax.driver.core.ArrayBackedRow;不重试。org.apache.spark.sparkeexception:由于阶段失败而中止作业:阶段1.0中的任务1.0(TID 2)有一个不可序列化的结果:com.datastax.driver.core.ArrayBackedRow
@Knight71然后按照breadcrumb:https://github.com/datastax/java-driver/blob/2.1/driver-core/src/main/java/com/datastax/driver/core/ArrayBackedRow.java
不可序列化。其思想是,map
操作创建的对象需要可序列化。您需要将该行
转换为某种可序列化形式:例如session.execute(query.format(it)).one().getXXXXX(“数据”)
(不确定“数据”是什么)谢谢。我试试看。