Scala-Spark:从特定节点返回顶点属性
我有一张图,我想计算最大度数。特别是最大度的顶点,我想知道所有的属性。 以下是代码片段:Scala-Spark:从特定节点返回顶点属性,scala,graph,spark-graphx,Scala,Graph,Spark Graphx,我有一张图,我想计算最大度数。特别是最大度的顶点,我想知道所有的属性。 以下是代码片段: def max(a: (VertexId, Int), b: (VertexId, Int)): (VertexId, Int) = { if (a._2 > b._2) a else b } val maxDegrees : (VertexId, Int) = graphX.degrees.reduce(max) max: (a: (org.apache.spark.graphx.Ver
def max(a: (VertexId, Int), b: (VertexId, Int)): (VertexId, Int) = {
if (a._2 > b._2) a else b
}
val maxDegrees : (VertexId, Int) = graphX.degrees.reduce(max)
max: (a: (org.apache.spark.graphx.VertexId, Int), b: (org.apache.spark.graphx.VertexId, Int))(org.apache.spark.graphx.VertexId, Int)
maxDegrees: (org.apache.spark.graphx.VertexId, Int) = (2063726182,56387)
val startVertexRDD = graphX.vertices.filter{case (hash_id, (id, state)) => hash_id == maxDegrees._1}
startVertexRDD.collect()
但它返回了这个异常:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 145.0 failed 1 times, most recent failure: Lost task 0.0 in stage 145.0 (TID 5380, localhost, executor driver): scala.MatchError: (1009147972,null) (of class scala.Tuple2)
我想这就是问题所在。在这里:
val startVertexRDD = graphX.vertices.filter{case (hash_id, (id, state)) => hash_id == maxDegrees._1}
所以它试图比较一些像这样的元组
(2063726182,56387)
期待这样的事情:
(hash_id, (id, state))
引发scala.MatchError,因为正在比较(VertextId,Int)的Tuple2与(VertexId,Tuple2(id,state))的Tuple2
对此也要小心:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 145.0 failed 1 times, most recent failure: Lost task 0.0 in stage 145.0 (TID 5380, localhost, executor driver): scala.MatchError: (1009147972,null) (of class scala.Tuple2)
具体来说:
scala.MatchError: (1009147972,null)
vertice 1009147972没有计算学位,因此在进行比较时也可能会出现一些问题
希望这有帮助 我认为这就是问题所在。在这里:
val startVertexRDD = graphX.vertices.filter{case (hash_id, (id, state)) => hash_id == maxDegrees._1}
所以它试图比较一些像这样的元组
(2063726182,56387)
期待这样的事情:
(hash_id, (id, state))
引发scala.MatchError,因为正在比较(VertextId,Int)的Tuple2与(VertexId,Tuple2(id,state))的Tuple2
对此也要小心:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 145.0 failed 1 times, most recent failure: Lost task 0.0 in stage 145.0 (TID 5380, localhost, executor driver): scala.MatchError: (1009147972,null) (of class scala.Tuple2)
具体来说:
scala.MatchError: (1009147972,null)
vertice 1009147972没有计算学位,因此在进行比较时也可能会出现一些问题
希望这有帮助 我用这段代码检查了是否存在隔离节点:val vertexDegree:VertexRDD[Int]=graphX.degrees val vertexNoDegree=vertexDegree.filter{case(id,degree)=>degree==null}vertexNoDegree.isEmpty()res6:Boolean=true没有隔离节点。。。我不知道该做什么我用这段代码检查了是否有孤立的节点:val vertexDegree:VertexRDD[Int]=graphX.degrees val vertexNoDegree=vertexDegree.filter{case(id,degree)=>degree==null}vertexNoDegree.isEmpty()res6:Boolean=true没有孤立的节点。。。我不知道该怎么办