Sbt 如何在Play2.2.3项目中使用Spark 1.2.0,因为它在NoSuchMethodError:akka.util.Helpers中失败?
您是否曾经在Play framework方面遇到过问题?在我的例子中,首先我已经在一个jar中构建了所有组件:Sbt 如何在Play2.2.3项目中使用Spark 1.2.0,因为它在NoSuchMethodError:akka.util.Helpers中失败?,sbt,apache-spark,playframework-2.2,typesafe-activator,Sbt,Apache Spark,Playframework 2.2,Typesafe Activator,您是否曾经在Play framework方面遇到过问题?在我的例子中,首先我已经在一个jar中构建了所有组件:spark-assebmly-1.2.0-hadoop2.4.0.jar,spark可以在一个shell中完美地工作。但有两个问题: 我应该在Play_项目中使用这个组装的Spark_罐子吗?如何使用??因为我试图将它转移到lib_directiry,但这无助于提供任何Spark_导入 如果我定义Spark库,比如:“org.apache.Spark”%%“Spark核心”%%“1.2.
spark-assebmly-1.2.0-hadoop2.4.0.jar
,spark可以在一个shell中完美地工作。但有两个问题:
“org.apache.Spark”%%“Spark核心”%%“1.2.0”
Build.scala
:
val-appDependencies=Seq(
jdbc
,“org.apache.spark”%%“spark流媒体”%%“1.2.0”
,“org.apache.spark”%%“spark核心”%%“1.2.0”
,“org.apache.spark”%%“spark sql”%%“1.2.0”
TestEntity.scala
:
包模型
导入org.apache.spark.SparkContext
导入org.apache.spark.SparkContext_
导入org.apache.hadoop.conf.Configuration
导入模型.SparkMain
导入org.apache.spark.rdd.rdd
导入org.apache.spark.SparkContext_
对象测试性{
val TestEntityPath=“/home/t/PROD/dict/TestEntity.txt”
val TestEntitySpark=SparkMain.sc.textFile(TestEntityPath,4).cache
val TestEntityData=TestEntitySpark.flatMap(u.split(“,”))
def getFive():Seq[字符串]={
println(“TestEntity.getFive”)
TestEntityData.take(5)
}
}
SparkMain.scala
:
包模型
导入org.apache.spark.SparkContext
导入org.apache.spark.SparkContext_
导入org.apache.hadoop.conf.Configuration
导入org.apache.spark.rdd.rdd
导入org.apache.spark.SparkContext_
导入org.apache.spark.streaming.{Seconds,StreamingContext}
导入StreamingContext_
导入org.apache.spark.sql.SQLContext
导入org.apache.spark.SparkConf
物体火花{
val driverPort=8080
val driverHost=“localhost”
val conf=new SparkConf(false)//跳过加载外部设置
.setMaster(“本地[4]”//使用足够的线程在本地运行
.setAppName(“firstSparkApp”)
.set(“spark.logConf”、“true”)
.set(“spark.driver.port”,s“$driverPort”)
.set(“spark.driver.host”,s“$driverHost”)
.set(“spark.akka.logLifecycleEvents”,“true”)
val sc=新的SparkContext(配置)
}
和控制器代码,使用Spark的东西:
def测试=操作{
隐式请求=>{
val chk=TestEntity.getFive
好的(“它起作用”)
}
}
..在运行时a中出现以下错误:
[info] o.a.s.SparkContext - Spark configuration:
spark.akka.logLifecycleEvents=true
spark.app.name=firstSparkApp
spark.driver.host=localhost
spark.driver.port=8080
spark.logConf=true
spark.master=local[4]
[warn] o.a.s.u.Utils - Your hostname, uisprk resolves to a loopback address: 127.0.1.1; using 10.0.2.15 instead (on interface eth0)
[warn] o.a.s.u.Utils - Set SPARK_LOCAL_IP if you need to bind to another address
[info] o.a.s.SecurityManager - Changing view acls to: t
[info] o.a.s.SecurityManager - Changing modify acls to: t
[info] o.a.s.SecurityManager - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(t); users with modify permissions: Set(t)
[error] application -
! @6l039e8d5 - Internal server error, for (GET) [/ui] ->
play.api.Application$$anon$1: Execution exception[[RuntimeException: java.lang.NoSuchMethodError: akka.util.Helpers$.ConfigOps(Lcom/typesafe/config/Config;)Lcom/typesafe/config/Config;]]
at play.api.Application$class.handleError(Application.scala:293) ~[play_2.10-2.2.3.jar:2.2.3]
at play.api.DefaultApplication.handleError(Application.scala:399) [play_2.10-2.2.3.jar:2.2.3]
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$13$$anonfun$apply$1.applyOrElse(PlayDefaultUpstreamHandler.scala:166) [play_2.10-2.2.3.jar:2.2.3]
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$13$$anonfun$apply$1.applyOrElse(PlayDefaultUpstreamHandler.scala:163) [play_2.10-2.2.3.jar:2.2.3]
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33) [scala-library-2.10.4.jar:na]
at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:185) [scala-library-2.10.4.jar:na]
Caused by: java.lang.RuntimeException: java.lang.NoSuchMethodError: akka.util.Helpers$.ConfigOps(Lcom/typesafe/config/Config;)Lcom/typesafe/config/Config;
at play.api.mvc.ActionBuilder$$anon$1.apply(Action.scala:314) ~[play_2.10-2.2.3.jar:2.2.3]
at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:109) ~[play_2.10-2.2.3.jar:2.2.3]
at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:109) ~[play_2.10-2.2.3.jar:2.2.3]
at play.utils.Threads$.withContextClassLoader(Threads.scala:18) ~[play_2.10-2.2.3.jar:2.2.3]
at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:108) ~[play_2.10-2.2.3.jar:2.2.3]
at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:107) ~[play_2.10-2.2.3.jar:2.2.3]
Caused by: java.lang.NoSuchMethodError: akka.util.Helpers$.ConfigOps(Lcom/typesafe/config/Config;)Lcom/typesafe/config/Config;
at akka.remote.RemoteSettings.<init>(RemoteSettings.scala:48) ~[akka-remote_2.10-2.3.4-spark.jar:na]
at akka.remote.RemoteActorRefProvider.<init>(RemoteActorRefProvider.scala:114) ~[akka-remote_2.10-2.3.4-spark.jar:na]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.7.0_72]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) ~[na:1.7.0_72]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.7.0_72]
at java.lang.reflect.Constructor.newInstance(Constructor.java:526) ~[na:1.7.0_72]
[info]o.a.s.SparkContext-Spark配置:
spark.akka.logLifecycleEvents=true
spark.app.name=firstSparkApp
spark.driver.host=localhost
火花驱动器端口=8080
spark.logConf=true
spark.master=本地[4]
[警告]o.a.s.u.Utils-您的主机名,uisprk解析为环回地址:127.0.1.1;改为使用10.0.2.15(在接口eth0上)
[警告]o.a.s.u.Utils-如果需要绑定到其他地址,请设置SPARK_LOCAL_IP
[信息]o.a.s.SecurityManager-将视图ACL更改为:t
[信息]o.a.s.SecurityManager-将修改ACL更改为:t
[信息]o.a.s.SecurityManager-SecurityManager:禁用身份验证;禁用ui ACL;具有查看权限的用户:设置(t);具有修改权限的用户:设置(t)
[错误]应用程序-
!@6l039e8d5-内部服务器错误,用于(GET)[/ui]->
play.api.Application$$anon$1:执行异常[[RuntimeException:java.lang.NoSuchMethodError:akka.util.Helpers$.ConfigOps(Lcom/typesafe/config/config;)Lcom/typesafe/config/config;]]
在play.api.Application$class.handleError(Application.scala:293)~[play_2.10-2.2.3.jar:2.2.3]
play.api.DefaultApplication.handleError(Application.scala:399)[play_2.10-2.2.3.jar:2.2.3]
在play.core.server.netty.playdaultUpstreamHandler$$anonfun$13$$anonfun$apply$1.applyOrElse(playdaultUpstreamHandler.scala:166)[play_2.10-2.2.3.jar:2.2.3]
在play.core.server.netty.playdaultUpstreamHandler$$anonfun$13$$anonfun$apply$1.applyOrElse(playdaultUpstreamHandler.scala:163)[play_2.10-2.2.3.jar:2.2.3]
在scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)[scala-library-2.10.4.jar:na]
在scala.util.Failure$$anonfun$recover$1.apply(Try.scala:185)[scala-library-2.10.4.jar:na]
原因:java.lang.RuntimeException:java.lang.NoSuchMethodError:akka.util.Helpers$.ConfigOps(Lcom/typesafe/config/config;)Lcom/typesafe/config/config;
在play.api.mvc.ActionBuilder$$anon$1.apply(Action.scala:314)~[play_2.10-2.2.3.jar:2.2.3]
在play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:109)~[play_2.10-2.2.3.jar:2.2.3]
在play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:109)~[play_2.10-2.2.3.jar:2.2.3]
在play.utils.Threads$.withContextClassLoader(Threads.scala:18)~[play_2.10-2.2.3.jar:2.2.3]
在play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:108)~[play_2.10-2.2.3.jar:2.2.3]
在play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:107)~[play_2.10-2.2.3.jar:2.2.3]
原因:java.lang.NoSuchMethodError:akka.util.Helpers$.ConfigOps(Lcom/typesafe/config/config;)Lcom/typesafe/config;
在akka.remote.RemoteSettings.(RemoteSettings.scala:48)~[akka-remote_2.10-2.3.4-spark.jar:na]
在akka.remote.RemoteActorRefProvider.(RemoteActorRefProvider.scala:114)~[akka-remote_2.10-2.3.4-spark.jar:na]
在sun.reflect.NativeConstructorAccessorImpl.newInstance0(本机方法)~[na:1.7.0\u 72]
在sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)~[na:1.7.0\U 72]
在sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)~[na:1.7.072]
在java.lang.reflect.Constructor.newInstance(C