Warning: file_get_contents(/data/phpspider/zhask/data//catemap/3/apache-spark/5.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Apache spark 警告:跳过远程jar hdfs_Apache Spark_Hadoop_Configuration_Hdfs_Spark Submit - Fatal编程技术网

Apache spark 警告:跳过远程jar hdfs

Apache spark 警告:跳过远程jar hdfs,apache-spark,hadoop,configuration,hdfs,spark-submit,Apache Spark,Hadoop,Configuration,Hdfs,Spark Submit,我想提交一个spark作业,在hdfs上配置额外的jar,但是hadoop警告我跳过远程jar。虽然我仍然可以在hdfs上获得最终结果,但我无法获得额外远程jar的效果。如果你能给我一些建议,我将不胜感激 非常感谢, root@cluster-1-m:~# hadoop fs -ls hdfs://10.146.0.4:8020/tmp/jvm-profiler-1.0.0.jar -rwxr-xr-x 2 hdfs hadoop 7097056 2019-01-23 14:44 hd

我想提交一个spark作业,在hdfs上配置额外的jar,但是hadoop警告我跳过远程jar。虽然我仍然可以在hdfs上获得最终结果,但我无法获得额外远程jar的效果。如果你能给我一些建议,我将不胜感激

非常感谢,

root@cluster-1-m:~# hadoop fs -ls hdfs://10.146.0.4:8020/tmp/jvm-profiler-1.0.0.jar
-rwxr-xr-x   2 hdfs hadoop    7097056 2019-01-23 14:44 hdfs://10.146.0.4:8020/tmp/jvm-profiler-1.0.0.jar

root@cluster-1-m:~#/usr/lib/spark/bin/spark-submit \
--deploy-mode cluster \
--master yarn \
--conf spark.jars=hdfs://10.146.0.4:8020/tmp/jvm-profiler-1.0.0.jar \
--conf spark.driver.extraJavaOptions=-javaagent:jvm-profiler-1.0.0.jar \
--conf spark.executor.extraJavaOptions=-javaagent:jvm-profiler-1.0.0.jar \
--class com.github.ehiggs.spark.terasort.TeraSort \
/root/spark-terasort-master/target/spark-terasort-1.1-SNAPSHOT-jar-with-dependencies.jar /tmp/data/terasort_in /tmp/data/terasort_out
警告:跳过远程jarhdfs://10.146.0.4:8020/tmp/jvm-分析器-1.0.0.jar.

19/01/24 02:20:31 INFO org.apache.hadoop.warn.client.RMProxy:连接到位于cluster-1-m/10.146.0.4:8032的ResourceManager

19/01/24 02:20:31 INFO org.apache.hadoop.warn.client.AHSProxy:连接到cluster-1-m/10.146.0.4:10200上的应用程序历史记录服务器

19/01/24 02:20:34 INFO org.apache.hadoop.warn.client.api.impl.YarnClientImpl:已提交的申请书\u 1548293702222\u 0002