Python H20Context运行Pybarkling with Spark 2.1时出错
我在AWS EMR集群上尝试运行Pybarkling脚本时遇到此错误。当我下载SparkingWater2.1.8并从PySparklingShell运行它时，我可以让一切正常工作。然而，spark submit似乎不起作用 错误：Python H20Context运行Pybarkling with Spark 2.1时出错,python,python-3.x,h2o,sparkling-water,Python,Python 3.x,H2o,Sparkling Water,我在AWS EMR集群上尝试运行Pybarkling脚本时遇到此错误。当我下载SparkingWater2.1.8并从PySparklingShell运行它时，我可以让一切正常工作。然而，spark submit似乎不起作用 错误： NameError: name 'H2OContext' is not defined 本人谨此陈辞： spark-submit --packages ai.h2o:sparkling-water-core_2.11:2.1.7,ai.h2o:sparkling-
NameError: name 'H2OContext' is not defined
spark-submit --packages ai.h2o:sparkling-water-core_2.11:2.1.7,ai.h2o:sparkling-water-examples_2.11:2.1.7 --conf spark.dynamicAllocation.enabled=false spark.py
from pysparkling import *
hc = H2OContext.getOrCreate(sc)
#!/usr/bin/env bash
# install conda (conda 4.2 defaults to pythone35)
wget --quiet http://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh -O ~/anaconda.sh \
&& /bin/bash ~/anaconda.sh -b -p $HOME/conda
echo -e '\nexport PATH=$HOME/conda/bin:$PATH' >> $HOME/.bashrc && source $HOME/.bashrc
# install packages
conda install -y ipython jupyter
# needed for PySparkling
conda install requests
conda install six
conda install future
conda install tabulate
# install pysparkling
pip install h2o
# pip install pysparkling
pip install h2o_pysparkling_2.1
[hadoop@ip-172-31-32-30 test]$ spark-submit --packages ai.h2o:sparkling-water-core_2.11:2.1.7,ai.h2o:sparkling-water-examples_2.11:2.1.7 --conf spark.dynamicAllocation.enabled=false spark.py
Ivy Default Cache set to: /home/hadoop/.ivy2/cache
The jars for the packages stored in: /home/hadoop/.ivy2/jars
:: loading settings :: url = jar:file:/usr/lib/spark/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
ai.h2o#sparkling-water-core_2.11 added as a dependency
ai.h2o#sparkling-water-examples_2.11 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
confs: [default]
found ai.h2o#sparkling-water-core_2.11;2.1.7 in central
found ai.h2o#h2o-genmodel;3.10.4.7 in central
found net.sf.opencsv#opencsv;2.3 in central
found ai.h2o#deepwater-backend-api;1.0.2 in central
found com.google.guava#guava;19.0 in central
found ai.h2o#h2o-core;3.10.4.7 in central
found joda-time#joda-time;2.3 in central
found gov.nist.math#jama;1.0.3 in central
found org.javassist#javassist;3.18.2-GA in central
found org.apache.commons#commons-math3;3.3 in central
found commons-io#commons-io;2.4 in central
found ai.h2o#google-analytics-java;1.1.2-H2O-CUSTOM in central
found org.apache.httpcomponents#httpclient;4.1 in central
found org.apache.httpcomponents#httpcore;4.1 in central
found commons-logging#commons-logging;1.1.1 in central
found commons-codec#commons-codec;1.4 in central
found org.eclipse.jetty.aggregate#jetty-servlet;8.1.17.v20150415 in central
found org.eclipse.jetty#jetty-plus;8.1.17.v20150415 in central
found org.eclipse.jetty.orbit#javax.transaction;1.1.1.v201105210645 in central
found org.eclipse.jetty#jetty-webapp;8.1.17.v20150415 in central
found org.eclipse.jetty#jetty-xml;8.1.17.v20150415 in central
found org.eclipse.jetty#jetty-util;8.1.17.v20150415 in central
found org.eclipse.jetty#jetty-servlet;8.1.17.v20150415 in central
found org.eclipse.jetty#jetty-security;8.1.17.v20150415 in central
found org.eclipse.jetty#jetty-server;8.1.17.v20150415 in central
found org.eclipse.jetty.orbit#javax.servlet;3.0.0.v201112011016 in central
found org.eclipse.jetty#jetty-continuation;8.1.17.v20150415 in central
found org.eclipse.jetty#jetty-http;8.1.17.v20150415 in central
found org.eclipse.jetty#jetty-io;8.1.17.v20150415 in central
found org.eclipse.jetty#jetty-jndi;8.1.17.v20150415 in central
found org.eclipse.jetty.orbit#javax.mail.glassfish;1.4.1.v201005082020 in central
found org.eclipse.jetty.orbit#javax.activation;1.1.0.v201105071233 in central
found com.github.rwl#jtransforms;2.4.0 in central
found ai.h2o#h2o-jaas-pam;3.10.4.7 in central
found org.kohsuke#libpam4j;1.8 in central
found net.java.dev.jna#jna;4.0.0 in central
found log4j#log4j;1.2.15 in central
found com.google.code.gson#gson;2.3.1 in central
found commons-lang#commons-lang;2.6 in central
found ai.h2o#reflections;0.9.11-h2o-custom in central
found com.google.code.findbugs#jsr305;3.0.0 in central
found ai.h2o#h2o-algos;3.10.4.7 in central
found ai.h2o#h2o-web;3.10.4.7 in central
found ai.h2o#h2o-avro-parser;3.10.4.7 in central
found ai.h2o#h2o-parquet-parser;3.10.4.7 in central
found ai.h2o#h2o-orc-parser;3.10.4.7 in central
found ai.h2o#h2o-scala_2.11;3.10.4.7 in central
found ai.h2o#h2o-persist-hdfs;3.10.4.7 in central
found net.java.dev.jets3t#jets3t;0.6.1 in central
found commons-httpclient#commons-httpclient;3.1 in central
found ai.h2o#h2o-persist-s3;3.10.4.7 in central
found com.amazonaws#aws-java-sdk-s3;1.10.47 in central
found com.amazonaws#aws-java-sdk-kms;1.10.47 in central
found com.amazonaws#aws-java-sdk-core;1.10.47 in central
found commons-logging#commons-logging;1.1.3 in central
found org.apache.httpcomponents#httpclient;4.3.6 in central
found org.apache.httpcomponents#httpcore;4.3.3 in central
found commons-codec#commons-codec;1.6 in central
found joda-time#joda-time;2.8.1 in central
found ai.h2o#sparkling-water-repl_2.11;2.1.7 in central
found org.joda#joda-convert;1.7 in central
found ai.h2o#sparkling-water-examples_2.11;2.1.7 in central
found ai.h2o#sparkling-water-ml_2.11;2.1.7 in central
downloading https://repo1.maven.org/maven2/ai/h2o/sparkling-water-core_2.11/2.1.7/sparkling-water-core_2.11-2.1.7.jar ...
[SUCCESSFUL ] ai.h2o#sparkling-water-core_2.11;2.1.7!sparkling-water-core_2.11.jar (56ms)
downloading https://repo1.maven.org/maven2/ai/h2o/sparkling-water-examples_2.11/2.1.7/sparkling-water-examples_2.11-2.1.7.jar ...
[SUCCESSFUL ] ai.h2o#sparkling-water-examples_2.11;2.1.7!sparkling-water-examples_2.11.jar (15ms)
downloading https://repo1.maven.org/maven2/ai/h2o/h2o-genmodel/3.10.4.7/h2o-genmodel-3.10.4.7.jar ...
[SUCCESSFUL ] ai.h2o#h2o-genmodel;3.10.4.7!h2o-genmodel.jar (7ms)
downloading https://repo1.maven.org/maven2/ai/h2o/h2o-core/3.10.4.7/h2o-core-3.10.4.7.jar ...
[SUCCESSFUL ] ai.h2o#h2o-core;3.10.4.7!h2o-core.jar (129ms)
downloading https://repo1.maven.org/maven2/ai/h2o/h2o-algos/3.10.4.7/h2o-algos-3.10.4.7.jar ...
[SUCCESSFUL ] ai.h2o#h2o-algos;3.10.4.7!h2o-algos.jar (35ms)
downloading https://repo1.maven.org/maven2/ai/h2o/h2o-web/3.10.4.7/h2o-web-3.10.4.7.jar ...
[SUCCESSFUL ] ai.h2o#h2o-web;3.10.4.7!h2o-web.jar (512ms)
downloading https://repo1.maven.org/maven2/ai/h2o/h2o-scala_2.11/3.10.4.7/h2o-scala_2.11-3.10.4.7.jar ...
[SUCCESSFUL ] ai.h2o#h2o-scala_2.11;3.10.4.7!h2o-scala_2.11.jar (4ms)
downloading https://repo1.maven.org/maven2/ai/h2o/h2o-persist-hdfs/3.10.4.7/h2o-persist-hdfs-3.10.4.7.jar ...
[SUCCESSFUL ] ai.h2o#h2o-persist-hdfs;3.10.4.7!h2o-persist-hdfs.jar (2ms)
downloading https://repo1.maven.org/maven2/ai/h2o/h2o-persist-s3/3.10.4.7/h2o-persist-s3-3.10.4.7.jar ...
[SUCCESSFUL ] ai.h2o#h2o-persist-s3;3.10.4.7!h2o-persist-s3.jar (2ms)
downloading https://repo1.maven.org/maven2/ai/h2o/sparkling-water-repl_2.11/2.1.7/sparkling-water-repl_2.11-2.1.7.jar ...
[SUCCESSFUL ] ai.h2o#sparkling-water-repl_2.11;2.1.7!sparkling-water-repl_2.11.jar (4ms)
downloading https://repo1.maven.org/maven2/ai/h2o/h2o-jaas-pam/3.10.4.7/h2o-jaas-pam-3.10.4.7.jar ...
[SUCCESSFUL ] ai.h2o#h2o-jaas-pam;3.10.4.7!h2o-jaas-pam.jar (2ms)
downloading https://repo1.maven.org/maven2/ai/h2o/sparkling-water-ml_2.11/2.1.7/sparkling-water-ml_2.11-2.1.7.jar ...
[SUCCESSFUL ] ai.h2o#sparkling-water-ml_2.11;2.1.7!sparkling-water-ml_2.11.jar (10ms)
:: resolution report :: resolve 6024ms :: artifacts dl 802ms
:: modules in use:
ai.h2o#deepwater-backend-api;1.0.2 from central in [default]
ai.h2o#google-analytics-java;1.1.2-H2O-CUSTOM from central in [default]
ai.h2o#h2o-algos;3.10.4.7 from central in [default]
ai.h2o#h2o-avro-parser;3.10.4.7 from central in [default]
ai.h2o#h2o-core;3.10.4.7 from central in [default]
ai.h2o#h2o-genmodel;3.10.4.7 from central in [default]
ai.h2o#h2o-jaas-pam;3.10.4.7 from central in [default]
ai.h2o#h2o-orc-parser;3.10.4.7 from central in [default]
ai.h2o#h2o-parquet-parser;3.10.4.7 from central in [default]
ai.h2o#h2o-persist-hdfs;3.10.4.7 from central in [default]
ai.h2o#h2o-persist-s3;3.10.4.7 from central in [default]
ai.h2o#h2o-scala_2.11;3.10.4.7 from central in [default]
ai.h2o#h2o-web;3.10.4.7 from central in [default]
ai.h2o#reflections;0.9.11-h2o-custom from central in [default]
ai.h2o#sparkling-water-core_2.11;2.1.7 from central in [default]
ai.h2o#sparkling-water-examples_2.11;2.1.7 from central in [default]
ai.h2o#sparkling-water-ml_2.11;2.1.7 from central in [default]
ai.h2o#sparkling-water-repl_2.11;2.1.7 from central in [default]
com.amazonaws#aws-java-sdk-core;1.10.47 from central in [default]
com.amazonaws#aws-java-sdk-kms;1.10.47 from central in [default]
com.amazonaws#aws-java-sdk-s3;1.10.47 from central in [default]
com.github.rwl#jtransforms;2.4.0 from central in [default]
com.google.code.findbugs#jsr305;3.0.0 from central in [default]
com.google.code.gson#gson;2.3.1 from central in [default]
com.google.guava#guava;19.0 from central in [default]
commons-codec#commons-codec;1.6 from central in [default]
commons-httpclient#commons-httpclient;3.1 from central in [default]
commons-io#commons-io;2.4 from central in [default]
commons-lang#commons-lang;2.6 from central in [default]
commons-logging#commons-logging;1.1.3 from central in [default]
gov.nist.math#jama;1.0.3 from central in [default]
joda-time#joda-time;2.8.1 from central in [default]
log4j#log4j;1.2.15 from central in [default]
net.java.dev.jets3t#jets3t;0.6.1 from central in [default]
net.java.dev.jna#jna;4.0.0 from central in [default]
net.sf.opencsv#opencsv;2.3 from central in [default]
org.apache.commons#commons-math3;3.3 from central in [default]
org.apache.httpcomponents#httpclient;4.3.6 from central in [default]
org.apache.httpcomponents#httpcore;4.3.3 from central in [default]
org.eclipse.jetty#jetty-continuation;8.1.17.v20150415 from central in [default]
org.eclipse.jetty#jetty-http;8.1.17.v20150415 from central in [default]
org.eclipse.jetty#jetty-io;8.1.17.v20150415 from central in [default]
org.eclipse.jetty#jetty-jndi;8.1.17.v20150415 from central in [default]
org.eclipse.jetty#jetty-plus;8.1.17.v20150415 from central in [default]
org.eclipse.jetty#jetty-security;8.1.17.v20150415 from central in [default]
org.eclipse.jetty#jetty-server;8.1.17.v20150415 from central in [default]
org.eclipse.jetty#jetty-servlet;8.1.17.v20150415 from central in [default]
org.eclipse.jetty#jetty-util;8.1.17.v20150415 from central in [default]
org.eclipse.jetty#jetty-webapp;8.1.17.v20150415 from central in [default]
org.eclipse.jetty#jetty-xml;8.1.17.v20150415 from central in [default]
org.eclipse.jetty.aggregate#jetty-servlet;8.1.17.v20150415 from central in [default]
org.eclipse.jetty.orbit#javax.activation;1.1.0.v201105071233 from central in [default]
org.eclipse.jetty.orbit#javax.mail.glassfish;1.4.1.v201005082020 from central in [default]
org.eclipse.jetty.orbit#javax.servlet;3.0.0.v201112011016 from central in [default]
org.eclipse.jetty.orbit#javax.transaction;1.1.1.v201105210645 from central in [default]
org.javassist#javassist;3.18.2-GA from central in [default]
org.joda#joda-convert;1.7 from central in [default]
org.kohsuke#libpam4j;1.8 from central in [default]
:: evicted modules:
joda-time#joda-time;2.3 by [joda-time#joda-time;2.8.1] in [default]
org.apache.httpcomponents#httpclient;4.1 by [org.apache.httpcomponents#httpclient;4.3.6] in [default]
org.apache.httpcomponents#httpcore;4.1 by [org.apache.httpcomponents#httpcore;4.3.3] in [default]
commons-logging#commons-logging;1.1.1 by [commons-logging#commons-logging;1.1.3] in [default]
commons-codec#commons-codec;1.4 by [commons-codec#commons-codec;1.6] in [default]
com.google.guava#guava;16.0.1 by [com.google.guava#guava;19.0] in [default]
com.google.guava#guava;18.0 by [com.google.guava#guava;19.0] in [default]
commons-codec#commons-codec;1.3 by [commons-codec#commons-codec;1.4] in [default]
commons-logging#commons-logging;1.0.4 by [commons-logging#commons-logging;1.1.1] in [default]
commons-codec#commons-codec;1.2 by [commons-codec#commons-codec;1.4] in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 68 | 15 | 15 | 10 || 55 | 12 |
---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent
confs: [default]
12 artifacts copied, 43 already retrieved (23416kB/63ms)
Traceback (most recent call last):
File "/home/hadoop/scripts/test/spark.py", line 3, in <module>
hc = H2OContext.getOrCreate(sc)
NameError: name 'H2OContext' is not defined
[hadoop@ip-172-31-32-30测试]$spark提交——包ai.h2o:Sparking-water-core_2.11:2.1.7，ai.h2o:Sparking-water-examples_2.11:2.1.7——配置spark.DynamicLocation.enabled=false spark.py
常春藤默认缓存设置为：/home/hadoop/.ivy2/Cache
存储在：/home/hadoop/.ivy2/jars中的包的jar
：：加载设置：：url=jar:file:/usr/lib/spark/jars/ivy-2.4.0.jar/org/apache/ivy/core/settings/ivysettings.xml
ai.h2o#sparkling-water-core_2.11作为依赖项添加
ai.h2o#起泡水-2.11作为依赖项添加
：：解析依赖项：：org.apache.spark#spark提交父级；1
confs:[默认值]
发现ai.h2o#起泡水芯#2.11；2.1.7在中环
建立了ai.h2o#h2o基因模型；3.10.4.7在中环
找到net.sf.opencsv#opencsv；2.3在中环
发现ai.h2o#深水后端api；中环1.0.2
找到com.google.guava#guava；中环19.0
发现ai.h2o#h2o核；3.10.4.7在中环
找到乔达时间#乔达时间；2.3在中环
找到gov.nist.math#jama；中环1.0.3
找到org.javassist#javassist；3.18.2-GA位于中环
找到org.apache.commons#commons-math3；3.3在中环
发现共享io#共享io；2.4在中环
发现ai.h2o#谷歌分析java；1.1.2-H2O-中环定制
找到org.apache.httpcomponents#httpclient；4.1在中环
找到org.apache.httpcomponents#httpcore；4.1在中环
找到共享日志#共享日志；1.1.1在中环
找到commons编解码器#commons编解码器；1.4在中环
找到org.eclipse.jetty.aggregate#jetty servlet；8.1.17.v20150415位于中环
找到org.eclipse.jetty#jetty plus；8.1.17.v20150415位于中环
找到org.eclipse.jetty.orbit#javax.transaction；1.1.1.v201105210645位于中环
找到org.eclipse.jetty#jetty webapp；8.1.17.v20150415位于中环
找到org.eclipse.jetty#jettyxml；8.1.17.v20150415位于中环
找到org.eclipse.jetty#jetty util；8.1.17.v20150415位于中环
找到org.eclipse.jetty#jetty servlet；8.1.17.v20150415位于中环
找到org.eclipse.jetty#jetty security；8.1.17.v20150415位于中环
找到org.eclipse.jetty#jetty服务器；8.1.17.v20150415位于中环
找到org.eclipse.jetty.orbit#javax.servlet；3.0.0.v201112011016位于中环
找到org.eclipse.jetty#jetty continuation；8.1.17.v20150415位于中环
找到org.eclipse.jetty#jetty http；8.1.17.v20150415位于中环
找到org.eclipse.jetty#jetty io；8.1.17.v20150415位于中环
找到org.eclipse.jetty#jetty jndi；8.1.17.v20150415位于中环
找到org.eclipse.jetty.orbit#javax.mail.glassfish；1.4.1.v201005082020位于中环
找到org.eclipse.jetty.orbit#javax.activation；1.1.0.v201105071233位于中环
找到com.github.rwl#jtransforms；中环2.4.0
发现ai.h2o#h2o jaas pam；3.10.4.7在中环
找到org.kohsuke#libpam4j；中环1.8
找到net.java.dev.jna#jna；中环4.0.0
找到log4j#log4j；1.2.15位于中环
找到com.google.code.gson#gson；2.3.1在中环
找到了commons lang#commons lang；2.6在中环
发现ai.h2o反射；中环0.9.11-h2o-定制
找到com.google.code.findbugs#jsr305；中环3.0.0
发现ai.h2o#h2o算法；3.10.4.7在中环
发现ai.h2o#h2o网络；3.10.4.7在中环
找到ai.h2o#h2o avro解析器；3.10.4.7在中环
找到ai.h2o#h2o拼花地板解析器；3.10.4.7在中环
找到ai.h2o#h2o orc解析器；3.10.4.7在中环
发现ai.h2o#h2o-scala_2.11；3.10.4.7在中环
发现ai.h2o#h2o持续存在hdfs；3.10.4.7在中环
找到net.java.dev.jets3t#jets3t；中环0.6.1
找到commons httpclient#commons httpclient；3.1在中环
发现ai.h2o#h2o-persist-s3；3.10.4.7在中环
找到com.amazonaws#aws-java-sdk-s3；1.10.47中环
找到com.amazonaws#aws java sdk kms；1.10.47中环
找到com.amazonaws#aws java sdk核心；1.10.47中环
找到共享日志#共享日志；1.1.3在中环
找到org.apache.httpcomponents#httpclient；4.3.6在中环
找到org.apache.httpcomponents#httpcore；4.3.3在中环
找到commons编解码器#commons编解码器；中环1.6
找到乔达时间#乔达时间；2.8.1在中环
发现ai.h2o#起泡水-repl_2.11；2.1.7在中环
找到org.joda#joda convert；中环1.7
发现ai.h2o#起泡水-U 2.11；2.1.7在中环
发现ai.h2o#起泡水-ml#U 2.11；2.1.7在中环
正在下载https://repo1.maven.org/maven2/ai/h2o/sparkling-water-core_2.11/2.1.7/sparkling-water-core_2.11-2.1.7.jar ...
[成功]ai.h2o#起泡水芯#2.11；2.1.7!sparkling-water-core_2.11.jar（56ms）
正在下载https://repo1.maven.org/maven2/ai/h2o/sparkling-water-examples_2.11/2.1.7/sparkling-water-examples_2.11-2.1.7.jar ...
[成功]ai.h2o#起泡水-U 2.11；2.1.7!起泡水-U 2.11.jar（15ms）
正在下载https://repo1.maven.org/maven2/ai/h2o/h2o-genmodel/3.10.4.7/h2o-genmodel-3.10.4.7.jar ...
[成功]ai.h2o#h2o模型；3.10.4.7!h2o-genmodel.jar（7ms）
正在下载https://repo1.maven.org/maven2/ai/h2o/h2o-core/3.10.4.7/h2o-core-3.10.4.7.jar ...
[成功]ai.h2o#h2o核心；3.10.4.7!h2o-core.jar（129ms）
正在下载https://repo1.maven.org/maven2/ai/h2o/h2o-algos/3.10.4.7/h2o-algos-3.10.
$SPARK_HOME/bin/pyspark --py-files h2o_pysparkling_2.1-2.1.8.zip