设置Cloudera CDH6.2以色调和x27编码pySpark;s笔记本和跑步
我已经按照office文档安装了cdh6.2,并且运行正常。但我仍然希望编码pySpark并在色调中运行它,而不是在pySpark shell中编码pySpark。我在CHD的色调配置页面中尝试了很多配置。我喜欢的是:设置Cloudera CDH6.2以色调和x27编码pySpark;s笔记本和跑步,pyspark,cloudera,hue,Pyspark,Cloudera,Hue,我已经按照office文档安装了cdh6.2,并且运行正常。但我仍然希望编码pySpark并在色调中运行它,而不是在pySpark shell中编码pySpark。我在CHD的色调配置页面中尝试了很多配置。我喜欢的是: [notebook] [[interpreters]] [[[spark]]] name=Scala interface=livy [[[pyspark]]] name=PySpark int
[notebook]
[[interpreters]]
[[[spark]]]
name=Scala
interface=livy
[[[pyspark]]]
name=PySpark
interface=livy
[[[r]]]
name=R
interface=livy
已安装版本为0.6.0的livy,livy服务器日志:
19/05/09 18:01:25 INFO server.AccessManager: AccessControlManager acls disabled;users with view permission: ;users with modify permission: ;users with super permission: ;other allowed users: *
19/05/09 18:01:25 INFO utils.LineBufferedStream: WARNING: User-defined SPARK_HOME (/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark) overrides detected (/opt/cloudera/parcels/CDH/lib/spark).
19/05/09 18:01:25 INFO utils.LineBufferedStream: WARNING: Running spark-class from user-defined location.
19/05/09 18:01:26 INFO utils.LineBufferedStream: Welcome to
19/05/09 18:01:26 INFO utils.LineBufferedStream: ____ __
19/05/09 18:01:26 INFO utils.LineBufferedStream: / __/__ ___ _____/ /__
19/05/09 18:01:26 INFO utils.LineBufferedStream: _\ \/ _ \/ _ `/ __/ '_/
19/05/09 18:01:26 INFO utils.LineBufferedStream: /___/ .__/\_,_/_/ /_/\_\ version 2.4.0-cdh6.2.0
19/05/09 18:01:26 INFO utils.LineBufferedStream: /_/
19/05/09 18:01:26 INFO utils.LineBufferedStream:
19/05/09 18:01:26 INFO utils.LineBufferedStream: Using Scala version 2.11.12, OpenJDK 64-Bit Server VM, 1.8.0_212
19/05/09 18:01:26 INFO utils.LineBufferedStream: Branch HEAD
19/05/09 18:01:26 INFO utils.LineBufferedStream: Compiled by user jenkins on 2019-03-14T07:01:24Z
19/05/09 18:01:26 INFO utils.LineBufferedStream: Revision HEAD
19/05/09 18:01:26 INFO utils.LineBufferedStream: Url
19/05/09 18:01:26 INFO utils.LineBufferedStream: Type --help for more information.
19/05/09 18:01:26 INFO recovery.StateStore$: Using BlackholeStateStore for recovery.
19/05/09 18:01:26 INFO sessions.BatchSessionManager: Recovered 0 batch sessions. Next session id: 0
19/05/09 18:01:26 INFO sessions.InteractiveSessionManager: Recovered 0 interactive sessions. Next session id: 0
19/05/09 18:01:26 INFO sessions.InteractiveSessionManager: Heartbeat watchdog thread started.
19/05/09 18:01:26 INFO util.log: Logging initialized @1490ms
19/05/09 18:01:26 INFO server.Server: jetty-9.3.24.v20180605, build timestamp: 2018-06-06T01:11:56+08:00, git hash: 84205aa28f11a4f31f2a3b86d1bba2cc8ab69827
19/05/09 18:01:26 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@1522d8a0{/,file:///opt/apache-livy-0.6.0-incubating-bin/src/main/org/apache/livy/server,AVAILABLE}
19/05/09 18:01:26 INFO server.AbstractNCSARequestLog: Opened /opt/apache-livy-0.6.0-incubating-bin/logs/2019_05_09.request.log
19/05/09 18:01:26 INFO server.AbstractConnector: Started ServerConnector@77a98a6a{HTTP/1.1,[http/1.1]}{0.0.0.0:8998}
19/05/09 18:01:26 INFO server.Server: Started @1674ms
19/05/09 18:01:26 INFO server.WebServer: Starting server on http://cdh.n1.com:8998
添加后您可以看到:
[desktop]
app_blacklist=
[notebook]
show_notebooks=true
我试过了,但仍然没有工作,我发现这里的问题是通过cdh安装和配置色调,用于连接其他组件的色调信息是由chd提供的,但在我的cdh6.2版本中没有关于livy的信息。安装livy并使用您提到的色调配置后。它起作用了。