Docker 是否还有其他配置需要与Livy服务器（Livy.conf）一起完成？

我已经为hadoop纱线设置了docker，我正在尝试设置livy apache服务器，以便为作业提交进行API调用

下面的日志表示livy服务器在特定时间启动并自动停止

19/08/17 07:09:35 INFO utils.LineBufferedStream: Welcome to
19/08/17 07:09:35 INFO utils.LineBufferedStream:       ____              __
19/08/17 07:09:35 INFO utils.LineBufferedStream:      / __/__  ___ _____/ /__
19/08/17 07:09:35 INFO utils.LineBufferedStream:     _\ \/ _ \/ _ `/ __/  '_/
19/08/17 07:09:35 INFO utils.LineBufferedStream:    /___/ .__/\_,_/_/ /_/\_\   version 2.2.1
19/08/17 07:09:35 INFO utils.LineBufferedStream:       /_/
19/08/17 07:09:35 INFO utils.LineBufferedStream:
19/08/17 07:09:35 INFO utils.LineBufferedStream: Using Scala version 2.11.8, OpenJDK 64-Bit Server VM, 1.8.0_222
19/08/17 07:09:35 INFO utils.LineBufferedStream: Branch
19/08/17 07:09:35 INFO utils.LineBufferedStream: Compiled by user felixcheung on 2017-11-24T23:19:45Z
19/08/17 07:09:35 INFO utils.LineBufferedStream: Revision
19/08/17 07:09:35 INFO utils.LineBufferedStream: Url
19/08/17 07:09:35 INFO utils.LineBufferedStream: Type --help for more information.
19/08/17 07:09:35 INFO recovery.StateStore$: Using BlackholeStateStore for recovery.
19/08/17 07:09:35 INFO sessions.BatchSessionManager: Recovered 0 batch sessions. Next session id: 0
19/08/17 07:09:35 INFO sessions.InteractiveSessionManager: Recovered 0 interactive sessions. Next session id: 0
19/08/17 07:09:35 INFO sessions.InteractiveSessionManager: Heartbeat watchdog thread started.
19/08/17 07:09:35 INFO util.log: Logging initialized @1944ms
19/08/17 07:09:36 INFO server.Server: jetty-9.3.24.v20180605, build timestamp: 2018-06-05T17:11:56Z, git hash: xxx0x0x0xx00xxxx0x0x0x0x0x0x0x0xxxx
19/08/17 07:09:36 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@3543df7d{/,file:///livy/apache-livy-0.6.0-incubating-bin/bin/src/main/org/apache/livy/server,AVAILABLE}
19/08/17 07:09:36 INFO server.AbstractNCSARequestLog: Opened /livy/apache-livy-0.6.0-incubating-bin/logs/2019_08_17.request.log
19/08/17 07:09:36 INFO server.AbstractConnector: Started ServerConnector@686449f9{HTTP/1.1,[http/1.1]}{x.x.x.x:8080}
19/08/17 07:09:36 INFO server.Server: Started @2304ms
19/08/17 07:09:36 INFO server.WebServer: Starting server on http://x.x.x.x:8080
19/08/17 07:10:01 INFO server.LivyServer: Shutting down Livy server.
19/08/17 07:10:01 INFO handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@3543df7d{/,file:///livy/apache-livy-0.6.0-incubating-bin/bin/src/main/org/apache/livy/server,UNAVAILABLE}
19/08/17 07:10:01 INFO server.AbstractConnector: Stopped ServerConnector@686449f9{HTTP/1.1,[http/1.1]}{x.x.x.x:8080}

我已经提供了livy.conf，其中提到了livy要运行的服务器ip和服务器端口。当我试图提交时，我也完成了他们的设置，我附上了下面的文件

docker compose

version: "2"

services:
 livy:
  image: namenode/hadoopspark:2.2.1
  command: /livy/apache-livy-0.6.0-incubating-bin/bin/livy-server start
  network_mode: "host"
  ports:
   - 8080:8080


#####################BASE DOCKERFILE#################

FROM ubuntu:14.04

ENV DAEMON_RUN=true
ENV SPARK_VERSION=2.2.1
ENV HADOOP_VERSION=2.7
ENV SPARK_HOME=/spark
ENV HADOOP_HOME=/hadoop

RUN apt-get update \
 && apt-get install -y software-properties-common openssh-server net-tools curl nano vim wget ca-certificates jq gnupg unzip

RUN add-apt-repository ppa:openjdk-r/ppa
RUN apt-get update
RUN apt-get install -y openjdk-8-jdk \
 supervisor

RUN ssh-keygen -q -N "" -t rsa -f /root/.ssh/id_rsa
RUN cp /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys

RUN wget https://www-eu.apache.org/dist/incubator/livy/0.6.0-incubating/apache-livy-0.6.0-incubating-bin.zip \
 && unzip apache-livy-0.6.0-incubating-bin.zip \
 && mkdir -p livy \
 && mv apache-livy-0.6.0-incubating-bin /livy

RUN wget https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz \
 &&  tar -xzf spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz \
 &&  mv spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION} /spark

RUN wget https://archive.apache.org/dist/hadoop/core/hadoop-2.7.3/hadoop-2.7.3.tar.gz \
 && tar -xzvf hadoop-2.7.3.tar.gz \
 && mv hadoop-2.7.3 /hadoop

ENV JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
ENV HADOOP_CONF_DIR=/hadoop/etc/hadoop

RUN echo "export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre/ \
export HADOOP_HOME=/hadoop \
export HADOOP_CONF_DIR=/hadoop/etc/hadoop \
export HADOOP_SSH_OPTS='"-p 22"' \
" >> /hadoop/etc/hadoop/hadoop-env.sh

ENV PATH=$SPARK_HOME/bin:$PATH
ENV PATH=$PATH:/hadoop/bin:/hadoop/sbin


################NAMENODE DOCKERFILE####################

FROM base/hadoopspark:2.2.1

COPY conf/* /tmp/

RUN cp /tmp/hdfs-site.xml $HADOOP_HOME/etc/hadoop/hdfs-site.xml && \
    cp /tmp/core-site.xml $HADOOP_HOME/etc/hadoop/core-site.xml && \
    cp /tmp/mapred-site.xml $HADOOP_HOME/etc/hadoop/mapred-site.xml && \
    cp /tmp/yarn-site.xml $HADOOP_HOME/etc/hadoop/yarn-site.xml && \
    cp /tmp/hdfs-site.xml $SPARK_HOME/conf/ && \
    cp /tmp/core-site.xml $SPARK_HOME/conf/ && \
    cp /tmp/mapred-site.xml $SPARK_HOME/conf/ && \
    cp /tmp/yarn-site.xml $SPARK_HOME/conf/ && \
    cp /tmp/spark-defaults.conf $SPARK_HOME/conf/ && \
    cp /tmp/livy.conf /livy/apache-livy-0.6.0-incubating-bin/conf

COPY Docker_WordCount_Spark-1.0.jar /opt/Docker_WordCount_Spark-1.0.jar
COPY sample.txt /opt/sample.txt

#RUN hdfs dfs -put /opt/Docker_WordCount_Spark-1.0.jar Docker_WordCount_Spark-1.0.jar
#RUN hdfs dfs -put /opt/sample.txt sample.txt

ENV LD_LIBRARY_PATH=/hadoop/lib/native:$LD_LIBRARY_PATH

RUN sudo service ssh restart
RUN sudo /hadoop/bin/hadoop namenode -format

EXPOSE 8998 8080

是否需要任何其他帮助来启动livy服务器。谢谢大家!

Docker需要命令才能在前台继续运行。否则，它会认为应用程序已停止，并关闭容器。由于livy服务器启动脚本在后台进程中运行，并且以后不会触发其他前台进程，所以当脚本结束时，容器会退出。您可以通过多种方式解决此问题，简单的解决方案是在Dockerfile中添加以下命令以启动liver服务器（从docker compose.yml中删除命令）

livy服务器docker映像为：

FROM base/hadoopspark:2.2.1

COPY conf/* /tmp/

ENV SPARK_HOME=/spark
ENV HADOOP_HOME=/hadoop

RUN cp /tmp/hdfs-site.xml $HADOOP_HOME/etc/hadoop/hdfs-site.xml && \
    cp /tmp/core-site.xml $HADOOP_HOME/etc/hadoop/core-site.xml && \
    cp /tmp/mapred-site.xml $HADOOP_HOME/etc/hadoop/mapred-site.xml && \
    cp /tmp/yarn-site.xml $HADOOP_HOME/etc/hadoop/yarn-site.xml && \
    cp /tmp/hdfs-site.xml $SPARK_HOME/conf/ && \
    cp /tmp/core-site.xml $SPARK_HOME/conf/ && \
    cp /tmp/mapred-site.xml $SPARK_HOME/conf/ && \
    cp /tmp/yarn-site.xml $SPARK_HOME/conf/ && \
    cp /tmp/spark-defaults.conf $SPARK_HOME/conf/ && \
    cp /tmp/livy.conf /livy/apache-livy-0.6.0-incubating-bin/conf

COPY Docker_WordCount_Spark-1.0.jar /opt/Docker_WordCount_Spark-1.0.jar
COPY sample.txt /opt/sample.txt

ENV LD_LIBRARY_PATH=/hadoop/lib/native:$LD_LIBRARY_PATH

RUN sudo service ssh restart
RUN sudo /hadoop/bin/hadoop namenode -format

ENV PATH=$SPARK_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

EXPOSE 8998 8080

CMD /livy/apache-livy-0.6.0-incubating-bin/bin/livy-server start && /bin/bash

Docker需要命令才能在前台继续运行。否则，它会认为应用程序已停止，并关闭容器。由于livy服务器启动脚本在后台进程中运行，并且以后不会触发其他前台进程，所以当脚本结束时，容器会退出。您可以通过多种方式解决此问题，简单的解决方案是在Dockerfile中添加以下命令以启动liver服务器（从docker compose.yml中删除命令）

livy服务器docker映像为：

FROM base/hadoopspark:2.2.1

COPY conf/* /tmp/

ENV SPARK_HOME=/spark
ENV HADOOP_HOME=/hadoop

RUN cp /tmp/hdfs-site.xml $HADOOP_HOME/etc/hadoop/hdfs-site.xml && \
    cp /tmp/core-site.xml $HADOOP_HOME/etc/hadoop/core-site.xml && \
    cp /tmp/mapred-site.xml $HADOOP_HOME/etc/hadoop/mapred-site.xml && \
    cp /tmp/yarn-site.xml $HADOOP_HOME/etc/hadoop/yarn-site.xml && \
    cp /tmp/hdfs-site.xml $SPARK_HOME/conf/ && \
    cp /tmp/core-site.xml $SPARK_HOME/conf/ && \
    cp /tmp/mapred-site.xml $SPARK_HOME/conf/ && \
    cp /tmp/yarn-site.xml $SPARK_HOME/conf/ && \
    cp /tmp/spark-defaults.conf $SPARK_HOME/conf/ && \
    cp /tmp/livy.conf /livy/apache-livy-0.6.0-incubating-bin/conf

COPY Docker_WordCount_Spark-1.0.jar /opt/Docker_WordCount_Spark-1.0.jar
COPY sample.txt /opt/sample.txt

ENV LD_LIBRARY_PATH=/hadoop/lib/native:$LD_LIBRARY_PATH

RUN sudo service ssh restart
RUN sudo /hadoop/bin/hadoop namenode -format

ENV PATH=$SPARK_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

EXPOSE 8998 8080

CMD /livy/apache-livy-0.6.0-incubating-bin/bin/livy-server start && /bin/bash

---

请尝试向docker compose添加tty参数，并在分离模式下运行容器

version: "2"

services:
 livy:
  image: namenode/hadoopspark:2.2.1
  command: /livy/apache-livy-0.6.0-incubating-bin/bin/livy-server start
  network_mode: "host"
  ports:
   - 8080:8080
  tty:  true

使用启动docker容器

---

docker compose up-d

请尝试向docker compose添加tty参数，并在分离模式下运行容器

version: "2"

services:
 livy:
  image: namenode/hadoopspark:2.2.1
  command: /livy/apache-livy-0.6.0-incubating-bin/bin/livy-server start
  network_mode: "host"
  ports:
   - 8080:8080
  tty:  true

使用启动docker容器

---

docker compose up-d

我试过了。它仍然以-Log attached的结果存在：

Creating namenode_livy_1 Attaching to namenode_livy_1 | starting/usr/lib/jvm/java-8-openjdk-amd64/bin/java-cp/livy/apache-livy-0.6.0-incubating-bin/jars/*：/livy/apache-livy-livy-0.6.0.0-incubating-bin/conf:/hadoop/etc/hadoop/org.apache.livy.server.livyser，登录到/livy/apache-livy-0.6.0-incubating-bin/logs/livy--server.out namenode_livy_1已退出，代码为0

请使用tty选项，我已使用代码格式在另一个答案中添加了docker compose，我尝试了它。它仍然以-Log attached的结果存在：

Creating namenode_livy_1 Attaching to namenode_livy_1 | starting/usr/lib/jvm/java-8-openjdk-amd64/bin/java-cp/livy/apache-livy-0.6.0-incubating-bin/jars/*：/livy/apache-livy-livy-0.6.0.0-incubating-bin/conf:/hadoop/etc/hadoop/org.apache.livy.server.livyser，登录到/livy/apache-livy-0.6.0-incubating-bin/logs/livy--server.out namenode_livy_1退出，代码为0

请使用tty选项，我在另一个答案中添加了docker compose，代码格式仍然相同。问题似乎出在Livy服务器本身，它不确定？。我已经尝试了分离模式和未连接模式，但容器仍然会退出。Livy服务器在启动并运行一秒钟后自动关闭，这真是可疑！我建议，跟踪livy服务器日志，以便您能够看到容器也在前台调试问题的原因。CMD[“/livy/apache-livy-0.6.0-incubating-bin/bin/livy-server start；tail-f/livy/apache-livy-0.6.0-incubating-bin/logs/livy-server.out；/bin/bash”]是的，当我输入退出的容器和尾部日志时-因为我无法在注释中附加日志，所以您可以查看已经存在问题的日志。如果你有时间，同样的事情现在也会发生。你能在你当地试试吗？我仍然面临同样的问题。问题似乎出在Livy服务器本身，它不确定？。我已经尝试了分离模式和未连接模式，但容器仍然会退出。Livy服务器在启动并运行一秒钟后自动关闭，这真是可疑！我建议，跟踪livy服务器日志，以便您能够看到容器也在前台调试问题的原因。CMD[“/livy/apache-livy-0.6.0-incubating-bin/bin/livy-server start；tail-f/livy/apache-livy-0.6.0-incubating-bin/logs/livy-server.out；/bin/bash”]是的，当我输入退出的容器和尾部日志时-因为我无法在注释中附加日志，所以您可以查看已经存在问题的日志。如果你有时间，同样的事情现在也会发生。你能在你当地的餐馆试试吗？