Spring boot spring引导的java spark日志记录级别问题_Spring Boot_Apache Spark

Spring boot spring引导的java spark日志记录级别问题

spring-boot apache-spark

Spring boot spring引导的java spark日志记录级别问题,spring-boot,apache-spark,Spring Boot,Apache Spark,我正在使用spark和java spring引导，希望将日志级别错误设置为控制台和文件，但它不起作用，尝试了以下解决方案 1） .set（“spark.executor.extraJavaOptions”，“-Dlog4jspark.root.logger=ERROR，console”） 2） .set（“spark.driver.extraJavaOptions”，“-Dorg.slf4j.simpleLogger.defaultLogLevel=ERROR，console”） spark.s

我正在使用spark和java spring引导，希望将日志级别错误设置为控制台和文件，但它不起作用，尝试了以下解决方案

1）

.set（“spark.executor.extraJavaOptions”，“-Dlog4jspark.root.logger=ERROR，console”）

2）

.set（“spark.driver.extraJavaOptions”，“-Dorg.slf4j.simpleLogger.defaultLogLevel=ERROR，console”）

spark.sparkContext.setLogLevel（'ERROR'）

但不起作用：(

火花配置：

    <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-library</artifactId>
        <version>${scala.version}</version>
        <scope>compile</scope>
    </dependency>

    <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-compiler</artifactId>
        <version>${scala.version}</version>
        <scope>compile</scope>
    </dependency>

    <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-reflect</artifactId>
        <version>${scala.version}</version>
        <scope>compile</scope>
    </dependency>

    
        <!-- For spring MVC -->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-mail</artifactId>
            <exclusions>
           <exclusion>
                    <groupId>ch.qos.logback</groupId>
                    <artifactId>logback-classic</artifactId>
                </exclusion>
        </exclusions> 
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>ch.qos.logback</groupId>
                    <artifactId>logback-classic</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>

        <!-- For Spring Integration -->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-integration</artifactId>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-mllib_2.10</artifactId>
            <version>2.0.0</version>
            <exclusions>
                <exclusion>
                    <artifactId>slf4j-api</artifactId>
                    <groupId>org.slf4j</groupId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.10</artifactId>
            <version>2.0.0</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>log4j</groupId>
                    <artifactId>log4j</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.10</artifactId>
            <version>2.0.0</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.10</artifactId>
            <version>2.0.0</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.slf4j/log4j-over-slf4j -->
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-simple</artifactId>
        <version>1.6.4</version>
    </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-devtools</artifactId>
            <scope>compile</scope>
        </dependency>
        
        <dependency>
        <groupId>commons-collections</groupId>
        <artifactId>commons-collections</artifactId>
        <version>3.1</version>
        <exclusions>
            <exclusion>
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
            </exclusion>
            <exclusion>
                <groupId>commons-beanutils</groupId>
                <artifactId>commons-beanutils-core</artifactId>
            </exclusion>
        </exclusions>
        <scope>compile</scope>
    </dependency>

    

    <dependency>
        <groupId>mysql</groupId>
        <artifactId>mysql-connector-java</artifactId>
        <version>5.1.39</version>
    </dependency>

    <dependency>
        <groupId>org.freemarker</groupId>
        <artifactId>freemarker</artifactId>
        <version>2.3.24-incubating</version>
    </dependency>

    <dependency>
        <groupId>com.jayway.jsonpath</groupId>
        <artifactId>json-path</artifactId>
        <version>2.1.0</version>
        <exclusions>
            <exclusion>
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-log4j12</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-api</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>log4j-over-slf4j</artifactId>
            </exclusion>
            <exclusion>
                <groupId>commons-beanutils</groupId>
                <artifactId>commons-beanutils-core</artifactId>
            </exclusion>
            <exclusion>
                <groupId>commons-collections</groupId>
                <artifactId>commons-collections</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-yarn-common</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

    <dependency>
        <groupId>com.h2database</groupId>
        <artifactId>h2</artifactId>
        <version>1.4.189</version>
        <scope>compile</scope>            
        <exclusions>
            <exclusion>
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-log4j12</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-api</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>log4j-over-slf4j</artifactId>
            </exclusion>
            <exclusion>
                <groupId>commons-beanutils</groupId>
                <artifactId>commons-beanutils-core</artifactId>
            </exclusion>
            <exclusion>
                <groupId>commons-collections</groupId>
                <artifactId>commons-collections</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-yarn-common</artifactId>
            </exclusion>
            <exclusion>
                <groupId>com.fasterxml.jackson.core</groupId>
                <artifactId>*</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>mysql</groupId>
        <artifactId>mysql-connector-java</artifactId>
        <version>5.1.17</version>
    </dependency>       
    <dependency> 
        <groupId>com.opencsv</groupId> 
        <artifactId>opencsv</artifactId> 
        <version>4.1</version> 
    </dependency> 

    <!--Test-->
    <dependency>
        <groupId>org.scalatest</groupId>
        <artifactId>scalatest_${scala.libversion}</artifactId>
        <version>3.0.1</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.12</version>
        <scope>test</scope>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.commons/commons-lang3 -->
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-lang3</artifactId>
        <version>3.3.2</version>
    </dependency>
            <!-- https://mvnrepository.com/artifact/com.healthmarketscience.sqlbuilder/sqlbuilder -->
    <dependency>
        <groupId>com.healthmarketscience.sqlbuilder</groupId>
        <artifactId>sqlbuilder</artifactId>
        <version>2.1.7</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/com.googlecode.json-simple/json-simple -->
    <dependency>
        <groupId>com.googlecode.json-simple</groupId>
        <artifactId>json-simple</artifactId>
        <version>1.1</version>
    </dependency>
    <dependency>
      <groupId>ca.krasnay</groupId>
      <artifactId>sqlbuilder</artifactId>
      <version>1.2</version>
    </dependency>
    
    <dependency>
      <groupId>com.crealytics</groupId>
      <artifactId>spark-excel_2.10</artifactId>
      <version>0.8.3</version>
    </dependency>
    <dependency>
      <groupId>com.eclipsesource.minimal-json</groupId>
      <artifactId>minimal-json</artifactId>
      <version>0.9.5</version>
    </dependency>
            <!-- https://mvnrepository.com/artifact/joda-time/joda-time -->
    <dependency>
        <groupId>joda-time</groupId>
        <artifactId>joda-time</artifactId>
        <version>2.3</version>
    </dependency>
    </dependencies>

pom.xml依赖性：

    <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-library</artifactId>
        <version>${scala.version}</version>
        <scope>compile</scope>
    </dependency>

    <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-compiler</artifactId>
        <version>${scala.version}</version>
        <scope>compile</scope>
    </dependency>

    <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-reflect</artifactId>
        <version>${scala.version}</version>
        <scope>compile</scope>
    </dependency>

    
        <!-- For spring MVC -->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-mail</artifactId>
            <exclusions>
           <exclusion>
                    <groupId>ch.qos.logback</groupId>
                    <artifactId>logback-classic</artifactId>
                </exclusion>
        </exclusions> 
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>ch.qos.logback</groupId>
                    <artifactId>logback-classic</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>

        <!-- For Spring Integration -->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-integration</artifactId>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-mllib_2.10</artifactId>
            <version>2.0.0</version>
            <exclusions>
                <exclusion>
                    <artifactId>slf4j-api</artifactId>
                    <groupId>org.slf4j</groupId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.10</artifactId>
            <version>2.0.0</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>log4j</groupId>
                    <artifactId>log4j</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.10</artifactId>
            <version>2.0.0</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.10</artifactId>
            <version>2.0.0</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.slf4j/log4j-over-slf4j -->
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-simple</artifactId>
        <version>1.6.4</version>
    </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-devtools</artifactId>
            <scope>compile</scope>
        </dependency>
        
        <dependency>
        <groupId>commons-collections</groupId>
        <artifactId>commons-collections</artifactId>
        <version>3.1</version>
        <exclusions>
            <exclusion>
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
            </exclusion>
            <exclusion>
                <groupId>commons-beanutils</groupId>
                <artifactId>commons-beanutils-core</artifactId>
            </exclusion>
        </exclusions>
        <scope>compile</scope>
    </dependency>

    

    <dependency>
        <groupId>mysql</groupId>
        <artifactId>mysql-connector-java</artifactId>
        <version>5.1.39</version>
    </dependency>

    <dependency>
        <groupId>org.freemarker</groupId>
        <artifactId>freemarker</artifactId>
        <version>2.3.24-incubating</version>
    </dependency>

    <dependency>
        <groupId>com.jayway.jsonpath</groupId>
        <artifactId>json-path</artifactId>
        <version>2.1.0</version>
        <exclusions>
            <exclusion>
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-log4j12</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-api</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>log4j-over-slf4j</artifactId>
            </exclusion>
            <exclusion>
                <groupId>commons-beanutils</groupId>
                <artifactId>commons-beanutils-core</artifactId>
            </exclusion>
            <exclusion>
                <groupId>commons-collections</groupId>
                <artifactId>commons-collections</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-yarn-common</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

    <dependency>
        <groupId>com.h2database</groupId>
        <artifactId>h2</artifactId>
        <version>1.4.189</version>
        <scope>compile</scope>            
        <exclusions>
            <exclusion>
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-log4j12</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-api</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>log4j-over-slf4j</artifactId>
            </exclusion>
            <exclusion>
                <groupId>commons-beanutils</groupId>
                <artifactId>commons-beanutils-core</artifactId>
            </exclusion>
            <exclusion>
                <groupId>commons-collections</groupId>
                <artifactId>commons-collections</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-yarn-common</artifactId>
            </exclusion>
            <exclusion>
                <groupId>com.fasterxml.jackson.core</groupId>
                <artifactId>*</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>mysql</groupId>
        <artifactId>mysql-connector-java</artifactId>
        <version>5.1.17</version>
    </dependency>       
    <dependency> 
        <groupId>com.opencsv</groupId> 
        <artifactId>opencsv</artifactId> 
        <version>4.1</version> 
    </dependency> 

    <!--Test-->
    <dependency>
        <groupId>org.scalatest</groupId>
        <artifactId>scalatest_${scala.libversion}</artifactId>
        <version>3.0.1</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.12</version>
        <scope>test</scope>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.commons/commons-lang3 -->
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-lang3</artifactId>
        <version>3.3.2</version>
    </dependency>
            <!-- https://mvnrepository.com/artifact/com.healthmarketscience.sqlbuilder/sqlbuilder -->
    <dependency>
        <groupId>com.healthmarketscience.sqlbuilder</groupId>
        <artifactId>sqlbuilder</artifactId>
        <version>2.1.7</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/com.googlecode.json-simple/json-simple -->
    <dependency>
        <groupId>com.googlecode.json-simple</groupId>
        <artifactId>json-simple</artifactId>
        <version>1.1</version>
    </dependency>
    <dependency>
      <groupId>ca.krasnay</groupId>
      <artifactId>sqlbuilder</artifactId>
      <version>1.2</version>
    </dependency>
    
    <dependency>
      <groupId>com.crealytics</groupId>
      <artifactId>spark-excel_2.10</artifactId>
      <version>0.8.3</version>
    </dependency>
    <dependency>
      <groupId>com.eclipsesource.minimal-json</groupId>
      <artifactId>minimal-json</artifactId>
      <version>0.9.5</version>
    </dependency>
            <!-- https://mvnrepository.com/artifact/joda-time/joda-time -->
    <dependency>
        <groupId>joda-time</groupId>
        <artifactId>joda-time</artifactId>
        <version>2.3</version>
    </dependency>
    </dependencies>


org.scala-lang
scala图书馆
${scala.version}
编译
org.scala-lang
scala编译器
${scala.version}
编译
org.scala-lang
斯卡拉反射
${scala.version}
编译
org.springframework.boot
春季启动邮件
回写
回归经典
org.springframework.boot
SpringBootStarterWeb
回写
回归经典
org.springframework.boot
弹簧起动试验
测试
org.springframework.boot
spring boot启动器集成
org.apache.spark
spark-mllib_2.10
2.0.0
slf4j api
org.slf4j
org.apache.spark
spark-core_2.10
2.0.0
org.slf4j
slf4j-log4j12
log4j
log4j
org.apache.spark
spark-2.10
2.0.0
org.apache.spark
spark-sql_2.10
2.0.0
org.slf4j
slf4j简单
1.6.4
org.springframework.boot
弹簧靴开发工具
编译
公地收藏
公地收藏
3.1
log4j
log4j
公地小海狸
公地豆核
编译
mysql
mysql连接器java
5.1.39
org.freemarker
自由标记
2.3.24-孵化
com.jayway.jsonpath
json路径
2.1.0
log4j
log4j
org.slf4j
slf4j-log4j12
org.slf4j
slf4j api
org.slf4j
log4j-over-slf4j
公地小海狸
公地豆核
公地收藏
公地收藏
org.apache.hadoop
hadoop普通纱线
com.h2数据库
氢
1.4.189
编撰
log4j
log4j
org.slf4j
slf4j-log4j12
org.slf4j
slf4j api
org.slf4j
log4j-over-slf4j
公地小海狸
公地豆核
公地收藏
公地收藏
org.apache.hadoop
hadoop普通纱线
com.fasterxml.jackson.core
*
mysql
mysql连接器java
5.1.17
com.opencsv
opencsv
4.1
org.scalatest
scalatest_${scala.libversion}
3.0.1
测试
朱尼特
朱尼特
4.12
测试
org.apache.commons
commons-lang3
3.3.2
com.healthmarketscience.sqlbuilder
sqlbuilder
2.1.7
com.googlecode.json-simple
简单json
1.1
克拉斯奈附近
sqlbuilder
1.2
com.crealytics
spark-excel_2.10
0.8.3
com.eclipsesource.minimal-json
最小json
0.9.5
乔达时间
乔达时间
2.3

对于我的当前配置：

弹簧靴：2.2.1.1释放
星火：2.12

我和你一样排除了Log4j

<dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.12</artifactId>
        <version>${spark.core.version}</version>
        <exclusions>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-api</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-log4j12</artifactId>
            </exclusion>
            <exclusion>
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
            </exclusion>
        </exclusions>
    </dependency>


org.apache.spark
spark-core_2.12
${spark.core.version}
org.slf4j
slf4j api
org.slf4j
slf4j-log4j12
log4j
log4j

然后，您应该在资源文件夹中配置logback.xml：

<configuration>

<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
    <encoder>
        <pattern>
            %d{dd-MM-yyyy HH:mm:ss.SSS} %magenta([%thread]) %highlight(%-5level) %logger{36}.%M - %msg%n
        </pattern>
    </encoder>
</appender>

<logger name="org.apache.spark" level="WARN"/>

<root level="info">
    <appender-ref ref="STDOUT"/>
</root>


%d{dd MM yyyy HH:MM:ss.SSS}%洋红色（[%thread]）%高亮显示（%-5level）%logger{36}。%M-%msg%n

将此记录器级别

编辑为c