Apache spark 如何更改spark submit console输出中的日志级别？

我是Spark（Pyspark）的新手，当我运行Spark提交时，Spark日志非常冗长。如何更改日志输出？

步骤1配置log4j属性

# Define the root logger with Appender file 
  log4j.rootLogger=WARN, console 
# Define the file appender 
  log4j.appender.FILE=org.apache.log4j.DailyRollingFileAppender 
# Name of the log file 
  log4j.appender.FILE.File=/tmp/logfile.out 
# Set immediate flush to true log4j.appender.FILE.ImmediateFlush=true 
# Set the threshold to DEBUG mode log4j.appender.FILE.Threshold=debug 
# Set File append to true. log4j.appender.FILE.Append=true 
# Set the Default Date pattern log4j.appender.FILE.DatePattern='.' yyyy-MM-dd 
# Default layout for the appender
  log4j.appender.FILE.layout=org.apache.log4j.PatternLayout
  log4j.appender.FILE.layout.conversionPattern=%m%n

我们将使用称为Appender的东西。根据log4j文件，Appender负责将LogEvents发送到其目的地

将以下行附加到log4j配置属性。您将在spark安装目录中找到该文件–spark/conf/log4j.properties

# Define the root logger with Appender file 
  log4j.rootLogger=WARN, console 
# Define the file appender 
  log4j.appender.FILE=org.apache.log4j.DailyRollingFileAppender 
# Name of the log file 
  log4j.appender.FILE.File=/tmp/logfile.out 
# Set immediate flush to true log4j.appender.FILE.ImmediateFlush=true 
# Set the threshold to DEBUG mode log4j.appender.FILE.Threshold=debug 
# Set File append to true. log4j.appender.FILE.Append=true 
# Set the Default Date pattern log4j.appender.FILE.DatePattern='.' yyyy-MM-dd 
# Default layout for the appender
  log4j.appender.FILE.layout=org.apache.log4j.PatternLayout
  log4j.appender.FILE.layout.conversionPattern=%m%n

步骤2：在Spark应用程序中使用它

在pyspark脚本中，需要初始化记录器以使用log4j。简单的是，你已经在pyspark环境中拥有了它

sc = SparkContext(conf=conf) 
log4jLogger = sc._jvm.org.apache.log4j 
log = log4jLogger.LogManager.getLogger(__name__) 
log.warn("Hello World!")