Apache spark Spark:ERROR LiveListenerBus:Listener EventLoggingListener引发异常
我执行了一个spark程序，在该程序中，我使用customer框架对事实对象应用Drools规则。我正在使用in-Flatmap函数调用此自定义框架，规则已成功执行，但在收集结果后，我看到以下异常。 有人面临过同样的问题吗？如何修复此错误？请在这个问题上帮助我Apache spark Spark:ERROR LiveListenerBus:Listener EventLoggingListener引发异常,apache-spark,Apache Spark,我执行了一个spark程序，在该程序中，我使用customer框架对事实对象应用Drools规则。我正在使用in-Flatmap函数调用此自定义框架，规则已成功执行，但在收集结果后，我看到以下异常。 有人面临过同样的问题吗？如何修复此错误？请在这个问题上帮助我 15/09/18 10:45:34 INFO JDBCRDD: closed connection 15/09/18 10:45:34 INFO Executor: Finished task 0.0 in stage 0.0 (TID
15/09/18 10:45:34 INFO JDBCRDD: closed connection
15/09/18 10:45:34 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 599 bytes result sent to driver
15/09/18 10:45:34 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 3405 ms on localhost (1/1)
15/09/18 10:45:34 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
15/09/18 10:45:34 INFO DAGScheduler: Stage 0 (collect at SMAAlertImpl.java:84) finished in 3.425 s
15/09/18 10:45:34 INFO DAGScheduler: Job 0 finished: collect at SMAAlertImpl.java:84, took 3.719422 s
15/09/18 10:45:35 ERROR LiveListenerBus: Listener EventLoggingListener threw an exception
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144)
at org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:144)
at org.apache.spark.scheduler.EventLoggingListener.onStageCompleted(EventLoggingListener.scala:165)
at org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:32)
at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:53)
at org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:36)
at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:76)
at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
at org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60)
Caused by: java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:792)
at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1998)
at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1959)
at org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:130)
... 19 more
15/09/18 10:45:35 ERROR LiveListenerBus: Listener EventLoggingListener threw an exception
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144)
at org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:144)
at org.apache.spark.scheduler.EventLoggingListener.onJobEnd(EventLoggingListener.scala:169)
at org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:36)
at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:53)
at org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:36)
at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:76)
at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
at org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60)
Caused by: java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:792)
at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1998)
at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1959)
at org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:130)
... 19 more
能够通过添加sc.close（）修复错误。（sc-SparkContext）将spark.eventLog.enabled设置为false“解决”了我的问题，因为在spark Shell中的操作失败（例如“计数”）后调用sc.close（）没有帮助 我还将配置选项spark.scheduler.listenerbus.eventqueue.size从默认值10000更改为500000，但这并不能解决我的问题 资料来源：
检查您是否有“FileSystem.close（）”操作，因为spark将使用相同的文件系统来写入日志，所以不要关闭它。您在哪里进行了更改？这将在spark脚本的末尾，我假设OP是用Python编写的。这并不能为我解决问题，但我不确定这个错误对我是否重要。很难测试，因为错误似乎是随机出现的，在我的工作流中很少出现。您确定它是
sc.close（）sc.stop（）close（）