Scala java.io.EOFException不是空文件上的SequenceFile

我正试着用spark阅读表格

spark.table("table_name")
sc.sequenceFile(path, classOf[Text], classOf[Text], 1000).
  map(x => x._2.toString.split(delimiter, -1))

如果没有空文件，这两种方法都有效；如果表中包含空文件，则这两种方法都会因

java.io.eofeexception:/path/to/file/1612735495084_12eed62a-b1ee-4cf5-8b71-a87149acd9c8.sf而失败

设置
spark.sql.files.ignoreCorruptFiles=true
没有帮助。似乎0字节文件未被视为已损坏

我不能修改源表，只能修改我自己的代码。在读取此表时是否有方法忽略空文件

使用spark 2.2、scala 2.11

     diagnostics: User class threw exception: org.apache.spark.SparkException: Job aborted.
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:224)
    at org.apache.spark.sql.hive.execution.SaveAsHiveFile$class.saveAsHiveFile(SaveAsHiveFile.scala:86)
    at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.saveAsHiveFile(InsertIntoHiveTable.scala:66)
    at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.processInsert(InsertIntoHiveTable.scala:195)
    at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.run(InsertIntoHiveTable.scala:99)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:122)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
    at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
    at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
    at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
    at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:656)
    at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:656)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77)
    at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:656)
    at org.apache.spark.sql.DataFrameWriter.insertInto(DataFrameWriter.scala:322)
    at org.apache.spark.sql.DataFrameWriter.insertInto(DataFrameWriter.scala:308)
    at ru.gjin.gjin.system.Main$.delayedEndpoint$ru$gjin$gjin$system$Main$1(Main.scala:30)
    at ru.gjin.gjin.system.Main$delayedInit$body.apply(Main.scala:12)
    at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
    at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
    at scala.App$$anonfun$main$1.apply(App.scala:76)
    at scala.App$$anonfun$main$1.apply(App.scala:76)
    at scala.collection.immutable.List.foreach(List.scala:381)
    at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
    at scala.App$class.main(App.scala:76)
    at ru.gjin.gjin.system.Main$.main(Main.scala:12)
    at ru.gjin.gjin.system.Main.main(Main.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:721)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 36305 in stage 9.0 failed 4 times, most recent failure: Lost task 36305.3 in stage 9.0 (TID 2694, hdp3-sp-024.dmp.vimpelcom.ru, executor 6): java.io.EOFException: host/path/to/file/1612735487120_c186faf5-72c0-4747-884b-58ce29433906.sf not a SequenceFile
    at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1964)
    at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1923)
    at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1872)
    at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1886)
    at org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:49)
    at org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:64)
    at org.apache.spark.rdd.HadoopRDD$$anon$1.liftedTree1$1(HadoopRDD.scala:257)
    at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:256)
    at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:214)
    at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:94)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:49)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:49)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:49)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:49)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:49)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
    at org.apache.spark.scheduler.Task.run(Task.scala:109)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1651)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1639)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1638)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1638)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
    at scala.Option.foreach(Option.scala:257)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:831)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1872)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1821)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1810)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:642)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2034)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:194)
    ... 37 more
Caused by: java.io.EOFException: host/path/to/file/1612735487120_c186faf5-72c0-4747-884b-58ce29433906.sf not a SequenceFile
    at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1964)
    at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1923)
    at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1872)
    at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1886)
    at org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:49)
    at org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:64)
    at org.apache.spark.rdd.HadoopRDD$$anon$1.liftedTree1$1(HadoopRDD.scala:257)
    at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:256)
    at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:214)
    at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:94)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:49)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:49)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:49)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:49)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:49)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
    at org.apache.spark.scheduler.Task.run(Task.scala:109)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

diagnostics:User类引发异常：org.apache.spark.SparkException:作业中止。
位于org.apache.spark.sql.execution.datasources.FileFormatWriter$.write（FileFormatWriter.scala:224）
位于org.apache.spark.sql.hive.execution.SaveAsHiveFile$class.SaveAsHiveFile（SaveAsHiveFile.scala:86）
位于org.apache.spark.sql.hive.execution.InsertIntoHiveTable.saveAsHiveFile（InsertIntoHiveTable.scala:66）
位于org.apache.spark.sql.hive.execution.InsertIntoHiveTable.processInsert（InsertIntoHiveTable.scala:195）
位于org.apache.spark.sql.hive.execution.InsertIntoHiveTable.run（InsertIntoHiveTable.scala:99）
位于org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute（commands.scala:104）
位于org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult（commands.scala:102）
位于org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute（commands.scala:122）
位于org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply（SparkPlan.scala:131）
位于org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply（SparkPlan.scala:127）
位于org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply（SparkPlan.scala:155）
位于org.apache.spark.rdd.RDDOperationScope$.withScope（RDDOperationScope.scala:151）
位于org.apache.spark.sql.execution.SparkPlan.executeQuery（SparkPlan.scala:152）
位于org.apache.spark.sql.execution.SparkPlan.execute（SparkPlan.scala:127）
位于org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute（QueryExecution.scala:80）
位于org.apache.spark.sql.execution.QueryExecution.toRdd（QueryExecution.scala:80）
位于org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply（DataFrameWriter.scala:656）
位于org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply（DataFrameWriter.scala:656）
位于org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId（SQLExecution.scala:77）
位于org.apache.spark.sql.DataFrameWriter.runCommand（DataFrameWriter.scala:656）
位于org.apache.spark.sql.DataFrameWriter.insertInto（DataFrameWriter.scala:322）
位于org.apache.spark.sql.DataFrameWriter.insertInto（DataFrameWriter.scala:308）
在ru.gjin.gjin.system.Main$.delayedEndpoint$ru$gjin$gjin$system$Main$1（Main.scala:30）
在ru.gjin.gjin.system.Main$delayedInit$body.apply（Main.scala:12）
在scala.Function0$class.apply$mcV$sp处（Function0.scala:34）
在scala.runtime.AbstractFunction0.apply$mcV$sp（AbstractFunction0.scala:12）中
位于scala.App$$anonfun$main$1.apply（App.scala:76）
位于scala.App$$anonfun$main$1.apply（App.scala:76）
位于scala.collection.immutable.List.foreach（List.scala:381）
位于scala.collection.generic.TraversableForwarder$class.foreach（TraversableForwarder.scala:35）
位于scala.App$class.main（App.scala:76）
在ru.gjin.gjin.system.Main$.Main（Main.scala:12）
at ru.gjin.gjin.system.Main.Main（Main.scala）
在sun.reflect.NativeMethodAccessorImpl.invoke0（本机方法）处
位于sun.reflect.NativeMethodAccessorImpl.invoke（NativeMethodAccessorImpl.java:62）
在sun.reflect.DelegatingMethodAccessorImpl.invoke（DelegatingMethodAccessorImpl.java:43）中
位于java.lang.reflect.Method.invoke（Method.java:498）
位于org.apache.spark.deploy.warn.ApplicationMaster$$anon$4.run（ApplicationMaster.scala:721）
原因：org.apache.spark.SparkException：作业因阶段失败而中止：阶段9.0中的任务36305失败4次，最近的失败：阶段9.0中的任务36305.3丢失（TID 2694，hdp3-sp-024.dmp.vimpelcom.ru，executor 6）：java.io.EOFException:host/path/to/file/1612735487120_c186faf5-72c0-4747-884b-58ce29433906.sf不是序列文件
位于org.apache.hadoop.io.SequenceFile$Reader.init（SequenceFile.java:1964）
位于org.apache.hadoop.io.SequenceFile$Reader.initialize（SequenceFile.java:1923）
位于org.apache.hadoop.io.SequenceFile$Reader。（SequenceFile.java:1872）
位于org.apache.hadoop.io.SequenceFile$Reader。（SequenceFile.java:1886）
位于org.apache.hadoop.mapred.SequenceFileRecordReader。（SequenceFileRecordReader.java:49）
位于org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader（SequenceFileInputFormat.java:64）
位于org.apache.spark.rdd.HadoopRDD$$anon$1.liftedTree1$1（HadoopRDD.scala:257）
位于org.apache.spark.rdd.HadoopRDD$$anon$1。（HadoopRDD.scala:256）
位于org.apache.spark.rdd.HadoopRDD.compute（HadoopRDD.scala:214）
位于org.apache.spark.rdd.HadoopRDD.compute（HadoopRDD.scala:94）
在org.apache.spark.rdd.rdd.computeOrReadCheckpoint（rdd.scala:324）
位于org.apache.spark.rdd.rdd.iterator（rdd.scala:288）
在org.apache.spark.rdd.MapPartitionsRDD.compute上（MapPartitionsRDD.scala:49）
在org.apache.spark.rdd.rdd.computeOrReadCheckpoint（rdd.scala:324）
位于org.apache.spark.rdd.rdd.iterator（rdd.scala:288）
在org.apache.spark.rdd.MapPartitionsRDD.compute上（MapPartitionsRDD.scala:49）
在org.apache.spark.rdd.rdd.computeOrReadCheckpoint（rdd.scala:324）
位于org.apache.spark.rdd.rdd.iterator（rdd.scala:288）
位于org.apache.spark.rdd.UnionRDD.compute（UnionRDD.scala:105）
在org.apache.spark.rdd.rdd.computeOrReadCheckpoint（rdd.scala:324）
位于org.apache.spark.rdd.rdd.iterator（rdd.scala:288）
在org.apache.spark.rdd.MapPartitionsRDD.compute上（MapPartitionsRDD.scala:49）
在org.apache.spark.rdd.rdd.computeOrReadCheckpoint（rdd.scala:324）
位于org.apache.spark.rdd.rdd.iterator（rdd.scala:288）
在org.apache.spark.rdd.M上
SEQ^F!org.apache.hadoop.io.LongWritable!org.apache.hadoop.io.LongWritable^A^@*org.apache.hadoop.io.compress.DefaultCodec^@^@^@^@ï<9c>p<84>º74K=æÅ3!<92>^A^F