Scala 用户代码中的错误未捕获可丢弃：spark Streaming中的java.lang.IllegalStateException

我有两个流式查询，如贝娄

val streamingQuery = injectableDependencies.writeStreamDfToDeltaFile(validDF, dataPath, checkPointPath, configuration.pollingTimerSeconds, queryName, configuration.outputMode, configuration.partitionBy)

//streamingQuery for Invalid tables
val streamingQueryInvalid = injectableDependencies.writeStreamDfToDeltaFile(inValidDF, dataPathInvalid, checkPointPath, configuration.pollingTimerSeconds, queryNameInvalid, configuration.outputMode, configuration.partitionBy)

//creating valid tables
injectableDependencies.createTableOverDeltaFile(configuration.tableName, configuration.databaseName, dataPath, streamingQuery)

//creating invalid tables
injectableDependencies.createTableOverDeltaFile("Invalid"+configuration.tableName, configuration.databaseName, dataPath, streamingQueryInvalid)

下面我正在尝试写流

  df
  .writeStream
  .format("DELTA")
  .option("path", dataPath)
  .option("checkpointLocation", checkPointPath)
  .partitionBy(partitionBy.getOrElse(List[String]()): _*)
  .outputMode(outputMode)
  .trigger(Trigger.ProcessingTime(pollingTimerSeconds.seconds))
  .queryName(queryName)
  .start()

我犯了一个错误

ERROR Uncaught throwable from user code: java.lang.IllegalStateException: Cannot start query with id 74e6b948-bc55-419c-af42-34ef7ea015ba as another query with same id is already active. Perhaps you are attempting to restart a query from the checkpoint that is already active.

---

有人能告诉我出了什么问题吗？

我也遇到了同样的问题，通过确保每个流式查询在

writeStream

调用中获得其专用的

checkpointLocation

作为选项来解决它。

很好！这对我也有用！