Apache spark 在dataframe中修改后，将创建多少阶段和任务

当一个数据框被拆分并再次与不同的列连接时，在DAG中创建了多少个阶段，以及如何在阶段中创建任务

4. How DAG works in Spark?

    The interpreter is the first layer, using a Scala interpreter, Spark interprets the code with some modifications.
    Spark creates an operator graph when you enter your code in Spark console.
    When we call an Action on Spark RDD at a high level, Spark submits the operator graph to the DAG Scheduler.
    Divide the operators into stages of the task in the DAG Scheduler. A stage contains task based on the partition of the input data. The DAG scheduler pipelines operators together. For example, map operators schedule in a single stage.
    The stages pass on to the Task Scheduler. It launches task through cluster manager. The dependencies of stages are unknown to the task scheduler.
    The Workers execute the task on the slave.

您可以从以下链接获得更多信息：