DAGScheduler

Its handleJobSubmitted does the real scheduling. First, it calculates the end of the DAG stages as finalStage, and submitted by submitStage.

From the finalStage, it will traceback and construct mapState if there is ShuffleDependency in getMissingParentStages. If the mapStage is already available, then there is no need to track the dependency back. Othewise, such process will be done recursively until there is no missing parent stage, e.g., read from file directly. All the traversed but pending stages is saved in waitingStages, which will be checked and resubmitted by **submitWaitingChildStages

submitMissingTasks will submit the stage ready to run to TaskScheduler through submitTasks. If the size of tasks is 0, it will invoke submitWaitingChildStages to submit all child stage that depends on the current parent stage. submitWaitingChildStages look for all pending stages and then call submitStage. Note that the children stage may depends on multiple parent stages, which will checked again.

When a task is complete (handled by handleTaskCompletion), and shuffleStage.pendingPartitions is empty, it will invoke submitWaitingChildStages to submit pending children stages.

If the task fails with FetchFailed, which contains the shuffleID, mapID, and reduceID. the corresponding map status will be removed (removeOutputLoc) from the parent stage (outputLocs) and MapOutputTrackerMaster (unregisterMapOutput). So that the parent stage will know which task in the stage needs to be recomputed (because we know exactly which partition is missing from the map) (submitMissingTasks invoke stage.findMissingPartitions() to get such information)