ExchangeCoordinator

When we invoke SparkPlan.execute, if there are ShuffleExchange registered, a map only stage will be submitted by SparkContext.submitMapStage, which will estimate the size of map task output, and construct the shuffleRDD based on the stats of map stage.

The ensureDistributionAndOrdering will create the ExchangeCoordinator with ensureDistributionAndOrdering. The coordinator is created with the expected exchange size of children.length.

All the ShuffleExchagne will register to the coordinator in doPrepare, and then in its doExecutor, it invokes postShuffleRDD, which will trigger the real execution of SparkContext.submitMapStage, and construct ShuffledRowRDD for the ShuffleExchange.

results matching ""

    No results matching ""