Shuffle

ShuffleManager is responsible to provide both reader/writer to read/write shuffle data on map/reduce side The BlockManager.shuffleClient will decide which shuffle service will be used, aux service in Yarn, i.e., ExternalShuffleClient or internal shuffle service, i.e., blockTransferService.

During shuffle, following factors needs to be considered.

1. The number of opened files, which decide which one is used, BypassMergeSortShuffleWriter or UnsafeShuffleWriter?
2. The serialization and deserialization overhead. In complicated case, the SortShuffleWriter is used.

For each core, the number of shuffle file created is the same as the number of mappers running on this executor. The index maintain the offset and length for each reducer. When a block is requested (refer to NettyBlockRpcServer.receive -> blockManager.getBlockData -> IndexShuffleBlockResolver.getBlockData), the index file is used to read data from the file. Please refer to IndexShuffleBlockSolver.writeIndexFileAndCommit.

The indexFile contains the offset for each partition.

Should there be multiple concurrent attempts for one stage Reduce shuffle file