Compression

spark.io.compression.codec

spark.shuffle.compress

spark.shuffle.spill.compress

Spill may use different shuffle compression, please refer to following code and reference on how it is determined. case : TempLocalBlockId => compressShuffleSpill case : TempShuffleBlockId => compressShuffle

Sort-based shuffle compression behavior is inconsistent

results matching ""

    No results matching ""