Javadoc
Merges spill files using Java FileStreams. This code path is typically slower than
the NIO-based merge,
UnsafeShuffleWriter#mergeSpillsWithTransferTo(SpillInfo[],File), and it's mostly used in cases where the IO compression codec does not support
concatenation of compressed data, when encryption is enabled, or when users have
explicitly disabled use of
transferTo in order to work around kernel bugs.
This code path might also be faster in cases where individual partition size in a spill
is small and UnsafeShuffleWriter#mergeSpillsWithTransferTo method performs many small
disk ios which is inefficient. In those case, Using large buffers for input and output
files helps reducing the number of disk ios, making the file merging faster.