How to reduce shuffling cost in map reduce

Question

It is said in many blogs that shuffling is a costly phase because it has to process all the key value pair. Is there a way to reduce this cost while shuffling?

Answer 1

You can define a Combiner function, which does the same calculations as Reducer but on the Mappers’ side. The role of the Combiner is to precalculate the data so that we send less data over the network. Another solution is to compress the intermediate output so that the time required to shuffle the data will be minimized.