How to reduce shuffling cost in map reduce?

0 votes
It is said in many blogs that shuffling is a costly phase because it has to process all the key value pair. Is there a way to reduce this cost while shuffling?
Jan 22 in Big Data Hadoop by slayer
• 29,050 points
74 views

1 answer to this question.

0 votes

You can define a Combiner function, which does the same calculations as Reducer but on the Mappers’ side. The role of the Combiner is to precalculate the data so that we send less data over the network. Another solution is to compress the intermediate output so that the time required to shuffle the data will be minimized. 

answered Jan 22 by Omkar
• 67,120 points

Related Questions In Big Data Hadoop

0 votes
1 answer

How to set the number of Map & Reduce tasks?

The map tasks created for a job ...READ MORE

answered Apr 18, 2018 in Big Data Hadoop by Shubham
• 13,110 points
45 views
+2 votes
1 answer

How to calculate Maximum salary of the employee with the name using the Map Reduce Technique

Please try the below code and it ...READ MORE

answered Jul 25, 2018 in Big Data Hadoop by Neha
• 6,260 points
301 views
0 votes
1 answer

How to run Map Reduce program using Ubuntu terminal?

 I used the following steps to execute it ...READ MORE

answered Aug 7, 2018 in Big Data Hadoop by Neha
• 6,260 points
76 views
0 votes
1 answer

How to pass large records to map/reduce tasks?

Hadoop is not designed for records about ...READ MORE

answered Sep 25, 2018 in Big Data Hadoop by Frankie
• 9,810 points
56 views
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,030 points
2,038 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,030 points
192 views
0 votes
10 answers

hadoop fs -put command?

copy command can be used to copy files ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Sujay
10,379 views
0 votes
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,240 points
760 views
+1 vote
1 answer

How to count number of rows in alias in PIG?

COUNT is part of pig LOGS= LOAD 'log'; LOGS_GROUP= ...READ MORE

answered Oct 15, 2018 in Big Data Hadoop by Omkar
• 67,120 points
50 views
0 votes
3 answers

How to specify KeyValueTextInputFormat Separator in Hadoop-.20 api?

conf.set("key.value.separator.in.input.line", ","); Job job = new J ...READ MORE

answered Dec 4, 2018 in Big Data Hadoop by Rio
118 views