How to reduce shuffling cost in map reduce

0 votes
It is said in many blogs that shuffling is a costly phase because it has to process all the key value pair. Is there a way to reduce this cost while shuffling?
Jan 22, 2019 in Big Data Hadoop by slayer
• 29,370 points
2,060 views

1 answer to this question.

0 votes

You can define a Combiner function, which does the same calculations as Reducer but on the Mappers’ side. The role of the Combiner is to precalculate the data so that we send less data over the network. Another solution is to compress the intermediate output so that the time required to shuffle the data will be minimized. 

answered Jan 22, 2019 by Omkar
• 69,220 points

Related Questions In Big Data Hadoop

0 votes
1 answer

How to set the number of Map & Reduce tasks?

The map tasks created for a job ...READ MORE

answered Apr 18, 2018 in Big Data Hadoop by Shubham
• 13,490 points
1,940 views
+2 votes
1 answer

How to calculate Maximum salary of the employee with the name using the Map Reduce Technique

Please try the below code and it ...READ MORE

answered Jul 25, 2018 in Big Data Hadoop by Neha
• 6,300 points
5,671 views
0 votes
1 answer

How to run Map Reduce program using Ubuntu terminal?

 I used the following steps to execute it ...READ MORE

answered Aug 7, 2018 in Big Data Hadoop by Neha
• 6,300 points
4,111 views
0 votes
1 answer

How to pass large records to map/reduce tasks?

Hadoop is not designed for records about ...READ MORE

answered Sep 25, 2018 in Big Data Hadoop by Frankie
• 9,830 points
1,459 views
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
11,067 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
2,569 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
109,015 views
–1 vote
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,350 points
4,633 views
+1 vote
1 answer

How to count number of rows in alias in PIG?

COUNT is part of pig LOGS= LOAD 'log'; LOGS_GROUP= ...READ MORE

answered Oct 15, 2018 in Big Data Hadoop by Omkar
• 69,220 points
2,834 views
0 votes
3 answers

How to specify KeyValueTextInputFormat Separator in Hadoop-.20 api?

conf.set("key.value.separator.in.input.line", ","); Job job = new ...READ MORE

answered Dec 5, 2018 in Big Data Hadoop by Rio
1,946 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP