Why Hadoop is not implemented using MPI?

0 votes

What are the technical reasons why hadoop does not use MPI for communication between different nodes.?

I could hazard a few guesses, but I do not know enough of how MPI is implemented "under the hood" to know whether or not I'm right.

Come to think of it, I'm not entirely familiar with Hadoop's internals either. I understand the framework at a conceptual level (map/combine/shuffle/reduce and how that works at a high level) but I don't know the nitty gritty implementation details. I've always assumed Hadoop was transmitting serialized data structures (perhaps GPBs) over a TCP connection, eg during the shuffle phase. Let me know if that's not true.

Dec 4, 2018 in Big Data Hadoop by Neha
• 6,180 points

edited Dec 4, 2018 by Neha 20 views

1 answer to this question.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
0 votes

 One of the big features of Hadoop/map-reduce is the fault tolerance. Fault tolerance is not supported in most of the current MPI implementations. It is being thought about for future versions of OpenMPI.

answered Dec 4, 2018 by Frankie
• 9,710 points

Related Questions In Big Data Hadoop

0 votes
1 answer

Why is using Combiner function not preferred in shuffling?

Combiner function is not preferred because when ...READ MORE

answered Jan 22 in Big Data Hadoop by Omkar
• 66,970 points
14 views
0 votes
1 answer

start-dfs.sh command is not starting Hadoop JobTracker & TaskTracker

start-dfs.sh script, only starts the HDFS daemons. ...READ MORE

answered Apr 15, 2018 in Big Data Hadoop by Shubham
• 12,890 points
482 views
0 votes
1 answer

Why am I not able to see the Hadoop daemons that are running?

I guess you are starting the services ...READ MORE

answered Apr 18, 2018 in Big Data Hadoop by Shubham
• 12,890 points
45 views
0 votes
2 answers

Hadoop is not able to find jps command

jps is actually not a command of ...READ MORE

answered Apr 18, 2018 in Big Data Hadoop by Shubham
• 12,890 points
1,297 views
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 9,530 points
1,860 views
0 votes
10 answers

hadoop fs -put command?

copy command can be used to copy files ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Sujay
9,267 views
0 votes
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,260 points
672 views
0 votes
1 answer
0 votes
1 answer

Why Hadoop is not implemented using Message Passing Interface (MPI)?

One of the big features of Hadoop/map-reduce ...READ MORE

answered Sep 20, 2018 in Big Data Hadoop by Frankie
• 9,710 points
49 views
0 votes
1 answer

© 2018 Brain4ce Education Solutions Pvt. Ltd. All rights Reserved.
"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.