Why Hadoop is not implemented using Message Passing Interface (MPI)?

0 votes

Correct me if I'm wrong, but my understanding is that Hadoop does not use MPI for communication between different nodes.

What are the technical reasons for this?

I could hazard a few guesses, but I do not know enough of how MPI is implemented "under the hood" to know whether or not I'm right.

Come to think of it, I'm not entirely familiar with Hadoop's internals either. I understand the framework at a conceptual level (map/combine/shuffle/reduce and how that works at a high level) but I don't know the nitty gritty implementation details. I've always assumed Hadoop was transmitting serialized data structures (perhaps GPBs) over a TCP connection, eg during the shuffle phase. Let me know if that's not true.

Sep 20, 2018 in Big Data Hadoop by Neha
• 6,280 points
65 views

1 answer to this question.

0 votes

One of the big features of Hadoop/map-reduce is the fault tolerance. Fault tolerance is not supported in most (any?) current MPI implementations. It is being thought about for future versions of OpenMPI.

Sandia labs has a version of map-reduce which uses MPI, but it lacks fault tolerance.

Even this can be the reason. Please read below:

MPI is Message Passing Interface. Right there in the name - there is no data locality. You send the data to another node for it to be computed on. Thus MPI is network-bound in terms of performance when working with large data.

MapReduce with the Hadoop Distributed File System duplicates data so that you can do your compute in local storage - streaming off the disk and straight to the processor. Thus MapReduce takes advantage of local storage to avoid the network bottleneck when working with large data.

This is not to say that MapReduce doesn't use the network... it does: and the shuffle is often the slowest part of a job! But it uses it as little, and as efficiently as possible.

To sum it up: Hadoop (and Google's stuff before it) did not use MPI because it could not have used MPI and worked. MapReduce systems were developed specifically to address MPI's shortcomings in light of trends in hardware: disk capacity exploding (and data with it), disk speed stagnant, networks slow, processor gigahertz peaked, multi-core taking over Moore's law

answered Sep 20, 2018 by Frankie
• 9,810 points

Related Questions In Big Data Hadoop

0 votes
1 answer

Why is using Combiner function not preferred in shuffling?

Combiner function is not preferred because when ...READ MORE

answered Jan 22 in Big Data Hadoop by Omkar
• 67,460 points
17 views
0 votes
2 answers

start-dfs.sh command is not starting Hadoop JobTracker & TaskTracker

On which version of hadoop do you ...READ MORE

answered Jul 24 in Big Data Hadoop by Lokesh Singh
657 views
0 votes
1 answer

Why am I not able to see the Hadoop daemons that are running?

I guess you are starting the services ...READ MORE

answered Apr 18, 2018 in Big Data Hadoop by Shubham
• 13,290 points
67 views
0 votes
2 answers

Hadoop is not able to find jps command

jps is actually not a command of ...READ MORE

answered Apr 18, 2018 in Big Data Hadoop by Shubham
• 13,290 points
1,807 views
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,670 points
2,695 views
0 votes
10 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyFr ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
13,413 views
0 votes
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,240 points
983 views
0 votes
1 answer
0 votes
1 answer

Why Hadoop is not implemented using MPI?

 One of the big features of Hadoop/map-reduce ...READ MORE

answered Dec 4, 2018 in Big Data Hadoop by Frankie
• 9,810 points
32 views
0 votes
1 answer