Big Data and Hadoop (170 Blogs) Become a Certified Professional
AWS Global Infrastructure

Big Data

Topics Covered
  • Big Data and Hadoop (146 Blogs)
  • Hadoop Administration (8 Blogs)
  • Apache Storm (4 Blogs)
  • Apache Spark and Scala (29 Blogs)

Hadoop Streaming: Writing A Hadoop MapReduce Program In Python

Last updated on Aug 11,2023 60.6K Views

Rakesh Ray
Rakesh is a Big Data Analytics enthusiast who works as a Technical... Rakesh is a Big Data Analytics enthusiast who works as a Technical Consultant at Edureka. He is passionate about new technologies which provides solution...
3 / 4 Blog from YARN & MapReduce

The quantity of digital data generated every day is growing exponentially with the advent of Digital Media, Internet of Things among other developments. This scenario has given rise to challenges in creating next generation tools and technologies to store and manipulate these data. This is where Hadoop Streaming comes in! Given below is a graph which depicts the growth of data generated annually in the world from 2013. IDC estimates that the amount of data created annually will reach 180 Zettabytes in 2025!


Source: IDC

IBM states that, every day, almost 2.5 quintillion bytes of data are created, with 90 percent of world’s data created in the last two years! It is a challenging task to store such an expansive amount of data. Hadoop can handle large volumes of structured and unstructured data more efficiently than the traditional enterprise Data Warehouse. It stores these enormous data sets across distributed clusters of computers. Hadoop Streaming uses MapReduce framework which can be used to write applications to process humongous amounts of data.

🐍 Ready to Unleash the Power of Python? Sign Up for Edureka’s Comprehensive Online instructor led Python Training with access to hundreds of Python learning Modules and 24/7 technical support.

Since MapReduce framework is based on Java, you might be wondering how a developer can work on it if he/ she does not have experience in Java. Well, developers can write mapper/Reducer application using their preferred language and without having much knowledge of Java, using Hadoop Streaming rather than switching to new tools or technologies like Pig and Hive. Learn more about Pig and Hive with help of this Big Data course designed by the Top Industry experts in Big Data Platform.

What is Hadoop Streaming?

Hadoop Streaming is a utility that comes with the Hadoop distribution. It can be used to execute programs for big data analysis. Hadoop streaming can be performed using languages like Python, Java, PHP, Scala, Perl, UNIX, and many more. The utility allows us to create and run Map/Reduce jobs with any executable or script as the mapper and/or the reducer. For example:

$HADOOP_HOME/bin/hadoop  jar $HADOOP_HOME/hadoop-streaming.jar

-input myInputDirs

-output myOutputDir

-mapper /bin/cat

-reducer /bin/wc

Parameters Description:


Python MapReduce Code:
import sys
#Word Count Example
# input comes from standard input STDIN
for line in sys.stdin:
line = line.strip() #remove leading and trailing whitespaces
words = line.split() #split the line into words and returns as a list
for word in words:
#write the results to standard output STDOUT
print'%s	%s' % (word,1) #Emit the word


import sys
from operator import itemgetter
# using a dictionary to map words to their counts
current_word = None
current_count = 0
word = None
# input comes from STDIN
for line in sys.stdin:
line = line.strip()
word,count = line.split('	',1)
count = int(count)
except ValueError:
if current_word == word:
current_count += count
if current_word:
print '%s	%s' % (current_word, current_count)
current_count = count
current_word = word
if current_word == word:
print '%s	%s' % (current_word,current_count)



  1. Create a file with the following content and name it word.txt.

Cat mouse lion deer Tiger lion Elephant lion deer

  1. Copy the and scripts to the same folder where the above file exists.


  1. Open terminal and  Locate the directory of the file.Command:ls    : to list all files in the directorycd   : to change directory/folder


  1. See the content of the file.
    Command:  cat file_name


> content of

   command:  cat


>Content of

command: cat



We can run mapper and reducer on local files (ex: word.txt). In order to run the Map and reduce on the Hadoop Distributed File System (HDFS), we need the Hadoop Streaming jar. So before we run the scripts on HDFS, let’s run them locally to ensure that they are working fine.

>Run the mapper

command: cat word.txt | python



command:  cat word.txt | python | sort -k1,1 | python


We can see that the mapper and reducer are working as expected so we won’t face any further issues.

Running the Python Code on Hadoop

Before we run the MapReduce task on Hadoop, copy local data (word.txt) to HDFS

>example: hdfs dfs -put source_directory hadoop_destination_directory

command: hdfs dfs -put /home/edureka/MapReduce/word.txt   /user/edureka


Copy the path of the jar file

The path of Hadoop Streaming jar based on the version of the jar is:


So locate the Hadoop Streaming jar on your terminal and copy the path.


ls /usr/lib/hadoop-2.2.0/share/hadoop/tools/lib/hadoop-streaming-2.2.0.jar


Run the MapReduce job


hadoop jar /usr/lib/hadoop-2.2.0/share/hadoop/tools/lib/hadoop-streaming-2.2.0.jar -file /home/edureka/ -mapper -file   /home/edureka/ -reducer -input /user/edureka/word -output /user/edureka/Wordcount



Hadoop provides a basic web interface for statistics and information. When Hadoop cluster is running open http://localhost:50070 in browser. Here is the screenshot of the Hadoop web interface.


Now browse the filesystem and locate the wordcount file generated to see the output. Below is the screenshot.


We can see the output on the terminal using this command

command: hadoop fs -cat /user/edureka/Wordcount/part-00000


You have now learnt how to execute a MapReduce program written in Python using Hadoop Streaming!

Edureka has a live and instructor-led course on Big Data & Hadoop, co-created by industry practitioners to get Big Data Certifications.The Edureka’s Big Data Masters Course helps learners become expert in HDFS, Yarn, MapReduce, Pig, Hive, HBase, Oozie, Flume and Sqoop using real-time use cases on Retail, Social Media, Aviation, Tourism, Finance domain.

Got a question for us? Please mention it in the comments section and we will get back to you.

  • Sanjay Soni says:

    how to subscribe blog .. for the daily update?

  • courserachin chin says:

    Nice Blog! I’m doing my college project on mapreduce wordcount… Could you please suggest me an idea where I can make the use of wordcount program? And where to get the large data from to perform the mapreduce wordcount.. I’m using cloudera on virtual box. Thank you very much!

  • Ajay says:

    Nice description of the topic !!!

    • EdurekaSupport says:

      Thanks for checking out the blog, Ajay! Do subscribe to our blog to stay updated on upcoming Hadoop posts. Cheers!

  • Gopesh kumar says:

    nice blog !!!

    • EdurekaSupport says:

      Thanks, Gopesh! Do subscribe to our blog to stay updated on upcoming Hadoop posts. Cheers!

Join the discussion

Browse Categories

webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP

Subscribe to our Newsletter, and get personalized recommendations.

image not found!
image not found!

Hadoop Streaming: Writing A Hadoop MapReduce Program In Python