Hadoop: Adding multiple inputs with a different mapper using python

0 votes
I want to add multiple inputs with a different mapper for each using python script. I have used Java for this but I dont know how to do it in python. Can someone tell me how to do this?
Nov 19, 2018 in Big Data Hadoop by digger
• 26,660 points
688 views

1 answer to this question.

0 votes
hadoop jar hadoop-multiple-streaming.jar \  
  -input    myInputDirs \  
  -multiple "outputDir1|mypackage.Mapper1|mypackage.Reducer1" \  
  -multiple "outputDir2|mapper2.sh|reducer2.sh" \  
  -multiple "outputDir3|mapper3.py|reducer3.py" \  
  -multiple "outputDir4|/bin/cat|/bin/wc" \  
  -libjars  "libDir/mypackage.jar" \
  -file     "libDir/mapper2.sh" \  
  -file     "libDir/mapper3.py" \  
  -file     "libDir/reducer2.sh" \  
  -file     "libDir/reducer3.py"

Source: https://github.com/hyonaldo/hadoop-multiple-streaming

answered Nov 19, 2018 by Omkar
• 69,000 points

Related Questions In Big Data Hadoop

0 votes
1 answer
0 votes
1 answer

Error running hadoop mapreduce in Python using Hadoop Streaming

Hi As you write mapper and reducer program  ...READ MORE

answered Jan 21 in Big Data Hadoop by anonymous
317 views
0 votes
1 answer

How to get started with Hadoop and do some development using Eclipse IDE?

Alright, there are couple of things that ...READ MORE

answered Apr 4, 2018 in Big Data Hadoop by Ashish
• 2,630 points
259 views
0 votes
1 answer

Different ports in a Hadoop cluster environment?

Below image will help you in understanding ...READ MORE

answered Apr 6, 2018 in Big Data Hadoop by nitinrawat895
• 10,870 points
237 views
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,870 points
4,587 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,870 points
652 views
+1 vote
11 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyFr ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
26,063 views
–1 vote
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,310 points
1,723 views
0 votes
1 answer

Hadoop Hive: How to split a single row into multiple rows?

Try this SELECT ID1, Sub FROM tableName lateral view ...READ MORE

answered Nov 14, 2018 in Big Data Hadoop by Omkar
• 69,000 points
2,229 views
0 votes
1 answer

Hadoop with Python: How to use python to stop Hadoop tasktracker?

Here you go os.system('sudo -u hadoop bin/hadoop-daemon.sh stop ...READ MORE

answered Nov 16, 2018 in Big Data Hadoop by Omkar
• 69,000 points
47 views