Unable to find file error- while Streaming Hadoop

+4 votes

I am trying to run a hadoop-streaming python job.

bin/hadoop jar contrib/streaming/hadoop-0.20.1-streaming.jar 
-D stream.non.zero.exit.is.failure=true 
-input /ixml 
-output /oxml 
-mapper scripts/mapper.py 
-file scripts/mapper.py 
-inputreader "StreamXmlRecordReader,begin=channel,end=/channel" 
-jobconf mapred.reduce.tasks=0 

I made sure mapper.py has all the permissions. It errors out saying

Caused by: java.io.IOException: Cannot run program "mapper.py":     
error=2, No such file or directory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
    at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:214)
... 19 more
Caused by: java.io.IOException: error=2, No such file or directory
    at java.lang.UNIXProcess.forkAndExec(Native Method)
    at java.lang.UNIXProcess.(UNIXProcess.java:53)
    at java.lang.ProcessImpl.start(ProcessImpl.java:91)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
Oct 15, 2018 in Big Data Hadoop by digger
• 26,550 points
330 views

3 answers to this question.

+1 vote

I looked at the example here, it seems that you should change

-mapper scripts/mapper.py 
-file scripts/mapper.py 

to

-mapper mapper.py 
-file scripts/mapper.py 

since "shipped files go to the working directory". You might also need to specify the python interpreter directly:

-mapper /path/to/python mapper.py 
-file scripts/mapper.py 
answered Oct 15, 2018 by Omkar
• 67,660 points
+1 vote

Give permission to the mapper.py and then execute:

chmod a+x scripts/mapper.py
answered Oct 16, 2018 by Joe
+1 vote

Is python installed running on the slaves that is running the TaskTracker? If not, install it. And then do this to your code:

#!/usr/bin/python -O

'''

Here goes your code

'''

The first line (shebang) should have the path where python is installed on TaskTrackers.

answered Oct 16, 2018 by Vinith

Related Questions In Big Data Hadoop

+1 vote
1 answer
0 votes
1 answer

Where can I find hadoop-streaming.jar JAR file?

You will find the streaming jar here: ...READ MORE

answered Mar 21, 2018 in Big Data Hadoop by coldcode
• 2,020 points
1,166 views
0 votes
1 answer

Error while copying the file from local to HDFS

Well, the reason you are getting such ...READ MORE

answered May 2, 2018 in Big Data Hadoop by Ashish
• 2,630 points
553 views
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,710 points
3,294 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,710 points
390 views
0 votes
10 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyFr ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
16,208 views
0 votes
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,260 points
1,182 views
–1 vote
1 answer

Getting error when writing to HDFS. Unable to load native-hadoop library for your platform

Try this: sudo service hadoop-master restart After that try ...READ MORE

answered Dec 19, 2018 in Big Data Hadoop by Omkar
• 67,660 points
207 views
0 votes
1 answer

Error while hitting the hadoop jar command to execute my map reduce

Since your jar file was inside the ...READ MORE

answered Feb 11 in Big Data Hadoop by Omkar
• 67,660 points
87 views