Unable to find file error- while Streaming Hadoop

+4 votes

I am trying to run a hadoop-streaming python job.

bin/hadoop jar contrib/streaming/hadoop-0.20.1-streaming.jar 
-D stream.non.zero.exit.is.failure=true 
-input /ixml 
-output /oxml 
-mapper scripts/mapper.py 
-file scripts/mapper.py 
-inputreader "StreamXmlRecordReader,begin=channel,end=/channel" 
-jobconf mapred.reduce.tasks=0 

I made sure mapper.py has all the permissions. It errors out saying

Caused by: java.io.IOException: Cannot run program "mapper.py":     
error=2, No such file or directory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
    at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:214)
... 19 more
Caused by: java.io.IOException: error=2, No such file or directory
    at java.lang.UNIXProcess.forkAndExec(Native Method)
    at java.lang.UNIXProcess.(UNIXProcess.java:53)
    at java.lang.ProcessImpl.start(ProcessImpl.java:91)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
Oct 15, 2018 in Big Data Hadoop by digger
• 27,630 points
231 views

3 answers to this question.

+1 vote

I looked at the example here, it seems that you should change

-mapper scripts/mapper.py 
-file scripts/mapper.py 

to

-mapper mapper.py 
-file scripts/mapper.py 

since "shipped files go to the working directory". You might also need to specify the python interpreter directly:

-mapper /path/to/python mapper.py 
-file scripts/mapper.py 
answered Oct 15, 2018 by Omkar
• 67,290 points
+1 vote

Give permission to the mapper.py and then execute:

chmod a+x scripts/mapper.py
answered Oct 16, 2018 by Joe
+1 vote

Is python installed running on the slaves that is running the TaskTracker? If not, install it. And then do this to your code:

#!/usr/bin/python -O

'''

Here goes your code

'''

The first line (shebang) should have the path where python is installed on TaskTrackers.

answered Oct 16, 2018 by Vinith

Related Questions In Big Data Hadoop

+1 vote
1 answer
0 votes
1 answer

Where can I find hadoop-streaming.jar JAR file?

You will find the streaming jar here: ...READ MORE

answered Mar 21, 2018 in Big Data Hadoop by coldcode
• 2,020 points
866 views
0 votes
1 answer

Error while copying the file from local to HDFS

Well, the reason you are getting such ...READ MORE

answered May 2, 2018 in Big Data Hadoop by Ashish
• 2,630 points
404 views
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,510 points
2,400 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,510 points
245 views
0 votes
10 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyFr ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
12,219 views
0 votes
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,240 points
896 views
0 votes
1 answer

Getting error when writing to HDFS. Unable to load native-hadoop library for your platform

Try this: sudo service hadoop-master restart After that try ...READ MORE

answered Dec 19, 2018 in Big Data Hadoop by Omkar
• 67,290 points
121 views
0 votes
1 answer

Error while hitting the hadoop jar command to execute my map reduce

Since your jar file was inside the ...READ MORE

answered Feb 11 in Big Data Hadoop by Omkar
• 67,290 points
53 views