Unable to find file error- while Streaming Hadoop

Question

I am trying to run a hadoop-streaming python job.

bin/hadoop jar contrib/streaming/hadoop-0.20.1-streaming.jar 
-D stream.non.zero.exit.is.failure=true 
-input /ixml 
-output /oxml 
-mapper scripts/mapper.py 
-file scripts/mapper.py 
-inputreader "StreamXmlRecordReader,begin=channel,end=/channel" 
-jobconf mapred.reduce.tasks=0 

I made sure mapper.py has all the permissions. It errors out saying

Caused by: java.io.IOException: Cannot run program "mapper.py":     
error=2, No such file or directory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
    at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:214)
... 19 more
Caused by: java.io.IOException: error=2, No such file or directory
    at java.lang.UNIXProcess.forkAndExec(Native Method)
    at java.lang.UNIXProcess.(UNIXProcess.java:53)
    at java.lang.ProcessImpl.start(ProcessImpl.java:91)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)

Answer 1

I looked at the example here, it seems that you should change

-mapper scripts/mapper.py 
-file scripts/mapper.py 

to

-mapper mapper.py 
-file scripts/mapper.py 

since "shipped files go to the working directory". You might also need to specify the python interpreter directly:

-mapper /path/to/python mapper.py 
-file scripts/mapper.py 

Answer 2

Give permission to the mapper.py and then execute:

chmod a+x scripts/mapper.py

Answer 3

Is python installed running on the slaves that is running the TaskTracker? If not, install it. And then do this to your code:

#!/usr/bin/python -O

'''

Here goes your code

'''

The first line (shebang) should have the path where python is installed on TaskTrackers.