Hadoop job getting executed in local mode only

Question

I am trying to execute hadoop job on a pseudo distributed mode hadoop cluster. I have setup the hadoop cluster on my own. Now, the problem is when I run the program in the cluster by default it gets launched as a local job though I have mentioned the complete hdfs path to the input file. Here, is what I am getting from the prompt:

hadoop -jar MyRandomForest_oob_distance.jar  hdfs://mn1:8020/user/grim/input/legendary.txt hdfs://mn1:8020/user/grim/output/
13/05/17 16:21:25 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=

13/05/17 16:21:25 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/05/17 16:21:25 INFO mapred.JobClient: Running job: job_local_0001

13/05/17 16:21:25 INFO mapred.MapTask: io.sort.mb = 100

13/05/17 16:21:25 INFO mapred.MapTask: data buffer = 79691776/99614720

13/05/17 16:21:25 INFO mapred.MapTask: record buffer = 262144/327680

13/05/17 16:21:25 WARN mapred.LocalJobRunner: job_local_0001
java.io.FileNotFoundException: File user/grim/input/legendary.txt does not exist.
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:356)
    at Data.Data.loadData(Data.java:103)
    at MapReduce.DearMapper.loadData(DearMapper.java:261)
    at MapReduce.DearMapper.setup(DearMapper.java:332)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
12/03/16 16:21:26 INFO mapred.JobClient:  map 0% reduce 0%
12/03/16 16:21:26 INFO mapred.JobClient: Job complete: job_local_0001
12/03/16 16:21:26 INFO mapred.JobClient: Counters: 0
Total Running time is: 1 secs

Answer 1

Basically, you have not configured the mapreduce.framework.name property in the mapred-site.xml or it has been set to local. This is why your jobs are getting executed in local mode. For this, check your mapred-site.xml file, whether it exists or not. In case it does not exist, create it and copy the template from a file called as mapped-site.xml.template which would be there. 

So first, you have to do is:

cp mapped-site.xml.template mapped-site.xml

Next, In the mapred-site.xml file add the following property or make the necessary changes in case it exists:

<configuration>
<property>  
 <name>mapreduce.framework.name</name>  
 <value>yarn</value>  
 </property>
</configuration>