Hadoop job getting executed in local mode only?

0 votes

I am trying to execute hadoop job on a pseudo distributed mode hadoop cluster. I have setup the hadoop cluster on my own. Now, the problem is when I run the program in the cluster by default it gets launched as a local job though I have mentioned the complete hdfs path to the input file. Here, is what I am getting from the prompt:

hadoop -jar MyRandomForest_oob_distance.jar  hdfs://mn1:8020/user/grim/input/legendary.txt hdfs://mn1:8020/user/grim/output/
13/05/17 16:21:25 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
13/05/17 16:21:25 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/05/17 16:21:25 INFO mapred.JobClient: Running job: job_local_0001
13/05/17 16:21:25 INFO mapred.MapTask: io.sort.mb = 100
13/05/17 16:21:25 INFO mapred.MapTask: data buffer = 79691776/99614720
13/05/17 16:21:25 INFO mapred.MapTask: record buffer = 262144/327680
13/05/17 16:21:25 WARN mapred.LocalJobRunner: job_local_0001
java.io.FileNotFoundException: File user/grim/input/legendary.txt does not exist.
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:356)
    at Data.Data.loadData(Data.java:103)
    at MapReduce.DearMapper.loadData(DearMapper.java:261)
    at MapReduce.DearMapper.setup(DearMapper.java:332)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
12/03/16 16:21:26 INFO mapred.JobClient:  map 0% reduce 0%
12/03/16 16:21:26 INFO mapred.JobClient: Job complete: job_local_0001
12/03/16 16:21:26 INFO mapred.JobClient: Counters: 0
Total Running time is: 1 secs
May 2, 2018 in Big Data Hadoop by Atul
• 10,240 points
472 views

1 answer to this question.

0 votes

Basically, you have not configured the mapreduce.framework.name property in the mapred-site.xml or it has been set to local. This is why your jobs are getting executed in local mode. For this, check your mapred-site.xml file, whether it exists or not. In case it does not exist, create it and copy the template from a file called as mapped-site.xml.template which would be there. 

So first, you have to do is:

cp mapped-site.xml.template mapped-site.xml

Next, In the mapred-site.xml file add the following property or make the necessary changes in case it exists:

<configuration>
<property>  
 <name>mapreduce.framework.name</name>  
 <value>yarn</value>  
 </property>
</configuration>

 
answered May 3, 2018 by Atul
• 10,240 points

Related Questions In Big Data Hadoop

0 votes
1 answer

Copy file from local to hdfs from the spark job in yarn mode

Please refer to the below code: import org.apache.hadoop.conf.Configuration import ...READ MORE

answered Jul 16 in Big Data Hadoop by Raj
136 views
+1 vote
1 answer

Is Hadoop only Framework in Big Data Ecosystem ?

Actually there are many other frameworks, one of ...READ MORE

answered Mar 26, 2018 in Big Data Hadoop by Ashish
• 2,630 points
67 views
0 votes
1 answer

Not able to start Job History Server in Hadoop 2.8.1

You have to start JobHistoryServer process specifically ...READ MORE

answered Mar 29, 2018 in Big Data Hadoop by Ashish
• 2,630 points
428 views
0 votes
1 answer

When hadoop-env.sh will be executed in hadoop

Yes you need to put in the ...READ MORE

answered Apr 3, 2018 in Big Data Hadoop by kurt_cobain
• 9,240 points
71 views
0 votes
1 answer
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,670 points
2,730 views
0 votes
10 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyFr ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
13,518 views
0 votes
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,240 points
989 views
0 votes
1 answer