Hadoop job getting executed in local mode only?

0 votes

I am trying to execute hadoop job on a pseudo distributed mode hadoop cluster. I have setup the hadoop cluster on my own. Now, the problem is when I run the program in the cluster by default it gets launched as a local job though I have mentioned the complete hdfs path to the input file. Here, is what I am getting from the prompt:

hadoop -jar MyRandomForest_oob_distance.jar  hdfs://mn1:8020/user/grim/input/legendary.txt hdfs://mn1:8020/user/grim/output/
13/05/17 16:21:25 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
13/05/17 16:21:25 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/05/17 16:21:25 INFO mapred.JobClient: Running job: job_local_0001
13/05/17 16:21:25 INFO mapred.MapTask: io.sort.mb = 100
13/05/17 16:21:25 INFO mapred.MapTask: data buffer = 79691776/99614720
13/05/17 16:21:25 INFO mapred.MapTask: record buffer = 262144/327680
13/05/17 16:21:25 WARN mapred.LocalJobRunner: job_local_0001
java.io.FileNotFoundException: File user/grim/input/legendary.txt does not exist.
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:356)
    at Data.Data.loadData(Data.java:103)
    at MapReduce.DearMapper.loadData(DearMapper.java:261)
    at MapReduce.DearMapper.setup(DearMapper.java:332)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
12/03/16 16:21:26 INFO mapred.JobClient:  map 0% reduce 0%
12/03/16 16:21:26 INFO mapred.JobClient: Job complete: job_local_0001
12/03/16 16:21:26 INFO mapred.JobClient: Counters: 0
Total Running time is: 1 secs
May 2, 2018 in Big Data Hadoop by Atul
• 10,200 points
351 views

1 answer to this question.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
0 votes

Basically, you have not configured the mapreduce.framework.name property in the mapred-site.xml or it has been set to local. This is why your jobs are getting executed in local mode. For this, check your mapred-site.xml file, whether it exists or not. In case it does not exist, create it and copy the template from a file called as mapped-site.xml.template which would be there. 

So first, you have to do is:

cp mapped-site.xml.template mapped-site.xml

Next, In the mapred-site.xml file add the following property or make the necessary changes in case it exists:

<configuration>
<property>  
 <name>mapreduce.framework.name</name>  
 <value>yarn</value>  
 </property>
</configuration>

 
answered May 3, 2018 by Atul
• 10,200 points

Related Questions In Big Data Hadoop

+1 vote
1 answer

Is Hadoop only Framework in Big Data Ecosystem ?

Actually there are many other frameworks, one of ...READ MORE

answered Mar 26, 2018 in Big Data Hadoop by Ashish
• 2,630 points
36 views
0 votes
1 answer

Not able to start Job History Server in Hadoop 2.8.1

You have to start JobHistoryServer process specifically ...READ MORE

answered Mar 29, 2018 in Big Data Hadoop by Ashish
• 2,630 points
229 views
0 votes
1 answer

When hadoop-env.sh will be executed in hadoop

Yes you need to put in the ...READ MORE

answered Apr 3, 2018 in Big Data Hadoop by kurt_cobain
• 9,260 points
22 views
0 votes
1 answer
0 votes
1 answer

Getting error in MapReduce job.setInputFormatClass

In old Hadoop API(i.e. below Hadoop 0.20.0), ...READ MORE

answered Apr 15, 2018 in Big Data Hadoop by Shubham
• 12,150 points
228 views
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 9,030 points
1,651 views
0 votes
10 answers

hadoop fs -put command?

copy command can be used to copy files ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Sujay
7,990 views
0 votes
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,260 points
555 views
0 votes
1 answer

© 2018 Brain4ce Education Solutions Pvt. Ltd. All rights Reserved.
"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.