Hadoop job getting executed in local mode only

0 votes

I am trying to execute hadoop job on a pseudo distributed mode hadoop cluster. I have setup the hadoop cluster on my own. Now, the problem is when I run the program in the cluster by default it gets launched as a local job though I have mentioned the complete hdfs path to the input file. Here, is what I am getting from the prompt:

hadoop -jar MyRandomForest_oob_distance.jar  hdfs://mn1:8020/user/grim/input/legendary.txt hdfs://mn1:8020/user/grim/output/
13/05/17 16:21:25 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
13/05/17 16:21:25 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/05/17 16:21:25 INFO mapred.JobClient: Running job: job_local_0001
13/05/17 16:21:25 INFO mapred.MapTask: io.sort.mb = 100
13/05/17 16:21:25 INFO mapred.MapTask: data buffer = 79691776/99614720
13/05/17 16:21:25 INFO mapred.MapTask: record buffer = 262144/327680
13/05/17 16:21:25 WARN mapred.LocalJobRunner: job_local_0001
java.io.FileNotFoundException: File user/grim/input/legendary.txt does not exist.
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:356)
    at Data.Data.loadData(Data.java:103)
    at MapReduce.DearMapper.loadData(DearMapper.java:261)
    at MapReduce.DearMapper.setup(DearMapper.java:332)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
12/03/16 16:21:26 INFO mapred.JobClient:  map 0% reduce 0%
12/03/16 16:21:26 INFO mapred.JobClient: Job complete: job_local_0001
12/03/16 16:21:26 INFO mapred.JobClient: Counters: 0
Total Running time is: 1 secs
May 3, 2018 in Big Data Hadoop by Atul
• 10,240 points

1 answer to this question.

0 votes

Basically, you have not configured the mapreduce.framework.name property in the mapred-site.xml or it has been set to local. This is why your jobs are getting executed in local mode. For this, check your mapred-site.xml file, whether it exists or not. In case it does not exist, create it and copy the template from a file called as mapped-site.xml.template which would be there. 

So first, you have to do is:

cp mapped-site.xml.template mapped-site.xml

Next, In the mapred-site.xml file add the following property or make the necessary changes in case it exists:


answered May 3, 2018 by Atul
• 10,240 points

Related Questions In Big Data Hadoop

0 votes
1 answer

Copy file from local to hdfs from the spark job in yarn mode

Please refer to the below code: import org.apache.hadoop.conf.Configuration import ...READ MORE

answered Jul 16, 2019 in Big Data Hadoop by Raj
0 votes
0 answers

How I can kill the jobs using jobID running in local mode with Hadoop

I am Running hadoop jobs in local ...READ MORE

Aug 26, 2020 in Big Data Hadoop by kamboj
• 140 points
+1 vote
1 answer

Is Hadoop only Framework in Big Data Ecosystem ?

Actually there are many other frameworks, one of ...READ MORE

answered Mar 26, 2018 in Big Data Hadoop by Ashish
• 2,650 points
0 votes
1 answer

Not able to start Job History Server in Hadoop 2.8.1

You have to start JobHistoryServer process specifically ...READ MORE

answered Mar 30, 2018 in Big Data Hadoop by Ashish
• 2,650 points
0 votes
1 answer

When hadoop-env.sh will be executed in hadoop

Yes you need to put in the ...READ MORE

answered Apr 3, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
–1 vote
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points
0 votes
1 answer
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP