Why do we use job waitForCompletion true

0 votes

I have seen this line being used in many mapreduce codes. I am new to mapreduce so I am not understanding it's significance, 

Why do we use job.waitForCompletion(true)? Is it because internally the code uses threads.

Jul 22, 2019 in Big Data Hadoop by Raghu
613 views

1 answer to this question.

0 votes

The main reason for job.waitForCompletion exists is that its method call returns only when the job gets finished, and it returns with its success or failure status which can be used to determine that further steps are to be run or not.

 Actually, the files is split in blocks and each block is executed on a separate node. All the map tasks run in parallel and then are fed to the reducer after they are done. There is no question of synchronization as you would think about in a multi threaded program. In multi-threaded program, all the threads are running on the same box and since they share some of the data you have to synchronize them. But here threads are not involved unless you use threads to submit jobs in parallel and wait for their completion. For that, you have to use a job class instance per thread.

answered Jul 22, 2019 by Kiran

Related Questions In Big Data Hadoop

0 votes
1 answer

Why do we use job.waitForCompletion(true)?

The main reason for job.waitForCompletion exists is ...READ MORE

answered Jul 10, 2019 in Big Data Hadoop by Josh
3,697 views
+1 vote
1 answer

Why do we use STORE command in pig?

Hey, We use store command to store the ...READ MORE

answered May 7, 2019 in Big Data Hadoop by Gitika
• 65,890 points
1,454 views
0 votes
1 answer

Why we are configuring mapred.job.tracker in YARN?

I really dont know the reason behind ...READ MORE

answered Mar 29, 2018 in Big Data Hadoop by Ashish
• 2,650 points
1,167 views
0 votes
1 answer

Why should we use "extends Mapper" for Mapreduce code?

The Mapper class belongs to package org.apache.hadoop.mapreduce ...READ MORE

answered Feb 8, 2019 in Big Data Hadoop by Omkar
• 69,190 points
450 views
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
9,361 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
1,719 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
86,763 views
–1 vote
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points
3,592 views
0 votes
1 answer

Why we need to install "ant -Dhadoopversion=23" ? What is the use of this?

Hi, We install ant to build pig, If the ...READ MORE

answered Apr 26, 2019 in Big Data Hadoop by Gitika
• 65,890 points
257 views
0 votes
1 answer

Why should we use "distinct" keyword in pig script?

Hey, The "distinct" statement is very simple. It ...READ MORE

answered May 3, 2019 in Big Data Hadoop by Gitika
• 65,890 points
365 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP