Increase Yarn wait time for Sparkcontext

0 votes
Hi. I want to increase the time that the Yarn application master waits for the SparkContext to be initialized. What is the command I have to use to do this?
Mar 27, 2019 in Apache Spark by Karan
1,065 views

1 answer to this question.

0 votes

The default time that the Yarn application waits for the SparkContext is 100s. If you want to change it, open the Spark shell and run the following command by setting the new wait time:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.yarn.am.waitTime=<new wait time>
answered Mar 27, 2019 by Rohit

Related Questions In Apache Spark

0 votes
1 answer

How to make Spark wait for more time for acknowledgement?

Use the following command to increase the ...READ MORE

answered Mar 11, 2019 in Apache Spark by Raj
1,325 views
0 votes
1 answer

How to increase wait time to launch data-local task?

You can increase the locality wait time ...READ MORE

answered Mar 11, 2019 in Apache Spark by Raj
201 views
0 votes
1 answer

Increase cores for yarn in Spark application

By default, only one core is used for ...READ MORE

answered Mar 26, 2019 in Apache Spark by Bhuvan
609 views
0 votes
1 answer

How to increase the amount of data to be transferred to shuffle service at the same time?

The amount of data to be transferred ...READ MORE

answered Mar 1, 2019 in Apache Spark by Omkar
• 69,190 points
400 views
+1 vote
2 answers
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
9,088 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
1,636 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
82,770 views
0 votes
1 answer

How to increase HDFS replication level in Spark?

Hi @Raunak. You can change the replication ...READ MORE

answered Mar 27, 2019 in Apache Spark by Yash
787 views
0 votes
1 answer

Need help setting Spark yarn history server address

If you are running history server and ...READ MORE

answered Mar 27, 2019 in Apache Spark by Neha
960 views
webinar REGISTER FOR FREE WEBINAR X
Send OTP
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP