When is speculation enabled in Spark

0 votes
Hi. I have a doubt regarding Task speculation in Spark. When does the speculation happen? When is it enabled?
Mar 12, 2019 in Apache Spark by Rajat
677 views

1 answer to this question.

0 votes

Speculation is enabled when a fraction of task is completed. By default, it is set to 0.75. So, when 75% of the task is complete, speculation is enabled. If you want to change this fraction, you can do it like this:

val sc = new SparkContext(new SparkConf())
./bin/spark-submit <all your existing options> --spark.speculation.quantile=<fraction>
answered Mar 12, 2019 by veer

Related Questions In Apache Spark

+1 vote
1 answer

Can anyone explain what is RDD in Spark?

RDD is a fundamental data structure of ...READ MORE

answered May 24, 2018 in Apache Spark by Shubham
• 13,490 points
1,548 views
0 votes
1 answer

Spark 2.3? What is new in it?

Here are the changes in new version ...READ MORE

answered May 28, 2018 in Apache Spark by kurt_cobain
• 9,390 points
290 views
0 votes
1 answer
0 votes
1 answer

Which is better in term of speed, Shark or Spark?

Spark is a framework for distributed data ...READ MORE

answered Jun 26, 2018 in Apache Spark by nitinrawat895
• 11,380 points
271 views
+1 vote
2 answers
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
8,604 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
1,517 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
75,861 views
+1 vote
3 answers

What is the difference between rdd and dataframes in Apache Spark ?

Comparison between Spark RDD vs DataFrame 1. Release ...READ MORE

answered Aug 28, 2018 in Apache Spark by shams
• 3,660 points
39,154 views
0 votes
1 answer

How many executors run when dynamic allocation is enabled?

The number of executors running by default ...READ MORE

answered Mar 13, 2019 in Apache Spark by Venu
321 views
webinar REGISTER FOR FREE WEBINAR X
Send OTP
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP