The number of stages in a job is equal to the number of RDDs in DAG however under one of the cgiven conditions the scheduler can truncate the lineage identify it

0 votes
The number of stages in a job is equal to the number of RDDs in DAG. however, under one of the cgiven conditions, the scheduler can truncate the lineage. identify it.
a)  when there is no movement of data from parent RDD
b)  when there is unshuffle in partition
c)  when the RDD is cached or persisted
d)  when the RDD is shuffled
Nov 25, 2020 in Apache Spark by Edureka
• 200 points
1,558 views

1 answer to this question.

0 votes

Hi@Edureka,

Spark's internal scheduler may truncate the lineage of the RDD graph if an existing RDD has already been persisted in cluster memory or on disk. So option C may be the right answer.

answered Nov 26, 2020 by MD
• 95,340 points

Related Questions In Apache Spark

0 votes
0 answers
0 votes
1 answer

The number of stages in a job is equal to the number of RDDs in DAG. however, under one of the cgiven conditions, the scheduler can truncate the lineage. identify it.

Hi@ritu, Spark's internal scheduler may truncate the lineage of the RDD graph if ...READ MORE

answered Nov 25, 2020 in Apache Spark by akhtar
• 38,240 points
1,171 views
0 votes
1 answer
0 votes
1 answer
0 votes
1 answer
0 votes
1 answer

How to get the number of elements in partition?

rdd.mapPartitions(iter => Array(iter.size).iterator, true) This command will ...READ MORE

answered May 8, 2018 in Apache Spark by kurt_cobain
• 9,390 points
1,082 views
webinar REGISTER FOR FREE WEBINAR X
Send OTP
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP