How to disable executor from fetching file from cache

0 votes
Hi. I am running a Spark application and there are many executors running. I think the executors are picking up data from the cache due to which the application is going into some problem with updating new data. Is there a way to disable this?
Mar 10, 2019 in Apache Spark by Suri
2,049 views

1 answer to this question.

0 votes

When a Spark application is running, the executors by default will fetch data from a local cache. To change this and make the executors get data own copy of files, you have to modify the spark.files.useFetchCach property. You can do it like this:

val sc = new SparkContext(new SparkConf())
./bin/spark-submit <all your existing options> --spark.files.useFetchCache=false
answered Mar 10, 2019 by Siri

Related Questions In Apache Spark

0 votes
1 answer

How to prevent executor from self-destructing?

I think there is a timeout set ...READ MORE

answered Mar 12, 2019 in Apache Spark by Veer
672 views
0 votes
1 answer

How to create RDD from an external file source in scala?

Hi, To create an RDD from external file ...READ MORE

answered Jul 4, 2019 in Apache Spark by Gitika
• 65,910 points
1,518 views
+1 vote
1 answer

How to read a data from text file in Spark?

Hey, You can try this: from pyspark import SparkContext SparkContext.stop(sc) sc ...READ MORE

answered Aug 6, 2019 in Apache Spark by Gitika
• 65,910 points
4,681 views
0 votes
1 answer

Efficient way to read specific columns from parquet file in spark

As parquet is a column based storage ...READ MORE

answered Apr 20, 2018 in Apache Spark by kurt_cobain
• 9,390 points
7,263 views
+1 vote
2 answers
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
10,555 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
2,184 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
104,199 views
0 votes
1 answer

How to disable existing directory check?

To disable this, run the below commands: val ...READ MORE

answered Mar 10, 2019 in Apache Spark by Siri
430 views
0 votes
1 answer

SparkContext.addFile() not able to update file.

Spark by default won't let you overwrite ...READ MORE

answered Mar 10, 2019 in Apache Spark by Siri
3,102 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP