How to disable executor from fetching file from cache?

0 votes
Hi. I am running a Spark application and there are many executors running. I think the executors are picking up data from the cache due to which the application is going into some problem with updating new data. Is there a way to disable this?
Mar 10, 2019 in Apache Spark by Suri
168 views

1 answer to this question.

0 votes

When a Spark application is running, the executors by default will fetch data from a local cache. To change this and make the executors get data own copy of files, you have to modify the spark.files.useFetchCach property. You can do it like this:

val sc = new SparkContext(new SparkConf())
./bin/spark-submit <all your existing options> --spark.files.useFetchCache=false
answered Mar 10, 2019 by Siri

Related Questions In Apache Spark

0 votes
1 answer

How to prevent executor from self-destructing?

I think there is a timeout set ...READ MORE

answered Mar 12, 2019 in Apache Spark by Veer
108 views
0 votes
1 answer

How to create RDD from an external file source in scala?

Hi, To create an RDD from external file ...READ MORE

answered Jul 3, 2019 in Apache Spark by Gitika
• 29,170 points
245 views
+1 vote
1 answer

How to read a data from text file in Spark?

Hey, You can try this: from pyspark import SparkContext SparkContext.stop(sc) sc ...READ MORE

answered Aug 6, 2019 in Apache Spark by Gitika
• 29,170 points
1,788 views
0 votes
1 answer

Efficient way to read specific columns from parquet file in spark

As parquet is a column based storage ...READ MORE

answered Apr 20, 2018 in Apache Spark by kurt_cobain
• 9,310 points
2,370 views
+1 vote
2 answers
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,920 points
5,069 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,920 points
722 views
+1 vote
11 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyF ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
29,963 views
0 votes
1 answer

How to disable existing directory check?

To disable this, run the below commands: val ...READ MORE

answered Mar 10, 2019 in Apache Spark by Siri
74 views
0 votes
1 answer

SparkContext.addFile() not able to update file.

Spark by default won't let you overwrite ...READ MORE

answered Mar 10, 2019 in Apache Spark by Siri
735 views