How to disable executor from fetching file from cache?

0 votes
Hi. I am running a Spark application and there are many executors running. I think the executors are picking up data from the cache due to which the application is going into some problem with updating new data. Is there a way to disable this?
Mar 10 in Apache Spark by Suri
58 views

1 answer to this question.

0 votes

When a Spark application is running, the executors by default will fetch data from a local cache. To change this and make the executors get data own copy of files, you have to modify the spark.files.useFetchCach property. You can do it like this:

val sc = new SparkContext(new SparkConf())
./bin/spark-submit <all your existing options> --spark.files.useFetchCache=false
answered Mar 10 by Siri

Related Questions In Apache Spark

0 votes
1 answer

How to prevent executor from self-destructing?

I think there is a timeout set ...READ MORE

answered Mar 12 in Apache Spark by Veer
43 views
0 votes
1 answer

How to create RDD from an external file source in scala?

Hi, To create an RDD from external file ...READ MORE

answered Jul 3 in Apache Spark by Gitika
• 25,340 points
40 views
0 votes
1 answer

How to read a data from text file in Spark?

Hey, You can try this: from pyspark import SparkContext SparkContext.stop(sc) sc ...READ MORE

answered Aug 6 in Apache Spark by Gitika
• 25,340 points
89 views
0 votes
1 answer

Efficient way to read specific columns from parquet file in spark

As parquet is a column based storage ...READ MORE

answered Apr 20, 2018 in Apache Spark by kurt_cobain
• 9,240 points
1,221 views
0 votes
1 answer
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,670 points
2,695 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,670 points
284 views
0 votes
10 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyFr ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
13,411 views
0 votes
1 answer

How to disable existing directory check?

To disable this, run the below commands: val ...READ MORE

answered Mar 10 in Apache Spark by Siri
17 views
0 votes
1 answer

SparkContext.addFile() not able to update file.

Spark by default won't let you overwrite ...READ MORE

answered Mar 10 in Apache Spark by Siri
175 views