How to store files in executor's working directory?

0 votes
I have a Spark application running and there are executors to execute the task. I have some files that I want to store in the executor's working directory. How can I do this?
Mar 28, 2019 in Apache Spark by Simran
924 views

1 answer to this question.

0 votes

You have to specify a comma-separated list of the files you want to store in the executor's working directory. To do this, open the spark shell and run the following command:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.yarn.dist.files=<list of files>
answered Mar 28, 2019 by Raj

Related Questions In Apache Spark

0 votes
1 answer

How to set executors for static allocation in Spark Yarn?

Open Spark shell and run the following ...READ MORE

answered Mar 28, 2019 in Apache Spark by Raj
370 views
0 votes
1 answer

How do I connect to a HIVE Meta store through a program in SparkSQL?

In spark 2.0.+ it should look something ...READ MORE

answered Sep 5, 2019 in Apache Spark by ravikiran
• 4,600 points
1,098 views
0 votes
1 answer

How to unzip a folder to individual files in HDFS?

Hi, @Amey, You can go through this regarding ...READ MORE

answered 6 days ago in Apache Spark by Gitika
• 29,290 points
38 views
0 votes
1 answer

How to get the number of elements in partition?

rdd.mapPartitions(iter => Array(iter.size).iterator, true) This command will ...READ MORE

answered May 8, 2018 in Apache Spark by kurt_cobain
• 9,310 points
445 views
+1 vote
2 answers
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,920 points
5,081 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,920 points
724 views
+1 vote
11 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyF ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
30,091 views
0 votes
1 answer

How to automatically kill executors on blacklisting?

You can set the property to directly ...READ MORE

answered Mar 12, 2019 in Apache Spark by Veer
127 views
0 votes
1 answer

How to enable dynamic resource allocation in Spark?

To dynamically enable dynamic resource allocation, you ...READ MORE

answered Mar 12, 2019 in Apache Spark by veer
421 views