How to store files in executor s working directory

Question

I have a Spark application running and there are executors to execute the task. I have some files that I want to store in the executor's working directory. How can I do this?

Answer 1

You have to specify a comma-separated list of the files you want to store in the executor's working directory. To do this, open the spark shell and run the following command:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.yarn.dist.files=<list of files>